feat(engine): open drawElement fast capture to Windows hardware GPU

[vanceingalls]

What

1. Widens the default-on drawElement clamp: darwin → darwin || win32 (still requires a non-software-GPU browser). Linux stays excluded — that fleet is headless/Docker SwiftShader, where DE has known defects and no speedup.
2. gpu_renderer telemetry — DE session init now keeps the WebGL UNMASKED_RENDERER_WEBGL string it was already reading and discarding (detectSwiftShader generalized to detectGpuBackend, same single page evaluate), threaded session → CapturePerfSummary → RenderPerfSummary → render_complete.

Why

30-day telemetry: ~206k non-CI hardware-GPU Windows renders (~78% of the win32 fleet, 18k installs) are pinned to the slow screenshot path purely by the darwin-only clamp — the second-largest perf population after macOS, carrying ~1,550 capture-hours/month in the DE-eligible ≥700f band at a measured ~2× speedup opportunity.

The darwin restriction was a validation envelope, not an architectural limit: the CanvasDrawElement flag ships on every platform. Every layer that made the macOS default-on release (v0.7.38) survivable is platform-neutral and untouched here:

• compile-time gates + SwiftShader init gate (software-GL still diverts per session, any OS)
• per-render worker-encode self-verification + screenshot fallback — worst case on an unvalidated D3D11 backend is identical to Metal: a caught bad frame and a screenshot re-render
• blank guard; 24 GB parallel floor; heap clamps

Why gpu_renderer specifically

drawElement damage proved compositor-backend-specific through the entire macOS rollout (Metal vs SwiftShader behaved differently at the paint-record level). D3D11 is a third backend, so its failures must cluster by ANGLE backend + GPU vendor (NVIDIA/AMD/Intel), not just os=win32. Post-merge watch on dashboard 1783183: de_self_verify_fallback, de_fallback_reason, de_verify_min_db segmented by os × gpu_renderer.

Refactor included

The two DE clamp branches became a pure, unit-tested resolveDefaultDrawElement(platform, gpuMode, workerEncode, explicitOptIn) — also drops resolveConfig's cyclomatic complexity below the fallow threshold. The win32 streaming-encode compound tests collapse onto one shared helper (fallow clone groups cleared).

Verification

• resolveDefaultDrawElement: darwin/win32 engage, linux doesn't, software clamps, opt-in overrides, no-worker-encode clamps — asserted directly per platform rather than conditioned on the host
• detectGpuBackend: renderer carried, null-WebGL, SwiftShader wrapper back-compat
• 102 engine config + drawElementService tests, typecheck clean across engine/producer/cli, fallow ✓ no issues in 10 changed files

Kill switches (unchanged)

PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false, --experimental-fast-capture=false, per-session init gates, per-frame verify.

Composes with #2838 / #2840: once all three land, win32 DE renders become parallel-router eligible under the same 700-frame floor.

🤖 Generated with Claude Code

[miga-heygen]

SSOT -- resolveDefaultDrawElement is a clean canonical decision point

The refactor extracts the inline platform + gpu + workerEncode clamp into a pure, tested function. I traced every combination against the old inline logic:

platform	gpuMode	workerEncode	explicitOptIn	old result	new result
darwin	hardware	true	false	true	true
darwin	auto	true	false	true	true
darwin	software	true	false	false	false
win32	hardware	true	false	false	true
win32	auto	true	false	false	true
win32	software	true	false	false	false
linux	hardware	true	false	false	false
any	any	false	false	false	false
linux	software	false	true	true	true

The intentional change (darwin-only to darwin|win32) is exactly where it should be, and explicitOptIn bypasses both the platform gate AND the workerEncode gate, preserving the old semantics. The isDrawElementPlatform helper is private, resolveDefaultDrawElement is the exported tested surface. The function is genuinely pure -- no process.platform read, everything injected. Good design.

The "auto" GPU mode correctly passes the clamp (it is not "software"), matching the old !== "software" check. Both tests and comments confirm this.

No issues here.

detectGpuBackend backward compatibility

detectSwiftShader is preserved as a thin wrapper over the new detectGpuBackend. The old page.evaluate callback returned a boolean; the new one returns { isSwiftShader, renderer }. The wrapper extracts .isSwiftShader. The only call site (frameCapture.ts line 522) is migrated to the new detectGpuBackend directly. Clean backward-compatible split.

The pre-existing as string cast on gl.getParameter(ext.UNMASKED_RENDERER_WEBGL) is carried forward (WebGL's getParameter returns any). Not new to this PR.

No issues here.

classifyGpuRenderer design

Low-cardinality bucket (<backend>/<vendor>) is the right abstraction for telemetry. The priority-ordered matching (swiftshader before metal before d3d11 before vulkan before opengl/angle) handles overlapping substrings correctly -- e.g. "ANGLE (Apple, ANGLE Metal Renderer..." hits metal before the catch-all angle check. The cardinality test (two different NVIDIA cards collapse to one bucket) is a nice explicit proof that the design goal holds.

The "other" fallback for both backend and vendor means unknown combinations still emit a bucket rather than crashing the telemetry pipeline. Undefined input returns undefined (not a bogus bucket). Clean.

No issues here.

gpu_renderer telemetry threading

I traced the full chain:

detectGpuBackend(page)                    -- raw renderer string
  -> classifyGpuRenderer(renderer)        -- "<backend>/<vendor>" bucket
  -> session.gpuRenderer                  -- CaptureSession field
  -> getCapturePerfSummary().gpuRenderer   -- CapturePerfSummary
  -> aggregateDrawElement().gpuRenderer    -- RenderPerfSummary (|-joined, Set-deduped)
  -> trackRenderMetrics: gpuRenderer       -- props to trackRenderComplete
  -> trackRenderComplete: gpu_renderer     -- telemetry event key

Parallel path (error resilience):
  -> probeSession?.gpuRenderer             -- live observability at pipeline start
  -> RenderCaptureObservability.deGpuRenderer
  -> captureDeGpuRenderer                  -- renderObservabilityTelemetryPayload
  -> renderObservabilityEventProperties    -- gpu_renderer in spread
  -> trackRenderError: gpu_renderer        -- via spread (only source on error path)

No gaps in the chain. The error path comment in trackRenderError explicitly documents that gpu_renderer arrives via the spread -- good.

Issue: gpu_renderer shadowed in trackRenderComplete

In trackRenderComplete, gpu_renderer is set twice:

1. Explicitly: gpu_renderer: props.gpuRenderer (from perf summary, the aggregated success-path value)
2. Via spread: ...renderObservabilityEventProperties(props) which includes gpu_renderer: props.captureDeGpuRenderer (from live observability, set at pipeline start)

The spread comes after the explicit property in the object literal, so it overwrites. The explicit gpu_renderer: props.gpuRenderer is dead code -- the observability value always wins.

On the success path both values should be identical (same host, same GPU), so this does not corrupt data. But it is misleading: a reader would expect the aggregated perf-summary value to be emitted, while in reality the pipeline-start observability value is what ships. If they ever diverge (e.g. multi-worker heterogeneous GPU in some future scenario), the wrong value would win silently.

Suggestion: Remove the explicit gpu_renderer: props.gpuRenderer from the trackRenderComplete event body (the spread already covers it), or add a comment noting the intentional override order. The trackRenderError path already has such a comment and handles this correctly by not adding a duplicate explicit property.

Severity: nit (no data impact today, but the dead code is confusing).

Standards

• No bare as T casts introduced by this PR (the as string on gl.getParameter and as WebGLRenderingContext | null on getContext are pre-existing patterns carried forward from the old detectSwiftShader).
• No ! non-null assertions.
• TypeScript types are clean: GpuBackendInfo interface, proper string | null | undefined handling in classifyGpuRenderer.
• Conventional commit format: feat(engine): -- correct.

Code quality

• isDrawElementPlatform is private, resolveDefaultDrawElement is exported. Good layering.
• The win32 compound test helper (expectCompoundOutcome) eliminates real duplication without losing coverage. Each test case is now a clean declarative object.
• The resolveDefaultDrawElement test suite covers all branches. Minor gap: no explicit test for useDrawElement: false as input, but this is a trivial early-return (if (!args.useDrawElement) return false) that the integration tests implicitly cover.
• Comments are thorough and explain the why (telemetry cardinality, safety contract parity, kill switches).

CI

All 47 check runs completed: 40 success, 7 skipped (expected -- skills, codex plugin, perf shards, GCP BeginFrame). No failures. Windows-specific checks (Render on windows-latest, Tests on windows-latest) both pass -- critical for this PR.

Verdict

Approve. The platform widening is mechanically sound, the safety contract (compile/init gates + worker-encode self-verify + screenshot fallback) is unchanged, and gpu_renderer telemetry gives the D3D11 cohort the attribution the rollout needs. The one dead-code nit (shadowed gpu_renderer in trackRenderComplete) is worth cleaning up but not blocking.

-- Miga

[miguel-heygen]

Exact-head review at 4520cd240b9a:

Strengths

• packages/engine/src/config.ts:731-742 makes the platform/GPU/worker-encode decision a pure boundary, and config.test.ts pins the intentional win32 widening plus every safety clamp.
• packages/engine/src/services/frameCapture.ts:717-720 reuses the existing single WebGL probe, retains only a bounded <backend>/<vendor> bucket, and preserves the SwiftShader fallback.
• The telemetry chain is complete on both outcomes: session → perf aggregation → render_complete, and live capture observability → render_error. packages/cli/src/telemetry/events.ts:253-287 spreads the observability fallback first, then writes the authoritative success-path gpuRenderer, so Miga's shadowing nit does not apply to this head.

The default still requires worker encode/self-verification, excludes Linux and software GPU, and preserves explicit opt-in/kill-switch behavior. All current checks are complete and green, including Windows render/tests, typecheck, producer tests, CodeQL, and all regression shards.

Verdict: APPROVE
Reasoning: The Windows expansion is isolated to the validated hardware-GPU envelope, the safety/fallback contract is unchanged, and backend telemetry reaches both success and failure events without raw high-cardinality renderer data.

— Magi

[james-russo-rames-d-jusso]

Reviewed at 4520cd24.

Clamp widens at packages/engine/src/config.ts:718-720 — platform === "darwin" || platform === "win32". Hardware-vs-software discriminator is args.browserGpuMode === "software" in resolveDefaultDrawElement at config.ts:731-742. Software-renderer allowlist (SwiftShader, LLVMpipe, Lavapipe, Softpipe, Mesa Offscreen, microsoft basic render driver, software rasterizer) at browserManager.ts:38-49 catches WARP, called from resolveBrowserGpuMode("auto") at browserManager.ts:504-517.

gpu_renderer bucketing at drawElementService.ts:135-163 (classifyGpuRenderer) — closed enum: 7 backends × 6 vendors = 42 buckets. No PII vectors, driver strings discarded post-classification. Type contract explicitly warns callers at drawElementService.ts:115-117 ("LOCAL USE ONLY … must not be shipped to telemetry verbatim") — verified .gpuRenderer never propagates further than the bucket.

Emitted on both events per [[feedback_ab_experiment_instrumentation_parity_precision]]: render_complete via events.ts:287 from perfSummary.drawElement.gpuRenderer, render_error via events.ts:112,386 from RenderCaptureObservability.deGpuRenderer (set at renderOrchestrator.ts:2651). No cross-repo consumers found in hyperframes-internal/EF/heygen-cli/pacific — this is a HF-only telemetry field.

Blockers

None.

Concerns

1. WARP escape hatch when PRODUCER_BROWSER_GPU_MODE=hardware is explicit. browserManager.ts:504-517 catches WARP (microsoft basic render driver), BUT only inside resolveBrowserGpuMode("auto"). When a user sets PRODUCER_BROWSER_GPU_MODE=hardware (explicit opt-in), the probe is skipped at browserManager.ts:512. The DE session's own gate at drawElementService.ts:173-185 (detectGpuBackend) only checks the substring "swiftshader" — WARP passes. classifyGpuRenderer then buckets it as d3d11/microsoft (not as software), and resolveDrawElementCaptureMode(false, transparent) returns "drawelement". DE engages on a CPU renderer.

Before this PR this was moot (Windows was clamped out entirely). Post-PR: a Windows user with a broken/missing GPU driver who has explicitly opted into hardware gets DE running on WARP with no perf gain — the PR body's GPU→CPU readback IPC savings claim doesn't hold on a CPU renderer, and there's no fallback trigger. Two options: (a) add WARP as a software-renderer substring to the detectGpuBackend/classifyGpuRenderer check so DE clamps down on WARP too, or (b) document that PRODUCER_BROWSER_GPU_MODE=hardware on Windows requires a hardware GPU and skip DE if the probe surfaces microsoft basic render driver. Not a blocker because the escape hatch exists on darwin too (software SwiftShader on darwin bypasses the auto-detect gate in the same way), but darwin doesn't have a WARP analogue with the same probability of being hit in the wild.

Nits

• N1 gpu_renderer shape asymmetry between events: render_complete (events.ts:287) emits a |-joined enum aggregating parallel workers (per perfSummary.ts:114-127), render_error (events.ts:112,386) emits a single bucket from the probe session. Analytics consumers must handle "d3d11/nvidia|d3d11/intel" on the success side but not on the error side. Documented but a foot-gun for GROUP BY queries.
• N2 Coverage gap on render_error gpu_renderer at renderOrchestrator.ts:2651: deGpuRenderer: probeSession?.gpuRenderer populates only after probe init reaches detectGpuBackend. A Chrome-launch-time failure (missing binary, permission denial, port collision) gives no gpu_renderer attribution — contradicting the comment at events.ts:382-385 that promises "MOST here". Not a regression from any prior state, but the failure surface the PR body attributes to gpu_renderer doesn't cover the pre-probe cohort.
• N3 ~206k renders/month claim in config.ts:904-906 is not traceable to any linked dashboard/query/artifact in the repo. PR-body-only. Not required by anything downstream, but consider linking the source PostHog query so future revisit can decompose the number.
• N4 config.test.ts:236-268 — no explicit darwin+software=off assertion; only win32+software=off is asserted directly. Coverage is by-implication via shared code path, but a direct assertion would prevent silent divergence.
• N5 drawElementService.test.ts:52-86 — no test case for Microsoft Basic Render Driver string in classifyGpuRenderer. Ties to Concern-1 above. If you add WARP as a software fingerprint, add a case with the raw D3D-adapter string that Chrome surfaces on a WARP fallback.

Questions

• Is there a Chrome-side flag that could FORCE the hardware GPU on Windows and fail loudly if no hardware GPU is available (instead of silently falling back to WARP)? That would be a cleaner discriminator than probing after the fact.
• On darwin, the exact same escape hatch exists (explicit hardware mode skips the probe; if Chrome fell back to SwiftShader for some reason, DE engages on CPU). Has this ever fired in prod on darwin, and if not, is the win32 concern hypothetical?

What I didn't verify

• Windows Chrome behavior when the hardware GPU is present but the D3D11 adapter probe fails — whether Chrome shortlists WARP as a first-class renderer or exits.
• The raw D3D adapter string Chrome writes on Windows hardware GPU renders — the bucket names (d3d11/microsoft, d3d11/nvidia, etc.) look right against the ANGLE conventions but no committed sample confirms.

— Review by Rames D Jusso