Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) - #3048
Draft
TooTallNate wants to merge 13 commits into
Draft
Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs)#3048TooTallNate wants to merge 13 commits into
TooTallNate wants to merge 13 commits into
Conversation
…-safe requeue, stable PRNG seed
🦋 Changeset detectedLatest commit: 7ccf0f0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an experimental QuickJS WASM-based workflow VM engine to @workflow/core, selectable via WORKFLOW_VM=quickjs (or per-run affinity via executionContext.workflowVm), while keeping the existing node:vm replay engine as default. This expands the runtime’s portability (WASM-only platforms) and lays groundwork for later snapshotting work, while maintaining the same full event-replay semantics.
Changes:
- Introduces QuickJS runtime + entrypoint implementing full replay, pending-op extraction, and durable side-effect dispatch/queueing.
- Adds VM-compatible serialization (devalue reducers/revivers + bundled serde IIFE) and QuickJS asset embedding (WASM + extensions).
- Extends CI/test matrix and docs to cover the new
WORKFLOW_VMoption.
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/create-test-matrix.mjs | Adds an e2e leg that runs nextjs-turbopack with WORKFLOW_VM=quickjs. |
| pnpm-workspace.yaml | Excludes quickjs-wasi from minimum release age checks. |
| pnpm-lock.yaml | Locks quickjs-wasi@3.1.0. |
| packages/core/turbo.json | Adds generated QuickJS/serde artifacts to Turbo build outputs. |
| packages/core/src/telemetry/semantic-conventions.ts | Adds semantic convention keys for VM engine + QuickJS runtime stats. |
| packages/core/src/source-map.ts | Adds stripInlineSourceMap() to reduce QuickJS heap usage. |
| packages/core/src/source-map.test.ts | Adds unit tests for inline source map stripping. |
| packages/core/src/serialization/workflow-vm.ts | Adds VM-safe workflow-mode (de)serialization (no Node deps). |
| packages/core/src/serialization/workflow-vm.test.ts | Validates VM serializer round-trips + Node/VM compatibility. |
| packages/core/src/serialization/vm-bundle-entry.ts | Entry point for esbuild’d VM serde bundle (installs serialize/deserialize + ULID generator). |
| packages/core/src/serialization/reducers/common-vm.ts | VM-safe reducers/revivers (base64 via atob/btoa, error subclass support, Web API-ish objects). |
| packages/core/src/serialization/compat.test.ts | Adds compatibility coverage between new modules and legacy serialization pipeline. |
| packages/core/src/serialization/codec-devalue-vm.ts | VM-compatible devalue codec, including AbortController/Signal handling for workflow VM context. |
| packages/core/src/runtime/vm-mode.ts | Adds WORKFLOW_VM parsing + run-affinity selection logic. |
| packages/core/src/runtime/vm-mode.test.ts | Unit tests for VM mode selection precedence and validation. |
| packages/core/src/runtime/start.ts | Stamps executionContext.workflowVm at run start when WORKFLOW_VM is set. |
| packages/core/src/runtime/quickjs-runtime.ts | Implements QuickJS VM execution + replay processing + pending-op model. |
| packages/core/src/runtime/quickjs-runtime.test.ts | Adds unit tests for replay semantics, determinism, clock, abort, etc. |
| packages/core/src/runtime/quickjs-entrypoint.ts | Implements host-side entrypoint: fetch events, run VM, create events/queue work, finalize runs. |
| packages/core/src/runtime.ts | Dispatches to QuickJS engine when selected, including turbo-safe reinvocation behavior. |
| packages/core/scripts/build-vm-serde-bundle.js | New build step to generate bundled serde string module for VM bootstrap. |
| packages/core/scripts/build-quickjs-assets.js | New build step to embed QuickJS WASM + extensions into TS for runtime import. |
| packages/core/package.json | Adds build steps + quickjs-wasi dependency. |
| packages/core/.gitignore | Ignores generated QuickJS asset + serde bundle TS outputs. |
| docs/content/docs/v5/configuration/runtime-tuning.mdx | Documents WORKFLOW_VM configuration and behavior. |
| .github/workflows/tests.yml | Adds vitest-plugin matrix for WORKFLOW_VM and plumbs WORKFLOW_VM into e2e jobs. |
| .changeset/quickjs-vm-engine.md | Adds changeset for @workflow/core + workflow minor bump. |
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
TooTallNate
marked this pull request as draft
July 22, 2026 08:47
…les; harden step-listing e2e assertions against eventually-consistent reads
…l column can lag terminal status)
… assertions (analytics listing can omit attempt entirely)
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental QuickJS WASM VM engine for workflow execution, opt-in via
WORKFLOW_VM=quickjs(or per-runexecutionContext.workflowVm). The engine performs the same full event replay as the defaultnode:vmengine, but runs workflow code in a QuickJS-NG VM compiled to WebAssembly viaquickjs-wasi@3.1.0.This is PR 1 of the incremental revival of the snapshot-runtime effort (#1300, RFC #1298): the QuickJS VM is deliberately decoupled from snapshotting. Motivations:
node:vm(e.g. Cloudflare Workers — WASM only)executionContextatstart(), so a run keeps executing on the engine it started onResults
WORKFLOW_VM=quickjsagainst the nextjs-turbopack workbench (local dev, world-local)How it works
WORKFLOW_VM=node|quickjs(runtime/vm-mode.ts), defaultnode. Per-run affinity viaexecutionContext.workflowVm, validated at startup.runtime/quickjs-runtime.ts: every invocation creates a fresh QuickJS VM, evaluates the workflow bundle, re-executes the workflow from the top, and resolves awaited primitives from the recorded event log (processEvents). Workflow primitives (useStep,sleep,createHook,setAttributes,AbortController) are implemented as VM-side bootstrap JS backed by a pending-op/resolver model.runtime/quickjs-entrypoint.ts: loads the full event log, runs the VM, and dispatches durable side effects for pending ops (step/hook/wait/attribute events + step queueing via the unified V2 queue) on suspension, plus a terminal drain (abort recordings, system-hook disposal, fire-and-forget writes) on completion/failure — mirroringdrainPendingQueueItems.vm-serde-bundle), with the WASM binary + native extensions (encoding, headers, url, structured-clone) base64-embedded so no filesystem access or file tracing is needed at runtime.Determinism model
Full replay requires every invocation of a run to regenerate identical correlationIds and observe identical time:
runId:workflowName:deploymentId(invocation-stable — notably NOTstartedAt, which differs between turbo's synthesized run object and the durably stored run), matching the node:vm engine. Identical ids across concurrent invocations are load-bearing for the world's per-(run, correlation) dedup.Date.now()/new Date()inside the VM read a host-controlled WASIclock_time_getoverride that starts at run creation and advances to each processed event'screatedAt— the same deterministic replay clock as the node:vm engine'supdateTimestamp.Feature parity work (beyond the original branch)
The WIP branch predated ~2.5 months of main. Ported into the VM model:
setAttributes(attr_set events + immediate re-invoke), hook-backedAbortController/AbortSignal(system hooks, abort stream packets, statics),hook.getConflict()+ conflict adoption (conflictingRunrevived through the VM class registry),HookConflictError/RuntimeDecryptionErrorserialization, byte-stream framing round-trip (webhook bodies), cross-run writable forwarding symbols, bound step proxies (.bindoverride preservingstepId/__boundThis), decrypt+decompress of specVersion≥5 payloads before VM handoff, and turbo-safe re-invocation viareinvoke().Scope / deliberately deferred
world.snapshotsinterface, no snapshot persistence, no compression/encryption of VM state — that is PR 3/4 (threshold-based snapshotting) per the roadmapWORKFLOW_VMset project-side); CI covers aquickjsaxis on the vitest-plugin job and a nextjs-turbopack leg across the local dev/prod/postgres e2e jobsquickjs-wasiadded tominimumReleaseAgeExclude(3.1.0 published same-day; vercel-labs package)Docs
Documented
WORKFLOW_VMin v5 Runtime Tuning (docs/content/docs/v5/configuration/runtime-tuning.mdx).