Build pipeline for agent SDK tarballs#320853
Closed
TylerLeonhardt wants to merge 3 commits into
Closed
Conversation
Adds the build-side tooling that packages, verifies, and uploads the Claude and Codex SDK tarballs that the runtime (PR #320709) downloads from main.vscode-cdn.net. build/agent-sdk/: - common.ts — shared helpers. getSdkVersion(sdk) reads the pinned exact version from the repo-root package.json devDeps; getTargets(sdk) reads the supported platform list from the SDK's own optionalDependencies — both are live reads, no parallel constants to keep in sync. - package.ts — builds one (sdk, target) tarball: npm install with npm_config_libc/os/cpu set, chmod platform binaries, normalize mtimes (lutimes for symlinks), tar+gzip with reproducible flags. Writes a JSON sidecar carrying {sdk, sdkVersion, sdkTarget, sha256}. - upload.ts — uploads tarball to vscodeweb's $web container at agent-sdk/<sdk>/<version>/<target>.tgz. HEAD-then-decide: absent → upload; matching sha → skip (idempotent); different sha → fail loud, refusing to overwrite content-addressed history. - aggregate.ts — scans all per-job sidecars, validates every expected (sdk, target) is present, prints the product.agentSdks JSON fragment for a human to paste into vscode-distro's product.json. build/azure-pipelines/agent-sdk/ + product-build.yml: - New AgentSDK stage that fans out 14 jobs (8 Claude + 6 Codex targets) via each-loop over claudeTargets/codexTargets parameter arrays. Aggregate job uses condition: succeededOrFailed() so the operator sees what's missing even when a per-target job fails. Also pins @openai/codex devDep to an exact version (no ^ range) — getSdkVersion rejects ranged specifiers because they'd let npm resolve a different version every day, breaking determinism.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new build-side pipeline + tooling to produce and (optionally) upload per-platform agent SDK tarballs for Claude and Codex, intended to feed the runtime-side SDK downloader via main.vscode-cdn.net and a product.agentSdks fragment.
Changes:
- Pin
@openai/codexto an exact devDependency version to support deterministic/content-addressed publishing. - Introduce
build/agent-sdk/*scripts to package, upload, and aggregate SDK tarballs + sha256 sidecars. - Add a new
AgentSDKstage to the scheduled product build that fans out per-target jobs and runs an aggregate job.
Show a summary per file
| File | Description |
|---|---|
| src/vs/platform/agentHost/node/claude/roadmap.md | Updates roadmap text to describe upload-time determinism enforcement. |
| package.json | Pins @openai/codex devDependency to an exact version. |
| package-lock.json | Aligns lockfile devDependency entry for @openai/codex to the exact version. |
| build/azure-pipelines/product-build.yml | Adds the new AgentSDK stage to the product build pipeline. |
| build/azure-pipelines/agent-sdk/product-build-agent-sdk.yml | Defines the fan-out matrix + aggregate job for SDK builds. |
| build/azure-pipelines/agent-sdk/product-build-agent-sdk-target.yml | Defines the per-(sdk,target) job that builds and conditionally uploads one tarball. |
| build/agent-sdk/README.md | Documents the new build/agent-sdk tooling and workflow. |
| build/agent-sdk/common.ts | Shared helpers (version/target discovery, sidecar I/O, hashing). |
| build/agent-sdk/package.ts | Produces a reproducible per-target SDK tarball + .tgz.json sidecar. |
| build/agent-sdk/upload.ts | Uploads tarball to Azure $web with HEAD-then-skip/fail idempotency guard. |
| build/agent-sdk/aggregate.ts | Validates completeness and prints a product.agentSdks JSON fragment. |
Copilot's findings
- Files reviewed: 10/11 changed files
- Comments generated: 11
Comment on lines
+11
to
+15
| Stages a minimal `package.json` pinning `SDK_VERSIONS[sdk]` (from | ||
| `common.ts`), runs `npm install --no-package-lock --ignore-scripts` with | ||
| `npm_config_libc/os/cpu` set to the target's triple to fetch the foreign | ||
| platform binary, normalizes mtimes (including symlinks via `lutimes`), and | ||
| tars with reproducible flags. Emits `<sdk>-<version>-<target>.tgz` + a |
| `npm_config_libc/os/cpu` set to the target's triple to fetch the foreign | ||
| platform binary, normalizes mtimes (including symlinks via `lutimes`), and | ||
| tars with reproducible flags. Emits `<sdk>-<version>-<target>.tgz` + a | ||
| `.tgz.json` sidecar carrying the sha256 and size. |
Comment on lines
+26
to
+27
| sidecars (one per per-target job's artifact) and prints a markdown table | ||
| with the JSON fragment a human pastes into `vscode-distro`'s `product.json`. |
Comment on lines
+12
to
+16
| * Where `<sdkTarget>` matches the npm `optionalDependencies` suffix the SDK | ||
| * ships its platform package under (e.g. `darwin-arm64`, `linux-x64-musl`, | ||
| * `win32-x64`). The supported set per SDK is `TARGETS` in `common.ts`; the | ||
| * pinned version is read from the repo-root `package.json` devDeps (via | ||
| * `getSdkVersion` in `common.ts`). |
Comment on lines
+24
to
+27
| * - SDK pinned to an exact version by `SDK_VERSIONS`. npm install fetches | ||
| * transitive deps fresh each run; day-to-day drift in those resolutions | ||
| * surfaces at upload time as a sha mismatch against the existing blob, | ||
| * where a human investigates whether it's an intentional SDK bump. |
Comment on lines
+11
to
+15
| import * as crypto from 'crypto'; | ||
| import * as fs from 'fs'; | ||
| import * as path from 'path'; | ||
| import { fileURLToPath } from 'url'; | ||
|
|
Comment on lines
+77
to
+94
| export function getTargets(sdk: Sdk): readonly string[] { | ||
| const thisDir = path.dirname(fileURLToPath(import.meta.url)); | ||
| const sdkPackageJson = path.resolve(thisDir, '..', '..', 'node_modules', PACKAGE_NAME[sdk], 'package.json'); | ||
| const json = JSON.parse(fs.readFileSync(sdkPackageJson, 'utf8')) as { | ||
| optionalDependencies?: Record<string, string>; | ||
| }; | ||
| const prefix = `${PACKAGE_NAME[sdk]}-`; | ||
| const targets: string[] = []; | ||
| for (const name of Object.keys(json.optionalDependencies ?? {})) { | ||
| if (name.startsWith(prefix)) { | ||
| targets.push(name.slice(prefix.length)); | ||
| } | ||
| } | ||
| if (targets.length === 0) { | ||
| throw new Error(`No platform packages found in ${sdkPackageJson}'s optionalDependencies (expected entries starting with '${prefix}')`); | ||
| } | ||
| return targets.sort(); | ||
| } |
Comment on lines
+73
to
+75
| * Requires the repo-root `npm install` to have run (the SDKs are devDeps | ||
| * in `package.json`). All pipeline jobs run after the standard install | ||
| * step so `node_modules/<sdk-package>/` exists for them to read. |
Comment on lines
+1
to
+4
| # Single (sdk, target) build + upload job. Invoked per matrix entry from | ||
| # product-build-agent-sdk.yml. Builds twice (determinism check), keeps the | ||
| # second build's tarball+sidecar as the publishable artifact, then uploads | ||
| # (gated on VSCODE_PUBLISH). |
Comment on lines
+56
to
+58
| # FAIL LOUD listing the missing targets (aggregate.ts validates completeness | ||
| # against the TARGETS constant in common.ts). Without this condition, Azure's | ||
| # default `succeeded()` silently skips aggregate on any failure — leaving the |
The pipeline jobs don't run a full repo npm install, so reading @anthropic-ai/claude-agent-sdk/package.json from node_modules/ fails with ENOENT. Switch getTargets() to call 'npm view <pkg>@<version> optionalDependencies --json' instead — works against the registry without needing the package installed locally. The private npm proxy the pipeline already uses handles the lookup transparently. Also drop the getTargets() call from parseTarget() — the pipeline matrix controls what targets get invoked in production, and an invalid target surfaces downstream when npm install can't resolve the platform package. Avoids a network round-trip per CLI parse.
The pipeline's claudeTargets/codexTargets arrays now drive both the matrix fan-out AND aggregate's completeness check (passed in as --claude-targets/--codex-targets CLI flags). One source of truth for which platforms we build; aggregate.ts has no parallel list to drift from. Add warn-only drift detection: aggregate queries each SDK's registry optionalDependencies and emits ##vso[task.logissue type=warning] when upstream declares a platform we don't build (or, separately, when we build one upstream has dropped). Failing here would block routine pipeline runs every time an SDK publishes a new platform overnight, so just shout — the operator sees the warning in the build UI. Renames getTargets() → getRegistryTargets() to make its purpose clear.
Member
Author
|
Superseded by the runtime simplification in #320883. The replacement build PR inlines SDK production into the per-platform packageTask in the existing gulpfiles (no separate AgentSDK stage, no aggregate). New PR coming shortly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build pipeline that produces the SDK tarballs the runtime side (#320709) downloads from
main.vscode-cdn.net.What this adds
build/agent-sdk/Four self-contained scripts:
package.ts— builds one(sdk, target)tarball.npm installwithnpm_config_libc/os/cpuset to fetch the foreign-platform binary, chmod platform executables, normalize mtimes (lutimesso symlinks like.bin/codexget their stamp too), tar+gzip with reproducible flags (--format=gnu --sort=name --owner=0 ... --mtime=@0 -n -9). Writes<sdk>-<version>-<target>.tgz+ a.tgz.jsonsidecar carrying{sdk, sdkVersion, sdkTarget, sha256}.upload.ts— uploads one tarball to thevscodewebstorage account's$webcontainer atagent-sdk/<sdk>/<version>/<target>.tgz. HEAD-then-decide: absent → upload; matchingmetadata.sha256→ skip (idempotent re-runs are free); different sha → fail loud, refusing to overwrite content-addressed history. Reuses the establishedClientAssertionCredentialauth pattern frombuild/azure-pipelines/upload-cdn.ts.aggregate.ts— scans all per-job sidecars, validates every expected(sdk, target)pair is present (reads each SDK's ownoptionalDependenciesfor the expected list — no parallel list to keep in sync), and prints theproduct.agentSdksJSON fragment for a human to paste intovscode-distro'sproduct.json.common.ts— shared types and helpers. The two interesting bits:getSdkVersion(sdk)reads the pinned exact version from the repo-rootpackage.jsondevDeps. Rejects ranged versions (^,~) — they'd let npm resolve different versions across days, breaking determinism.getTargets(sdk)reads the supported platform list from the SDK package's ownoptionalDependencies. The SDK declares which platforms it ships; we ask it directly instead of duplicating the list.build/azure-pipelines/agent-sdk/+product-build.ymlA new
AgentSDKstage in the existing scheduled product build, fanning out 14 jobs:product-build-agent-sdk-target.yml) — one per(sdk, target)pair, each invoked from aneachloop overclaudeTargets/codexTargetsparameter arrays. Builds the tarball, publishes it as a pipeline artifact, and (whenVSCODE_PUBLISH=true) uploads to CDN.product-build-agent-sdk.yml) — downloads every per-target artifact and runsaggregate.ts. Usescondition: succeededOrFailed()deliberately: when one or more per-target jobs fail, the operator still sees a loud "Missing sidecars for N expected pairs" message naming each missing target, instead of having to click through 14 sibling jobs.How a SDK bump now works
package.json(@anthropic-ai/claude-agent-sdkor@openai/codex) to the new exact version.npm install, commit lockfile.vscode-distro'sproduct.jsonunderagentSdks.No lockfiles to regenerate, no parallel "supported targets" list to update, no
SDK_VERSIONSconstant to bump.Determinism
The build is byte-reproducible for a given Linux runner image + Node version through:
optionalDependenciesto exact versions, so the platform binary fetched is deterministic).lutimes(notutimes) for symlinks so things likenode_modules/.bin/codexget normalized too.Cross-run drift (agent image bump changes gzip output, registry-side transitive resolution shifts, etc.) is caught at upload time by the
upload.tsHEAD-then-fail check — that's the actual defense, and it catches across-day drift that a "build twice in a row" check never could. We deliberately don't have averify-determinism.tsscript anymore for that reason.Decisions worth knowing
vscodewebstorage account. Same account as the existing web bundle uploads, just under theagent-sdk/prefix. No new infra to provision, no new credentials.package.jsondevDeps. Targets → SDK'soptionalDependencies. SDK package name →PACKAGE_NAMEconstant (unavoidable — defines what to look up).Risks / things to verify
replace()template expression in YAML — used to convertdarwin-arm64→darwin_arm64for Azure Pipelines job names (which disallow hyphens). Documented in Azure's template-expression reference but no other prior art in this repo. If it doesn't expand, the fallback is one extra layer: pass target arrays as{slug, target}objects.metadatapermissions —BlobServiceClient.uploadFilewithmetadatamay requireStorage Blob Data Ownerrather thanContributorrole on the storage account. We'll find out on first upload attempt.Test plan
VSCODE_PUBLISH=false. Expect 14 per-target jobs green + aggregate job prints the fragment to its log (without uploading anything).VSCODE_PUBLISH=trueonce the dry run is green. Verify blobs land athttps://main.vscode-cdn.net/agent-sdk/<sdk>/<version>/<target>.tgzand HEAD-then-fail idempotency works on a re-run.IProductConfiguration.agentSdksshape fromsrc/vs/base/common/product.ts.🤖 Generated with Claude Code