refactor: delete the legacy generation, tier the barrel, split the god files, unify verdicts - #581
Merged
Conversation
…odules Delete src/executor.ts, src/benchmark.ts, and src/reporter.ts. No consumer repo imports their exports; the campaign engine owns this job. Trim src/types.ts to the members that still have importers. Delete the no-op normalizeScores from src/statistics.ts. Delete 27 root modules with zero importers in 19 consumer repos: behavior-dsl, bisector, causal-attribution, ci-gate, clustered-paired-binary, cross-trace-diff, description-length-gate, flow-layer, harness-optimizer, live-proof, muffled-gate-scanner, observability, orthogonality, otel-pipeline, paraphrase, playbook, propose-review-control, pr-review-benchmark, reference-replay-steering, release-report, replay, reward-model-export, run-evidence, runtime-trajectory, sandbox-pool, self-play, state-continuity, traced-analyst, traced-judges, visual-diff. Remove the dead ./storyboard subpath and its directory. Remove the dead ./control subpath entry; its modules stay on the root barrel. Repin the analyst benchmark digests; package.json and statistics.ts are in the pinned set.
…and documented front doors The root barrel re-exports only: symbols external consumer repos import from '@tangle-network/agent-eval', the documented front doors (defineAgentEval, selfImprove, analyzeRuns, analyzeTraces, runCampaign, llmJudge, ensembleJudge, RunRecord, trace stores, ChatClient, statistics, CostLedger), and the types those symbols reference. Everything else stays reachable through its subpath only. The barrel is grouped into six commented sections: contract, formats, engine, analysis, verification, utilities. Star re-exports and the benchmarks namespace are gone; every export is named. The profile namespace stays (16 consumer files import it from the root). Surface count: 1,220 export entries (plus ~200 names hidden behind three star re-exports) down to 674 named entries. Front doors defineAgentEval, selfImprove, analyzeRuns, and runCampaign gain root exports with their option and result types. The four pipeline views (budgetBreachView, failureClusterView, judgeAgreementView, toolWasteView) gain root exports: legal-agent and insurance-agent scripts import them from the root specifier. Internal tests that pinned root parity now pin the module or subpath surface. The consumer-contract pin list is regenerated from a 2026-08-11 scan of 12 consumer repos. verify-package-exports exercises the tiered root plus subpaths. Docs: control-runtime and feature-guide stop naming runProposeReviewAsControlLoop, which no longer exists.
# Conflicts: # src/analyst/benchmark-implementation.ts
…ternal seams src/statistics.ts (2453 lines, 71 exports) becomes src/statistics/ with ten focused modules, one shared internal-helper module, and an index that re-exports the identical surface. src/judge-calibration.ts now imports ./statistics/descriptive directly, which removes the statistics<->judge-calibration file cycle. The statistics tests move beside their modules; the normalCdf, Student-t, and Welch describes move to tests/normal-cdf, tests/student-t, and tests/baseline. src/campaign/run-campaign.ts (1389 -> 377 lines) keeps runCampaign, its options type, and the same export surface. The phases move to sibling modules: cell-schedule, campaign-manifest, cell-cache, execute-cell, judge-cell, cell-aggregates, and plan-campaign-run. src/run-record.ts stays whole. The schema and its validator are one contract, and a field change must touch both in the same diff. src/analyst: the row-decode loop in prime-protocol.ts now calls the shared decodeContractRows from reply-contract.ts. The benchmark implementation manifest and digest are repinned for the new statistics module paths, in both benchmark-implementation.ts and its test mirror. Also fixes two pre-existing lint findings in tests/trace-repair/fixtures.ts and tests/wire/rpc.test.ts.
…uced certification Every pass/fail-with-score path now returns the shared verdict spine, and verdict.certification is produced, not just typed. The multi-layer verifier certifies as composite. The completion verifier carries its checker's own attestation (judge for the LLM checker, schema for token recall). Trace contracts certify as invariant. Declarative oracles certify as test. Trajectory replay and finding verification certify as replication from the pinned image. The repair grader certifies as test, pinned to the row's suite and policy digests, and only on a measured outcome — a closed gate certifies nothing. The blind statement-equivalence protocol merges into src/verification-strategy.ts: it is an instance of the StrategyChecker port, not a second abstraction. equivalenceVerdict lands a finished record in the spine. packageVersion() moves out of wire/handlers so checker identities can cite the exact package build. The root barrel re-exports the vocabulary and the oracles as the documented front door. docs/verdicts.md states the vocabulary; verifier docs cross-link it; the charter records the certifications as produced.
…imports The packed tarball is the proof artifact: agent-runtime and traces at origin/main typecheck against it via a pnpm override. agent-runtime is clean. traces shows one error, and the control run against a tarball packed from origin/main shows the identical error — pre-existing drift, not this branch. A per-symbol import scan of the 11 census repos at their origin default branches, plus agent-builder, finds zero missing root imports after this commit. The scan compares against a baseline build of origin/main, so pre-existing breaks do not count against the branch. Restored to the root barrel because a consumer imports them today: - traces: runCounterfactual and its context, mutation, result, runner types. - discovery-lab: runProposeReviewAsControlLoop (module restored), createLlmReviewer, jsonlReviewStore, makeProposalFinding, and the propose-review types their signatures reference. - agent-runtime: isToolSpan, OUTPUT_VALUE. - agent-dev-container: ErrorCluster, knowledgeReadinessTracePayload, pairRunRecords, projectRuntimeTrajectoryEvidence and the restored runtime-trajectory module. - creative-agent: the product-benchmark packaging surface, BudgetGuard, BudgetBreachError. - gtm-agent: pairedDeltaTest with its options and result types. - agent-builder: LlmClient, ActionableSideInfo, assignFeedbackSplit, steering types, trace span types, InMemoryRawProviderSink, SandboxDriver, Finding, RunStatus. - physim, braid, workcomp-agent, loops, browser-agent-driver, agent-knowledge, redteam: anti-slop, steering optimizer, chat-client options, bounded trace stores, run-record backend, paretoChart, preflightModels, KnowledgeBundle, reference-replay types. - run-capsule: the ./storyboard subpath and its module — the package declares itself a consumer of it. tests/consumer-contract.test.ts pins each restored symbol with the importing repo named, so the next tiering pass cannot drop them silently. The dependency-lock digest is repinned: package.json regains the ./storyboard exports entry. Test purge audit: nine scan classes over 370 test files found no test that fails the what-bug-would-this-catch question. Earlier waves already removed the 30 test files that covered deleted code. No test asserts the absence of a deleted thing.
tangletools
previously approved these changes
Aug 11, 2026
tangletools
left a comment
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 07a14750
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-11T21:59:13Z
tangletools
approved these changes
Aug 11, 2026
tangletools
left a comment
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — f5850b4b
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-11T22:38:55Z
tangletools
approved these changes
Aug 11, 2026
tangletools
left a comment
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — f5850b4b
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-11T22:39:11Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five waves on one branch: delete the gen-1 eval spine and every zero-importer module, tier the root barrel to what consumers import, split the two god files along their seams, land every verifier in
DefaultVerdict, and prove the result against real consumers with a packed tarball.The numbers (origin/main dc59202 → this branch)
./control)Full verification on the final tree:
pnpm typecheckclean,pnpm test5,190 passed / 3 env-gated skips,pnpm buildclean,pnpm verify:packageexit 0 (publint + attw + packed-tarball export walk), biome clean over 723 files, both analyst digest checks valid (implementation81709d08…, dependency lock repinned tob0d2e697…afterpackage.jsonregained the./storyboardentry).Per-wave commits
executor,benchmark,reporter) + 27 zero-importer root modules + dead subpathsstatistics.ts(2,453 lines → 10 modules) andrun-campaign.ts(1,389 → 377 + 7 phase modules), identical surfacesDefaultVerdictwith a produced certification; statement-equivalence merges intoverification-strategyThe law that guarded consumers
A symbol imported by any external consumer keeps its exact import specifier. Proof, not assertion:
pnpm pack+ pnpm override): agent-runtime at origin/main compiles clean (tsc + examples). traces at origin/main shows exactly one error — and the control run against a tarball packed from origin/main shows the identical error (DspyRlmTraceEngineOptions.apiKey, a 0.143→0.144 drift that predates this branch). Zero new breaks.import {…} from '@tangle-network/agent-eval[/…]'across a repo, checked against the built d.ts of both origin/main and this branch): the 11 census repos at their origin default branches + agent-builder = 0 regressions. The baseline comparison means pre-existing breaks never count against the branch.runCounterfactual(traces, committed to traces main the same day as the census),runProposeReviewAsControlLoop(discovery-labtools/propose-v9.mjs, a.mjsfile the census's TS-only scan missed),isToolSpan/OUTPUT_VALUE(agent-runtime supervise surface), the product-benchmark packaging surface (creative-agent),pairedDeltaTest(gtm-agent), runtime-trajectory +ErrorCluster+knowledgeReadinessTracePayload(agent-dev-container), 13 symbols for agent-builder, and the./storyboardsubpath for run-capsule, whose package description names itself a consumer of it. Each restored symbol is now pinned intests/consumer-contract.test.tswith the importing repo named.Test purge (wave 5 charter)
Nine scan classes over all 370 test files: snapshot assertions (0 found), long exact-prose equality (1 — behavioral, kept), long
toThrowprose (2 — fail-loud contracts, kept), absence-of-deleted-thing assertions (0 — none added by any wave), export-count pins (only the consumer contract, which is the law), directory-listing counts (fixture-driven, kept), pinned hex digests (recomputed not stale, kept), vacuous assertions (0), permanently-skipped tests (0 — all 5 skips are env/platform-gated). Deleted this wave: 0. The 30 test files covering deleted code left with their code in waves 1–2. Case count still fell 5,396 → 5,193 from those waves; wave 5 added back the 2 storyboard test files (real behavior tests for a consumed subpath) and 2 new consumer-contract pins.Deliberately spared
runEvalCampaignand the rooteval-campaignrunner — imported by consumers andrl/rl-campaign.profilenamespace export — 16 consumer files import it from the root.ProductClient/AgentDriver— still consumed; their warn-once deprecation contract is tested.experiment-tracker— consumer-facing stats verdicts still imported from the root.src/analyst— deliberate, byte-identity is test-pinned.propose-review-control,runtime-trajectory,src/storyboard/— deleted as zero-importer by wave 1, but living repos outside the census's file-type or repo scope import them.Not restored, on the record
Two repos still reference deleted symbols: starter-foundry (83 import records, pins
^0.99.0, 45 minors behind, already had broken imports at baseline) and phony (75, personal Webb project outside the Tangle fleet). Both were unable to bump cleanly before this branch; restoring their surfaces would resurrect the gen-1 trainer scaffolding for consumers that cannot use current releases anyway.Scope note: the 11 census repos + agent-builder were verified at origin default branches; the remaining fleet sweep used local checkouts and is advisory.