fix(insights): round 2 says the derived outcome is 52%, not 63% — Step 1 reopens (v0.329.1)

CHANGELOG.md

@@ -8,6 +8,24 @@ new version heading in the same commit.
 
 ## [Unreleased]
 
+## [0.329.1] — 2026-08-09
+### Changed
+- **Round 2 of the derived-outcome falsifier: the honest out-of-sample number is 52%, not 63%, and
+  Step 1 is reopened** (`docs/insights-revisit.md`). Round 1 tuned two rules on the rows it had just
+  scored, so its 63% measured the fitting. This round samples 32 conversations round 1 never touched
+  (`outcome-label-sample.cjs --exclude`), labelled blind, scored against the rules exactly as shipped in
+  v0.323.0: **52% exact against a 43% always-success baseline** (78% on sign — did work land or not).
+  Nine points over the trivial baseline is not enough to build Step 2 on. The 11 errors cluster into
+  three causes, one of which reverses a Step 1 decision: **5 of them are the `no-evidence` residual the
+  Stop-hook half was deferred over** — 6% of the corpus, but 45% of the mistakes, because those
+  conversations are concentrated in exactly the runs a human judges instantly and the OS cannot.
+  Measuring a residual by its share rather than by its share of the errors is the mistake; the hook is
+  back in scope. The other two: quota deaths with zero tool calls fall through `died-early`'s
+  `tool_calls > 0` guard and read as `noop` (3×), and `task-retried` beat a later success in one 4-run
+  conversation (a fold-order bug). No rule was changed in response — that is what round 3 is for.
+- `outcome-label-score.cjs` takes a labels-file argument; `outcome-label-sample.cjs` takes `--exclude` so
+  a re-validation can only draw rows an earlier round never saw.
+
 ## [0.329.0] — 2026-08-08
 ### Changed
 - **The status grammar reaches the last four surfaces** (Inbox, Approvals, Agents, Cockpit), finishing the

docs/insights-revisit.md

@@ -176,7 +176,7 @@ no longer derives a percentage from them, and states that outcome is self-report
   case is the fixture Step 1 must satisfy: two states differing **only** in how many runs reported,
   with identical real failures, must produce identical guidance.
 
-### Step 1 — an outcome that isn't self-graded ✅ shipped v0.323.0
+### Step 1 — an outcome that isn't self-graded 🟡 shipped v0.323.0, NOT closed (see round 2)
 
 `src/edge/outcome.ts`. Rules over facts the OS observed itself, ordered, each carrying the `basis` that
 decided it so any number traces back to its evidence. Live 30-day instapods corpus, 443 conversations:
@@ -211,8 +211,43 @@ continues a transcript — scoring rows counts one job several times), and **not
   After both, **63% exact against a 32% baseline** (19 judged, 9 declared unscorable, 7 unlabelable for
   lack of a transcript on this box).
 - **⚠ The 63% is not a clean number.** The labels were blind, but the rules were revised *after* seeing
-  which rows v1 got wrong, so it is partly fitted to 28 rows. **Before Step 2 leans on this, draw a fresh
-  blind sample against the current rules.** The honest, unfitted number is v1's 50%.
+  which rows v1 got wrong, so it is partly fitted to 28 rows. The honest, unfitted number is v1's 50%.
+
+#### Round 2 — the re-validation, and what it says (2026-08-09)
+
+32 conversations round 1 never touched (`--exclude`), stratified across all 9 bases, labelled blind,
+scored against the rules **exactly as shipped in v0.323.0** — nothing changed after seeing the result.
+
+| | exact | baseline | sign |
+|---|---|---|---|
+| round 1, v1 rules | 50% | 43% | — |
+| round 1, after tuning | 63% | 32% | 74% |
+| **round 2, out of sample** | **52%** | **43%** | **78%** |
+
+**The 63% was fitting.** True out-of-sample accuracy is ~52% — nine points above "call everything a
+success". Sign agreement (did work land, yes or no) holds up better at 78%, which is the number Step 2
+would actually lean on, but the verdict-level signal is weak.
+
+**This is not good enough to build Step 2 on.** The 11 disagreements cluster into three fixable causes,
+and one of them reverses a decision made in Step 1:
+
+1. **`no-evidence` → `unknown` on 5 conversations a human read in seconds** (4 successes, 1 partial) —
+   runs that did substantial work and left no observable trace. This is exactly the residual the
+   Stop-hook half was deferred over, and it is now the single largest error class. **That deferral was
+   wrong; the hook is the fix, and Step 1 is not finished without it.**
+2. **Quota deaths with zero tool calls read as `noop`** (3×). `died-early` requires `tool_calls > 0`, a
+   guard round 1 bought to stop an agent replying "looks like a test message" being called a failure.
+   Both shapes are an automation run, ~0 tools, a few seconds — the DB does not currently distinguish
+   "died on the first API call" from "answered and stopped". It needs a fact neither has today.
+3. **`task-retried` beat a later success** in a 4-run conversation (1×) — a fold-order bug, not a
+   sensing problem, and the cheapest of the three to fix.
+
+Two more disagreements were probably **my labels, not the rules**: I called two automation runs with no
+assistant output at all `noop`, where `died-early` called them failures. An unattended run that produces
+nothing is closer to a failure. Corrected, the exact number is ~57%.
+
+- **Verdict: Step 1 stays open.** Step 2 does not start until (1) is closed and a third round clears
+  60% out of sample on verdicts.
 - **Known coverage gap:** the OS declines to judge 134 of 443 conversations (30%) — every interactive
   human session. Fine for Steps 2–5, which are about unattended work; it would not be fine for a
   fleet-wide "how are we doing" claim.
@@ -261,6 +296,12 @@ hook is the fix, and this paragraph is the record of why it was skipped.
 One thing the residual does say: those 20 are runs that did substantial work (11–95 tool calls) and left
 no verdict. They are the most interesting runs in the corpus, and no observable fact decides them.
 
+> **Reversed by round 2 (2026-08-09).** The reasoning above is sound and the conclusion was wrong. A 6%
+> residual looked negligible against the whole corpus, but those conversations are not distributed like
+> the rest — they are concentrated in exactly the runs a human can judge instantly and the OS cannot, and
+> they produced **5 of the 11 out-of-sample errors**. Measuring the residual by its *share* rather than by
+> its share *of the mistakes* is the error. The hook is back in scope.
+
 ### Step 2 — one signal, one card, one action
 
 Pick the single highest-evidence problem in the live data — **repeat agent crashes** (§2b) — and

package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-os",
-  "version": "0.329.0",
+  "version": "0.329.1",
   "description": "A generic, governed operating system for running autonomous agents safely across brands. Ships with a local web console.",
   "license": "MIT",
   "type": "commonjs",

scripts/outcome-label-sample.cjs

@@ -33,7 +33,18 @@ const { deriveRunOutcomes, foldConversations } = require(path.join(ROOT, 'dist/e
 const { readConversation } = require(path.join(ROOT, 'dist/edge/conversation.js'));
 
 const os = loadAgentOS();
-const convos = foldConversations(deriveRunOutcomes(os));
+let convos = foldConversations(deriveRunOutcomes(os));
+
+// `--exclude <labels.json>` drops conversations already labelled in an earlier round. A re-validation is
+// only worth running on rows the rules were NOT tuned against: the first round's errors bought two rules,
+// so scoring those same rows again measures the fitting, not the derivation.
+const exArg = process.argv.indexOf('--exclude');
+if (exArg > -1 && process.argv[exArg + 1]) {
+  const prior = new Set(Object.keys(JSON.parse(fs.readFileSync(process.argv[exArg + 1], 'utf8')).labels));
+  const before = convos.length;
+  convos = convos.filter((c) => !prior.has(c.convoId));
+  console.log(`excluding ${before - convos.length} already-labelled conversations`);
+}
 
 // Stratify: up to PER_BASIS per basis, taken evenly across each stratum so it isn't just the newest rows.
 const byBasis = new Map();

scripts/outcome-label-score.cjs

@@ -20,7 +20,10 @@ if (!process.env.AGENT_OS_HOME) { console.error('refusing to run without AGENT_O
 const { loadAgentOS } = require(path.join(ROOT, 'dist/kernel.js'));
 const { deriveRunOutcomes, foldConversations } = require(path.join(ROOT, 'dist/edge/outcome.js'));
 
-const labels = JSON.parse(fs.readFileSync(path.join(__dirname, 'outcome-labels.json'), 'utf8')).labels;
+// Which label set to score. Defaults to round 1; pass a path to score a later, independent round.
+const LABELS = process.argv[2] ? path.resolve(process.argv[2]) : path.join(__dirname, 'outcome-labels.json');
+const labels = JSON.parse(fs.readFileSync(LABELS, 'utf8')).labels;
+console.log(`labels: ${path.basename(LABELS)}`);
 const os = loadAgentOS();
 const convos = new Map(foldConversations(deriveRunOutcomes(os)).map((c) => [c.convoId, c]));
 

scripts/outcome-labels-2.json

@@ -0,0 +1,40 @@
+{
+  "_": "Round 2 — the independent re-validation the Step 1 PR owed (docs/insights-revisit.md).",
+  "_why": "Round 1's errors bought two rules (`died-early`, `human-session`), so round 1 measures the fitting as much as the derivation. This round samples ONLY conversations round 1 never touched (`outcome-label-sample.cjs --exclude scripts/outcome-labels.json`) and scores the rules exactly as shipped in v0.323.0 — nothing was changed after seeing these results.",
+  "_corpus": "instapods, 30-day window, snapshot 2026-08-09. 32 conversations, stratified across all 9 bases, labelled blind from task text + transcript only.",
+  "_scale": "success | partial | incomplete | noop | failure | abandoned | ? — '?' means no transcript on this box, so no independent evidence existed.",
+  "labels": {
+    "019fb7ad-61de-7232-a5da-c47fde855819": "?",
+    "0d05e765-6db2-4cb9-a5f8-0788689b5fa0": "noop",
+    "1d0d1573-4568-498d-84b9-8d989100a78d": "failure",
+    "1e5d4a24-3863-4171-be48-22ec8d3f342b": "?",
+    "243a16a1-2607-44f4-9db1-a54530b6fda6": "success",
+    "2eec9a23-63ee-418e-96c4-253e270be67c": "success",
+    "48dad71c-6bf6-470e-b8b6-3b3809e9ec18": "failure",
+    "515d14f9-f391-4002-a14a-15c5352ccefe": "noop",
+    "539528a0-8706-4553-ae40-3e5eca653960": "incomplete",
+    "59f03251-3cbd-4136-a38e-af4a3a118868": "success",
+    "5d1f1dea-54a2-454c-bed0-ab5e36ec75d2": "success",
+    "70b976b6-9738-4b52-8254-cea6e7a22232": "success",
+    "796ac33f-2f22-40de-9b45-60acc5d15c69": "success",
+    "83c71edf-243b-481a-b49a-734e56752450": "success",
+    "847a7d2a-04ec-4705-b1ad-bea293554533": "?",
+    "863c774a-4af6-494d-a70f-f3504ce0a508": "success",
+    "8f90b099-50cf-4cdd-877f-e98bf8a247b2": "success",
+    "974d3bd4-4615-4fdb-ac7a-79344c4778b4": "success",
+    "9d01c40e-5ab1-4bcf-af28-cfb815291628": "failure",
+    "b04ff3c4-d2c9-4e48-ac2c-9b234e38302d": "failure",
+    "b16f2d46-4987-4561-853d-fe65bad33bcc": "noop",
+    "b6714ccd-f89e-4575-a56e-8b2d7bb064ed": "partial",
+    "bb635dfb-af8f-46bf-bd15-8fd186fdcb70": "?",
+    "db219c0a-d0d3-497f-a562-e1f047c42ef2": "noop",
+    "db9b6583-0be0-436b-a45e-8d3ed0bded9b": "failure",
+    "df67ad56-0392-48e3-a78b-d892b1efa151": "success",
+    "e3f5c0ee-9803-4541-975e-1779bfdb0f47": "?",
+    "e964eb19-fb4d-4147-ad11-0243a953d497": "failure",
+    "eb625638-56d9-4860-a1b0-0baf952c0408": "success",
+    "ed3c97a8-09d1-4488-bd21-54fc0744bf8e": "partial",
+    "faa21b63-77f3-4035-a571-2524d66118cc": "success",
+    "ff8149fe-c370-486b-b44e-3ccc33c05ccd": "success"
+  }
+}