feat(insights): derive a run outcome from observed facts instead of the agent's self-grade (v0.321.0)

CHANGELOG.md

@@ -8,6 +8,36 @@ new version heading in the same commit.
 
 ## [Unreleased]
 
+## [0.323.0] — 2026-08-08
+### Added
+- **A run outcome derived from what the OS observed, not from the agent's grade of its own homework**
+  (`src/edge/outcome.ts`, `docs/insights-revisit.md` Step 1). Step 0 deleted every channel that broadcast
+  the old self-reported rate; this is the replacement it had to earn. Ordered rules over observed facts —
+  the process crashed, the run made no tool calls, an unattended run died in seconds, a task closed while
+  this run held it, another run had to pick the same task up — each carrying the `basis` that decided it,
+  so a number can always be traced to its evidence. `report` stays one input among several; a human's
+  👍/👎 outranks everything. Two framing calls did more work than any rule: **the unit is a conversation**
+  (a `poke:` resume continues a transcript, so scoring rows counts one job several times) and **not
+  everything is scorable** (a person closing their own pane is not a failure — those leave the denominator
+  instead of quietly counting as not-success).
+  On the live 30-day instapods corpus (443 conversations): **unknown 6%**, down from ~40%, and **28
+  failures against the 1** the fleet self-reported over the same window.
+  Two rules were bought by the falsifier rather than designed: **`died-early`** — unattended runs split by
+  wall-clock at 2m+ → 96% report, 30–120s → 84%, **<30s → 0 of 44**, which are quota/auth deaths
+  (`You've hit your weekly limit`, `401 … token has expired`), this fleet's most common real failure and
+  structurally impossible to self-report since the agent is what stopped existing (19 found in 30 days);
+  and **`human-session`**, after v1 scored four completed interactive sessions as `abandoned`.
+  Falsifier: 35 conversations sampled stratified by basis and labelled blind from transcripts
+  (`scripts/outcome-label-sample.cjs`, `outcome-labels.json`, `outcome-label-score.cjs`). **v1: 50% exact
+  vs a 43% always-success baseline. After the two rules above: 63% vs 32%** — with the caveat, stated in
+  the doc, that the rules were revised after seeing v1's errors, so the unfitted number is 50% and a fresh
+  blind sample is owed before Step 2 leans on it. Pinned by `scripts/outcome-derivation-test.cjs`
+  (23 assertions, in `test:governance`), including the property that the metric must move when work fails
+  and *not* when reporting discipline changes.
+  Nothing consumes this yet and nothing is stamped or written — it is a pure read, evaluated as-of a time,
+  so a task that reopens tomorrow changes yesterday's verdict. The Stop-hook half of the plan was **not**
+  built: once runs were classified, most of the hole was unscorable or already decidable, leaving 6%
+  without touching the teardown path. Deferred with the reasoning recorded, not cancelled.
 ## [0.322.0] — 2026-08-08
 ### Changed
 - **Session status is an ICON now, not a coloured dot + a word.** v0.321.0 unified the vocabulary but

docs/insights-revisit.md

@@ -176,7 +176,51 @@ no longer derives a percentage from them, and states that outcome is self-report
   case is the fixture Step 1 must satisfy: two states differing **only** in how many runs reported,
   with identical real failures, must produce identical guidance.
 
-### Step 1 — an outcome that isn't self-graded
+### Step 1 — an outcome that isn't self-graded ✅ shipped v0.323.0
+
+`src/edge/outcome.ts`. Rules over facts the OS observed itself, ordered, each carrying the `basis` that
+decided it so any number traces back to its evidence. Live 30-day instapods corpus, 443 conversations:
+
+| | conversations | note |
+|---|---|---|
+| scorable | **309** | the denominator |
+| success | 186 | |
+| partial | 37 | |
+| failure | **28** | against **1** self-reported failure in the same corpus |
+| noop | 30 | ran, called nothing — previously invisible |
+| incomplete | 8 | someone else had to pick the task up |
+| **unknown** | **20 = 6%** | was ~40% |
+| unscorable | 134 | a person's own interactive session — outside the denominator, not counted as not-success |
+
+Two framing decisions did more work than any rule: **the unit is a conversation** (a `poke:` resume
+continues a transcript — scoring rows counts one job several times), and **not everything is scorable**
+(a human closing their own pane is not a failure).
+
+- **Exit — met.** `unknown` 6% (bar: <10%). Failure rate has variance: 9% derived vs 0.3% self-reported,
+  and non-success is 40% where the old metric's complement was mostly non-reporting.
+- **Falsifier — run, and it bit.** 35 conversations sampled stratified by basis, labelled blind from
+  transcripts only (`scripts/outcome-label-sample.cjs` → `outcome-labels.json` → `outcome-label-score.cjs`).
+  **v1 scored 50% exact against a 43% always-success baseline** — beating the baseline, but not by enough
+  to build on. The disagreements were clustered and diagnostic, and bought two rules:
+  - **`died-early`.** Unattended runs split by wall-clock: 2m+ → 96% report, 30–120s → 84%, **<30s → 0 of
+    44**. Those are quota/auth deaths (`You've hit your weekly limit`, `401 … token has expired`) — this
+    fleet's most common real failure, structurally invisible to the agent because the agent is what
+    stopped existing. 19 found in 30 days.
+  - **`human-session`.** v1 called a person closing their own pane `abandoned`; the labels called four of
+    those successes. The OS has no verdict on an interactive session — same posture as chat.
+  After both, **63% exact against a 32% baseline** (19 judged, 9 declared unscorable, 7 unlabelable for
+  lack of a transcript on this box).
+- **⚠ The 63% is not a clean number.** The labels were blind, but the rules were revised *after* seeing
+  which rows v1 got wrong, so it is partly fitted to 28 rows. **Before Step 2 leans on this, draw a fresh
+  blind sample against the current rules.** The honest, unfitted number is v1's 50%.
+- **Known coverage gap:** the OS declines to judge 134 of 443 conversations (30%) — every interactive
+  human session. Fine for Steps 2–5, which are about unattended work; it would not be fine for a
+  fleet-wide "how are we doing" claim.
+- **Pinned:** `scripts/outcome-derivation-test.cjs` (23 assertions, in `test:governance`), including the
+  property that the metric must move when work fails and *not* when reporting discipline changes.
+- **Not done:** the Stop-hook half. See "What Step 1 did not do" below.
+
+### Step 1 (original plan)
 
 The blocking dependency for everything else. Derive a per-run outcome from **observable facts already
 in the DB**, not from the agent's own `report`:
@@ -198,6 +242,25 @@ becomes rare rather than modal.
   publish the confusion counts in the PR. If the derived signal doesn't beat "always success", stop
   and rethink before Step 2.
 
+#### What Step 1 did not do
+
+**The Stop-hook half was not built, and the reason is the evidence rather than the effort.** The plan
+assumed the 40% hole had to be closed at the source — make unattended runs report. Once the runs were
+classified it turned out most of the hole was not missing information at all:
+
+- 134 conversations were a person's own interactive session — never scorable, hook or no hook;
+- 30 called no tool at all, and 19 died in seconds — both fully decidable from what the OS already
+  recorded, and *neither could ever have self-reported* (a run killed by a quota limit has nothing left
+  to report with).
+
+That left `unknown` at 6%, under the exit bar, without touching the hook. Building it anyway would have
+added a runtime change to the teardown path for a residual the derivation already handles — so it is
+**deferred, not cancelled**: if Step 2's cards turn out to be blocked by those 20 conversations, the
+hook is the fix, and this paragraph is the record of why it was skipped.
+
+One thing the residual does say: those 20 are runs that did substantial work (11–95 tool calls) and left
+no verdict. They are the most interesting runs in the corpus, and no observable fact decides them.
+
 ### Step 2 — one signal, one card, one action
 
 Pick the single highest-evidence problem in the live data — **repeat agent crashes** (§2b) — and

package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-os",
-  "version": "0.322.0",
+  "version": "0.323.0",
   "description": "A generic, governed operating system for running autonomous agents safely across brands. Ships with a local web console.",
   "license": "MIT",
   "type": "commonjs",
@@ -27,7 +27,7 @@
     "check-deps": "bash scripts/install-deps.sh --check",
     "dev": "ts-node src/cli.ts serve",
     "demo:dev": "ts-node src/demo.ts",
-    "test:governance": "node scripts/governance-conformance.cjs && node scripts/tier-a-policy-test.cjs && node scripts/capability-registry-test.cjs && node scripts/idle-reaper-test.cjs && node scripts/dm-continuity-test.cjs && node scripts/alert-staleness-test.cjs && node scripts/run-as-identity-test.cjs && node scripts/deps-freshness-test.cjs && node scripts/runtime-account-test.cjs && node scripts/runtime-login-test.cjs && node scripts/claude-config-seed-test.cjs && node scripts/verbosity-test.cjs && node scripts/chain-model-test.cjs && node scripts/tuning-patch-test.cjs && node scripts/task-runs-test.cjs && node scripts/task-resume-test.cjs && node scripts/warm-chat-test.cjs && node scripts/agent-edit-guard-test.cjs && node scripts/goal-update-guard-test.cjs && node scripts/insights-signal-test.cjs",
+    "test:governance": "node scripts/governance-conformance.cjs && node scripts/tier-a-policy-test.cjs && node scripts/capability-registry-test.cjs && node scripts/idle-reaper-test.cjs && node scripts/dm-continuity-test.cjs && node scripts/alert-staleness-test.cjs && node scripts/run-as-identity-test.cjs && node scripts/deps-freshness-test.cjs && node scripts/runtime-account-test.cjs && node scripts/runtime-login-test.cjs && node scripts/claude-config-seed-test.cjs && node scripts/verbosity-test.cjs && node scripts/chain-model-test.cjs && node scripts/tuning-patch-test.cjs && node scripts/task-runs-test.cjs && node scripts/task-resume-test.cjs && node scripts/warm-chat-test.cjs && node scripts/agent-edit-guard-test.cjs && node scripts/goal-update-guard-test.cjs && node scripts/insights-signal-test.cjs && node scripts/outcome-derivation-test.cjs",
     "test:alert-staleness": "node scripts/alert-staleness-test.cjs",
     "test:deps": "node scripts/deps-freshness-test.cjs && node scripts/runtime-account-test.cjs && node scripts/runtime-login-test.cjs && node scripts/claude-config-seed-test.cjs",
     "test:dm-continuity": "node scripts/dm-continuity-test.cjs",

scripts/outcome-derivation-test.cjs

@@ -0,0 +1,139 @@
+#!/usr/bin/env node
+/* Derived-outcome rules (docs/insights-revisit.md, Step 1).
+ *
+ * Synthetic rows, so each rule and each precedence edge is exact. The live-corpus check is a separate,
+ * manual pair — `outcome-label-sample.cjs` + `outcome-label-score.cjs` against a DB snapshot — because it
+ * needs transcripts and a human read; this file is what CI can run.
+ *
+ * The property at the bottom is the one that matters most: the metric must move when real failures move
+ * and NOT when reporting discipline does. That is the exact defect Step 0 deleted, restated as a test. */
+const fs = require('fs');
+const os_ = require('os');
+const path = require('path');
+
+const ROOT = path.resolve(__dirname, '..');
+const HOME = fs.mkdtempSync(path.join(os_.tmpdir(), 'aos-outcome-test-'));
+process.env.AGENT_OS_HOME = HOME;
+process.env.AGENT_OS_TENANT = 'testco';
+delete process.env.AGENT_OS_SECRET_KEY;
+
+let pass = 0, fail = 0;
+const assert = (c, name, d) => c ? (pass++, console.log(`  \x1b[32m✓\x1b[0m ${name}`)) : (fail++, console.log(`  \x1b[31m✗ ${name}\x1b[0m${d ? ' — ' + d : ''}`));
+
+const { loadAgentOS } = require(path.join(ROOT, 'dist/kernel.js'));
+const { deriveRunOutcomes, foldConversations, summarize } = require(path.join(ROOT, 'dist/edge/outcome.js'));
+
+const aos = loadAgentOS();
+const NOW = Date.now();
+let n = 0;
+
+/** One terminated run. Everything the rules read is explicit — no hidden defaults to argue with later. */
+function mkRun(o) {
+  const id = 'ts_' + (++n);
+  const at = o.at ?? NOW - 3600_000;
+  aos.db.prepare(
+    'INSERT INTO term_sessions (id,agent,title,task,tmux,status,headless,spawned_by,created_at,updated_at,' +
+      'claude_session_id,rating,outcome,tool_calls,active_ms) VALUES (?,?,?,?,?,?,1,?,?,?,?,?,?,?,?)',
+  ).run(id, o.agent ?? 'worker', 't', 'x', 'aos-' + id, o.status ?? 'done', o.spawnedBy ?? 'automation:au_1',
+    at, at + (o.activeMs ?? 300_000), o.convo ?? null, o.rating ?? null, o.outcome ?? null,
+    o.toolCalls === undefined ? 20 : o.toolCalls, o.activeMs ?? 300_000);
+  return id;
+}
+const verdictOf = (id) => deriveRunOutcomes(aos, { since: 0, until: NOW + 1 }).find((r) => r.runId === id);
+
+console.log('\n\x1b[1mDerived outcome — rules over observed facts, not the agent\'s own grade\x1b[0m');
+
+// ── each rule fires ────────────────────────────────────────────────────────────────────────────────
+{
+  const up = mkRun({ rating: 'up', outcome: 'failure' });        // human outranks the report
+  const down = mkRun({ rating: 'down', outcome: 'success' });
+  assert(verdictOf(up).verdict === 'success' && verdictOf(up).basis === 'human-rating', '👍 outranks a reported failure');
+  assert(verdictOf(down).verdict === 'failure' && verdictOf(down).basis === 'human-rating', '👎 outranks a reported success');
+
+  const crashed = mkRun({ status: 'crashed', outcome: 'success' });
+  assert(verdictOf(crashed).verdict === 'failure' && verdictOf(crashed).basis === 'crashed', 'a crash outranks a reported success');
+
+  const early = mkRun({ activeMs: 9_000, toolCalls: 2 });
+  assert(verdictOf(early).basis === 'died-early' && verdictOf(early).verdict === 'failure', 'an automation run dead in 9s is a failure');
+
+  const noop = mkRun({ toolCalls: 0, activeMs: 600_000 });
+  assert(verdictOf(noop).verdict === 'noop' && verdictOf(noop).basis === 'no-tool-calls', 'zero tool calls is a noop, not an unknown');
+
+  const reported = mkRun({ outcome: 'partial' });
+  assert(verdictOf(reported).verdict === 'partial' && verdictOf(reported).basis === 'reported', 'the report is still used when it exists');
+
+  const nothing = mkRun({});
+  assert(verdictOf(nothing).verdict === 'unknown' && verdictOf(nothing).basis === 'no-evidence', 'no evidence stays honestly unknown');
+}
+
+// ── the guards that cost a wrong answer on live data ───────────────────────────────────────────────
+{
+  const shortTask = mkRun({ spawnedBy: 'task:tsk_short', activeMs: 8_000, toolCalls: 3 });
+  assert(verdictOf(shortTask).basis !== 'died-early', 'a TASK run finishing in 8s is not a died-early failure (smoke tests are fast)');
+
+  const chatty = mkRun({ activeMs: 5_000, toolCalls: 0 });
+  assert(verdictOf(chatty).verdict === 'noop', 'a short automation run that called nothing is a noop, not a failure');
+
+  const human = mkRun({ spawnedBy: 'm_alice', status: 'stopped', toolCalls: 40 });
+  assert(verdictOf(human).verdict === 'abandoned' && verdictOf(human).basis === 'human-session', "a person's own interactive session is unscorable, not a failure");
+
+  const chat = mkRun({ spawnedBy: 'chat:support' });
+  assert(!verdictOf(chat), 'chat-triggered runs are excluded entirely');
+
+  const stopped = mkRun({ spawnedBy: 'task:tsk_stop', status: 'stopped', toolCalls: 90 });
+  assert(verdictOf(stopped).verdict === 'incomplete' && verdictOf(stopped).basis === 'stopped-midway', 'a killed unattended run is incomplete, never a success');
+}
+
+// ── conversation folding ───────────────────────────────────────────────────────────────────────────
+{
+  const cid = 'convo-1';
+  mkRun({ convo: cid, at: NOW - 7200_000, spawnedBy: 'task:tsk_a' });                    // unknown
+  mkRun({ convo: cid, at: NOW - 3600_000, spawnedBy: 'poke:tsk_a', outcome: 'success' }); // then finished
+  const folded = foldConversations(deriveRunOutcomes(aos, { since: 0, until: NOW + 1 })).find((c) => c.convoId === cid);
+  assert(folded && folded.runs === 2, 'resumes fold into one conversation, not two outcomes');
+  assert(folded && folded.verdict === 'success', 'a hand-off that needed a poke and then landed is a success');
+
+  const cid2 = 'convo-2';
+  mkRun({ convo: cid2, at: NOW - 7200_000, status: 'crashed' });
+  mkRun({ convo: cid2, at: NOW - 3600_000, outcome: 'success' });
+  const f2 = foldConversations(deriveRunOutcomes(aos, { since: 0, until: NOW + 1 })).find((c) => c.convoId === cid2);
+  assert(f2 && f2.verdict === 'failure', 'a later clean run does not erase an earlier crash');
+}
+
+// ── the summary contract ───────────────────────────────────────────────────────────────────────────
+{
+  const all = summarize(foldConversations(deriveRunOutcomes(aos, { since: 0, until: NOW + 1 })));
+  assert(all.scorable === all.success + all.partial + all.incomplete + all.noop + all.failure + all.unknown,
+    'scorable is exactly the decided verdicts plus the honest unknowns', JSON.stringify(all));
+  assert(!Number.isNaN(all.abandoned) && all.abandoned > 0 && all.scorable > 0, 'abandoned is tracked but sits outside scorable');
+  const tiny = summarize([{ convoId: 'x', agent: 'a', verdict: 'success', basis: 'reported', runs: 1, at: NOW }]);
+  assert(tiny.successRate === null, 'no success rate below the minimum sample');
+  assert(tiny.unknownShare === 0, 'unknown share is still reported on a tiny sample');
+}
+
+// ── THE property: the metric must move with failures, not with reporting discipline ────────────────
+{
+  const base = () => {
+    const rows = [];
+    for (let i = 0; i < 20; i++) rows.push({ convoId: 'c' + i, agent: 'a', verdict: 'success', basis: 'reported', runs: 1, at: NOW });
+    return rows;
+  };
+  // Ten runs stop calling `report` — but nothing actually failed. Under the old metric this alone moved
+  // the rate 50 points. Here they become `unknown`: the rate barely moves and the unknown SHARE shouts.
+  const quiet = base();
+  for (let i = 0; i < 10; i++) quiet[i] = { ...quiet[i], verdict: 'unknown', basis: 'no-evidence' };
+  // Ten runs genuinely fail.
+  const broken = base();
+  for (let i = 0; i < 10; i++) broken[i] = { ...broken[i], verdict: 'failure', basis: 'crashed' };
+
+  const b = summarize(base()), q = summarize(quiet), r = summarize(broken);
+  assert(q.unknownShare === 50 && q.unknown === 10, 'ten silent runs show up as a 50% unknown share, not as failures', JSON.stringify(q));
+  assert(r.failure === 10 && r.unknownShare === 0, 'ten real failures show up as failures with no unknowns', JSON.stringify(r));
+  assert(b.successRate === 100 && r.successRate === 50, 'the rate moves when work actually fails');
+  assert(q.successRate === 50 && q.unknownShare === 50,
+    'when the rate moves for a REPORTING reason, the unknown share moves with it — the reader can always tell the two apart');
+}
+
+fs.rmSync(HOME, { recursive: true, force: true });
+console.log(`\n${fail ? '\x1b[31m' : '\x1b[32m'}${pass} passed, ${fail} failed\x1b[0m\n`);
+process.exit(fail ? 1 : 0);