feat(insights): derive a run outcome from observed facts instead of the agent's self-grade (v0.321.0)

[vikasprogrammer]

Step 1 of docs/insights-revisit.md — the replacement Step 0's deletions owed.

src/edge/outcome.ts classifies each run from facts the OS observed itself, each verdict carrying the basis that decided it so a number always traces back to its evidence. report stays one input among several; a human's 👍/👎 outranks everything (and covers 1 run in 512 — exactly why it can't be the metric).

Two framing calls did more work than any rule: the unit is a conversation (a poke: resume continues a transcript, so scoring rows counts one job several times) and not everything is scorable (a person closing their own pane is not a failure — those leave the denominator instead of quietly counting as not-success).

Live 30-day instapods corpus — 443 conversations

	n
scorable	309	the denominator
success	186
partial	37
failure	28	against 1 self-reported failure, same window
noop	30	ran, called nothing — previously invisible
incomplete	8	someone else picked the task up
unknown	20 = 6%	was ~40%
unscorable	134	a person's own interactive session

Exit criteria met: unknown <10%, and the failure rate has variance (9% derived vs 0.3% self-reported).

The falsifier bit, and that's the interesting part

35 conversations sampled stratified by basis, labelled blind from transcripts only (outcome-label-sample.cjs → outcome-labels.json → outcome-label-score.cjs).

v1: 50% exact vs a 43% always-success baseline. Above the bar, but not enough to build on. The disagreements were clustered and bought two rules:

• died-early — unattended runs split by wall-clock: 2m+ → 96% report, 30–120s → 84%, <30s → 0 of 44. Those are quota/auth deaths (You've hit your weekly limit, 401 … token has expired) — this fleet's most common real failure, and structurally impossible to self-report because the agent is what stopped existing. 19 found in 30 days.
• human-session — v1 scored a person closing their own pane abandoned; the labels called four of those completed successes.

After both: 63% exact vs a 32% baseline (19 judged, 9 declared unscorable, 7 unlabelable — no transcript on this box).

⚠ The 63% is partly fitted. Labels were blind, but the rules were revised after seeing which rows v1 missed. The honest unfitted number is 50%, and a fresh blind sample is owed before Step 2 leans on this. Also a real coverage gap: the OS declines to judge 30% of conversations (every interactive session) — fine for Steps 2–5, which are about unattended work; not fine for a fleet-wide claim.

What I did not build

The Stop-hook half. The plan assumed the 40% hole had to be closed at the source. Once runs were classified, most of it wasn't missing information — 134 unscorable interactive sessions, plus 30 noops and 19 early deaths that are fully decidable from what's already recorded and could never have self-reported anyway. That left 6% without touching the teardown path. Deferred with the reasoning in the doc, not cancelled: if Step 2's cards are blocked by the residual 20, the hook is the fix.

Those residual 20 did substantial work (11–95 tool calls) and left no verdict — the most interesting runs in the corpus, and no observable fact decides them.

Verification

scripts/outcome-derivation-test.cjs — 23 assertions, added to test:governance, covering every rule, the precedence edges that cost a wrong answer (a task smoke test finishing in 8s must not read as died-early), conversation folding, and the property that the metric must move when work fails and not when reporting discipline changes.

Full test:governance green. npm run typecheck and cd web && npm run build clean. Nothing consumes the module yet; nothing is written or stamped — it's a pure read evaluated as-of a time.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AiTKjtdcF4ESbuYVWj2CfU