Observability phases 0–1: health, alerting, structured logs, request-id correlation

[disnet]

Implements Phases 0–1 of the observability plan. Phase 0 ("see breakage") landed in the first commit; this update adds Phase 1 and fixes the four review findings against Phase 0.

---

Phase 1 — structured logs + request-id correlation

Phase 0 made outages page. Phase 1 makes the aftermath legible: one id now finds every log line and every exception for the request, cron run, or poll cycle that caused an alert — across both runtimes.

• backend/src/utils/logger.ts — log.info('event', { … }) emits one object per line. Workers Logs indexes the fields of a logged object but treats a string (even a JSON one) as opaque text, so this turns "all errors on route X" and "everything for request Y" into queries instead of greps. The plan said "one-line JSON"; the object form is what actually gets indexed.
• backend/src/utils/request-context.ts — request id, route class and DID in AsyncLocalStorage (nodejs_als was already on for Sentry). Ambient rather than threaded, so the id reaches a log line, a reportError() tag and an outbound header without changing every handler signature. Route classes collapse identifiers (/api/saved/:id) and cap depth, so logs aggregate by endpoint and a path scanner can't mint unbounded route values.
• Converted the high-value paths only — request summary (route, status, durationMs), top-level catch, cron run + per-phase failures, Jetstream poll cycle. Leaf-level console.logs are left alone deliberately; a big-bang rewrite is churn.
• Firehose lag is now logged every poll cycle, computed from the Jetstream cursors' microsecond timestamps. Phase 2's alert threshold has a source to read.
• Cross-service: the id is returned as X-Request-Id (CORS-exposed, so a browser can quote it in a bug report) and sent on every feed-proxy call; the proxy adopts it, echoes it, and tags its own errors with it. It is minted, never accepted from a client — there's no trusted upstream, and a client sending a constant id would silently merge unrelated requests.

Review fixes on Phase 0

1. Deep health no longer reports the poller dead during normal operation (error). getAlarm() returns null while the alarm handler runs — 5–16s of every 60s cycle — so isRunning alone made a healthy poller look wedged on a large fraction of probes, the exact false page the runbook says to fix rather than ignore. /status now also returns lastAlarmStart, and the check mirrors the recency rule the DO's own /start branch already used (shared ALARM_ACTIVE_WINDOW_MS, so the two can't drift). lastPollAgeMs remains the real staleness signal; the body reports alarmScheduled and lastAlarmAgeMs separately so the distinction stays visible. 4 new tests drive the mid-alarm window deterministically.

2. Scrubbing now covers free text, not just structured fields (warning). Breadcrumb messages and data, exception messages, and event.message are scrubbed for tokenized URLs, Bearer …/DPoP …, and inline secret=… pairs. And the Console integration is dropped outright: call sites log raw session ids and OAuth state on failure paths, and no scrubber can reliably tell a credential from prose once it's been interpolated into a sentence — so the safe move is not to ship that channel. The defensive breadcrumb scrubbing stays for anything added another way. Fire drill #3 in the runbook now asks for the whole event, not just headers.

3. Pages deploys carry a version stamp after all (info). Each svelte.config.js stamps kit.version.name from $GITHUB_SHA, and all five smoke checks now assert the served /_app/version.json equals the SHA CI just built — the same proof the Workers get, so a deploy that silently didn't roll out goes red instead of passing a sniff against markup that never changes. The content sniff stays; it catches a different failure (deployed, but renders nothing). Verified by building all three apps with a fake SHA.

4. nodejs_als is now documented as a deliberate choice (info), narrower than the nodejs_compat the SDK's README asks for, with what to re-check on upgrade — plus its second consumer, the request context.

Docs

docs/RUNBOOK.md gains a "Reading the logs" section (the query patterns that pay for themselves, mapped to the events emitted), corrects the "Pages can't carry a version stamp" claim, and widens fire drill #3. backend/CLAUDE.md documents the logging convention.

Checks

• backend: npm run check (tsc + prettier) clean; full vitest suite green — 46 files / 448 tests, run in batches of 6 because this sandbox can't host 46 concurrent workerd isolates. 21 of those tests are new.
• feed-proxy: tsc --noEmit clean. bun test was not run — bun can't be installed in this container. The proxy delta is one middleware and the onError tags.
• frontend / admin / linkblog-site: npm run check clean; all three build and emit _app/version.json with the injected SHA.

Not verified here

Anything needing real infrastructure: the uptime/heartbeat vendor setup and the fire drills; the staging soak for nodejs_als + global_fetch_strictly_public; whether Workers Logs' field indexing behaves as documented on the deployed Worker (first event = request query after deploy will tell). Playwright suites weren't run (no browsers in this container) — no frontend runtime code changed, only svelte.config.js.

Phases 2–4 (system-status table + admin ops panel, client signal, SLOs/alert pruning) are still open.

Radial artifact