feat(seer): Record a night shift result row for every triage verdict#118976
Conversation
Night shift previously persisted SeerNightShiftRunResult rows only for AUTOFIX verdicts whose trigger succeeded, and nothing at all for dry runs. SKIP and ROOT_CAUSE_ONLY verdicts, failed triggers, and dry-run verdicts existed only in log lines, making verdict-outcome analysis impossible from the database. Delivery now writes one result row per verdict regardless of action, carrying the action, a capped reason, and a trigger_error marker for autofix verdicts that failed to start. Row creation moves into _process_verdicts in a single bulk_create; the autofix trigger loop (previously _run_autofix_for_candidates in cron, its only caller being delivery) moves into delivery.py and only triggers, returning run state ids. Redelivered shard results are ignored via an existing-rows check plus a partial unique constraint on (run, group), preceded by a dedupe data migration. The run row also gets num_eligible_projects and num_candidates stamped into extras so zero-shard runs are distinguishable in analysis (no eligible projects vs no scored candidates vs dispatch failure). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This PR has a migration; here is the generated SQL for for --
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQLfor --
-- Create constraint seer_nightshiftrunresult_unique_run_group on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_group" ON "seer_nightshiftrunissue" ("run_id", "group_id") WHERE "group_id" IS NOT NULL; |
Landing the constraint after this code deploys shrinks the window where old code without the recorded-rows check can mint a new duplicate between the dedupe and the concurrent index build. The bulk_create keeps ignore_conflicts, which is a no-op until the constraint lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_process_verdicts filtered unknown/already-recorded groups, marked SKIPs, and built the autofix candidate list in three separate loops over the incoming verdicts. Merge them into one loop; no behavior change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The ROOT_CAUSE_ONLY test reassigned `result` (the request payload dict) to the fetched SeerNightShiftRunResult row, failing mypy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_trigger_autofix_for_candidates took Sequence[TriageResult] but never read the action field — the caller pre-filtered to AUTOFIX, so the type implied a polymorphism that didn't exist. Inline the loop into _process_verdicts and carry plain Groups, looking reasons up from the verdicts. Drops delivery's TriageResult dependency; no behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ) | ||
| ) | ||
| # ignore_conflicts: concurrent redeliveries can race past the recorded-rows check. | ||
| SeerNightShiftRunResult.objects.bulk_create(rows, ignore_conflicts=True) |
There was a problem hiding this comment.
Bug: bulk_create with ignore_conflicts=True is used without a database unique constraint, allowing concurrent redeliveries to create duplicate rows due to a race condition.
Severity: MEDIUM
Suggested Fix
Deploy the database migration that adds the UniqueConstraint on (run, group) to the SeerNightShiftRunResult model. This will make the ignore_conflicts=True parameter effective and prevent duplicate row creation at the database level, resolving the race condition.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: src/sentry/seer/night_shift/delivery.py#L237
Potential issue: The code at `delivery.py:237` calls `bulk_create` with
`ignore_conflicts=True`, but the corresponding `(run, group)` unique constraint on the
`SeerNightShiftRunResult` model does not exist in the database yet. Without this
constraint, `ignore_conflicts=True` has no effect. An idempotency check on lines 134-138
is insufficient to prevent duplicates under concurrent redeliveries. A
time-of-check/time-of-use (TOCTOU) race condition exists where two workers can both
check for existing rows (finding none), and then both proceed to insert the same data,
resulting in duplicate `SeerNightShiftRunResult` rows.
Did we get this right? 👍 / 👎 to inform future reviews.
There was a problem hiding this comment.
fixing in a stacked PR follow-up
…loads (#119168) Night shift shard dispatch crashed with `night_shift.shard_dispatch_failed` when a candidate group's title contained a literal NUL character (e.g. `System.FormatException: The input string '\u0000...'`). The `SEER_RUN_CREATE` outbox payload is a `JSONField`, and Postgres jsonb rejects `\u0000` — the whole shard failed to enqueue. The existing `clean_bad_params` execute wrapper doesn't help here: it only scrubs plain string params, and a `JSONField` value reaches the cursor wrapped in psycopg2's `Json` adapter. Group titles come from `Group.data`, a compressed blob field with no such restriction, so NULs survive in Sentry and only blow up when copied into jsonb. The fix recursively strips NUL characters and lone surrogates (also rejected by jsonb) from the payload in `enqueue_seer_run`, so any Seer run body built from customer event data persists cleanly — not just night shift. The regression test reproduces the production `DataError` against a real outbox insert without the fix. Stacked on #118976. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Night shift previously persisted
SeerNightShiftRunResultrows only for AUTOFIX verdicts whose trigger succeeded — SKIP and ROOT_CAUSE_ONLY verdicts, failed triggers, and dry runs existed only in log lines, so verdict outcomes couldn't be analyzed from the database.Delivery now writes one row per verdict in a single
bulk_create:extrascarries the action and a capped reason; autofix verdicts additionally linkresult_seer_runon success or setextras["trigger_error"]when the trigger raised. Dry runs persist rows too.Supporting changes:
bulk_createpassesignore_conflicts=Truefor the(run, group)unique constraint landing in feat(seer): Scope night shift result dedupe to (run, kind, idempotency_key) #119141 (stacked; merges after this deploys).cron.pyinto delivery's_process_verdicts— delivery was its only caller, previously reached through a circular-import workaround.extrasnow stampnum_eligible_projectsandnum_candidates, so zero-shard runs are distinguishable: no eligible projects vs. no scored candidates vs. dispatch failure.No API or frontend changes needed: the
issuespayload already exposesextras["action"]and the UI has labels for every action, so the Explorer now shows the full triage picture instead of only autofixed issues.