feat(seer): Scope night shift result dedupe to (run, kind, idempotency_key)#119141
feat(seer): Scope night shift result dedupe to (run, kind, idempotency_key)#119141trevor-e wants to merge 9 commits into
Conversation
Night shift previously persisted SeerNightShiftRunResult rows only for AUTOFIX verdicts whose trigger succeeded, and nothing at all for dry runs. SKIP and ROOT_CAUSE_ONLY verdicts, failed triggers, and dry-run verdicts existed only in log lines, making verdict-outcome analysis impossible from the database. Delivery now writes one result row per verdict regardless of action, carrying the action, a capped reason, and a trigger_error marker for autofix verdicts that failed to start. Row creation moves into _process_verdicts in a single bulk_create; the autofix trigger loop (previously _run_autofix_for_candidates in cron, its only caller being delivery) moves into delivery.py and only triggers, returning run state ids. Redelivered shard results are ignored via an existing-rows check plus a partial unique constraint on (run, group), preceded by a dedupe data migration. The run row also gets num_eligible_projects and num_candidates stamped into extras so zero-shard runs are distinguishable in analysis (no eligible projects vs no scored candidates vs dispatch failure). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Landing the constraint after this code deploys shrinks the window where old code without the recorded-rows check can mint a new duplicate between the dedupe and the concurrent index build. The bulk_create keeps ignore_conflicts, which is a no-op until the constraint lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stacked on the all-verdict result rows PR, which added a code-level recorded-rows check for redelivered shard results. This adds the DB backstop for the remaining race of two concurrent deliveries of the same shard: dedupe existing duplicate rows (keeping the first of each pair), then add a partial unique constraint on (run, group). Delivery already bulk_creates with ignore_conflicts, so conflicting rows drop silently once the constraint exists. Landing this after the parent PR deploys means the recorded-rows check is live while the dedupe and concurrent index build run, minimizing the chance a fresh duplicate fails the migration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sentry Snapshot Testing
|
|
This PR has a migration; here is the generated SQL for for --
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQLfor --
-- Create constraint seer_nightshiftrunresult_unique_run_group on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_group" ON "seer_nightshiftrunissue" ("run_id", "group_id") WHERE "group_id" IS NOT NULL; |
…118976) Night shift previously persisted `SeerNightShiftRunResult` rows only for AUTOFIX verdicts whose trigger succeeded — SKIP and ROOT_CAUSE_ONLY verdicts, failed triggers, and dry runs existed only in log lines, so verdict outcomes couldn't be analyzed from the database. Delivery now writes one row per verdict in a single `bulk_create`: `extras` carries the action and a capped reason; autofix verdicts additionally link `result_seer_run` on success or set `extras["trigger_error"]` when the trigger raised. Dry runs persist rows too. Supporting changes: - **Redelivery is now idempotent.** Delivery skips groups that already have a row for the run. Seer re-delivers shard results whenever a deploy kills a worker holding an unacked task, and each redelivery previously re-triggered autofix for every AUTOFIX verdict and wrote duplicate rows. The `bulk_create` passes `ignore_conflicts=True` for the `(run, group)` unique constraint landing in #119141 (stacked; merges after this deploys). - The autofix trigger loop moved from `cron.py` into delivery's `_process_verdicts` — delivery was its only caller, previously reached through a circular-import workaround. - The run's `extras` now stamp `num_eligible_projects` and `num_candidates`, so zero-shard runs are distinguishable: no eligible projects vs. no scored candidates vs. dispatch failure. No API or frontend changes needed: the `issues` payload already exposes `extras["action"]` and the UI has labels for every action, so the Explorer now shows the full triage picture instead of only autofixed issues. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…-verdict-unique-constraint # Conflicts: # src/sentry/seer/night_shift/delivery.py # tests/sentry/seer/night_shift/test_delivery.py
Night shift is meant to be a general agentic workflow, but the dedupe key was (run, group) — a shape that assumes every workflow's unit of work is a single group. Replace it with an idempotency_key charfield, unique per (run, kind), so a future workflow kind that isn't group-shaped (e.g. keyed by repo or PR) can still get redelivery-safe dedupe without touching the group FK's semantics. AGENTIC_TRIAGE keeps working the same way today: it just sets idempotency_key to str(group_id) instead of relying on group_id directly for the uniqueness constraint and the recorded-rows check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
This PR has a migration; here is the generated SQL for for --
-- Add field idempotency_key to seernightshiftrunresult
--
ALTER TABLE "seer_nightshiftrunissue" ADD COLUMN "idempotency_key" varchar(256) NULL;for --
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQLfor --
-- Create constraint seer_nightshiftrunresult_unique_run_kind_idempotency_key on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_kind_idempotency_key" ON "seer_nightshiftrunissue" ("run_id", "kind", "idempotency_key") WHERE "idempotency_key" IS NOT NULL; |
Historical rows will never be redelivered, so there's nothing to protect by backfilling idempotency_key on them — leaving it null is fine, since the unique constraint's partial condition already excludes null keys. Those null rows stay queryable if we ever need to investigate a past duplicate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit afc1266. Configure here.
|
This PR has a migration; here is the generated SQL for for --
-- Add field idempotency_key to seernightshiftrunresult
--
ALTER TABLE "seer_nightshiftrunissue" ADD COLUMN "idempotency_key" varchar(256) NULL;for --
-- Create constraint seer_nightshiftrunresult_unique_run_kind_idempotency_key on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_kind_idempotency_key" ON "seer_nightshiftrunissue" ("run_id", "kind", "idempotency_key") WHERE "idempotency_key" IS NOT NULL; |
No backfill sits between them anymore, so there's nothing forcing them apart — CheckedMigration is non-atomic by default, which is all CREATE UNIQUE INDEX CONCURRENTLY needs, and this codebase already has precedent for combining AddField with AddConstraint in one file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
This PR has a migration; here is the generated SQL for for --
-- Add field idempotency_key to seernightshiftrunresult
--
ALTER TABLE "seer_nightshiftrunissue" ADD COLUMN "idempotency_key" varchar(256) NULL;
--
-- Create constraint seer_nightshiftrunresult_unique_run_kind_idempotency_key on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_kind_idempotency_key" ON "seer_nightshiftrunissue" ("run_id", "kind", "idempotency_key") WHERE "idempotency_key" IS NOT NULL; |
Switching the redelivery-dedupe check to idempotency_key broke it for every row written before this deploys, since those rows have a null idempotency_key. That's a much bigger hole than the accepted gap in the DB constraint (which only guards a rare concurrent-write race): any redelivery of a pre-deploy run's shard would now re-trigger autofix. group_id is always populated for AGENTIC_TRIAGE results, before and after this change, so match on it directly instead — idempotency_key only needs to exist for the DB constraint and future non-group workflow kinds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@sentry review |

Description
Stacked on #118976, which added a code-level recorded-rows check so redelivered shard results neither re-trigger autofix nor write duplicate result rows. This PR adds the DB backstop for the one residual race — two deliveries of the same shard processed concurrently, both passing the check before either writes.
Originally scoped to
(run, group), but night shift is meant to support workflows beyond group-based triage, so results need a dedupe key that isn't always a group. Generalized to anidempotency_keyfield, unique per(run, kind):idempotency_keycolumn and a partial unique constraint on(run, kind, idempotency_key) WHERE idempotency_key IS NOT NULL(createdCONCURRENTLYby the migration framework).Rows written before this deploys keep
idempotency_key = NULL— they're excluded from the constraint by its partial condition, so no backfill is needed. They'll never be redelivered at this point, so there's nothing to protect; leaving them null also keeps any past duplicates queryable for investigation.delivery.pynow setsidempotency_key = str(group_id)for triage results and checks it (instead of rawgroup_id) for redelivery dedupe. It already usedbulk_create(..., ignore_conflicts=True), a no-op until the constraint exists.Merge sequencing: land after #118976 has fully deployed. That way the recorded-rows check is live while the index build runs, minimizing the window where a fresh duplicate could fail the concurrent index creation.