Skip to content

feat(seer): Scope night shift result dedupe to (run, kind, idempotency_key)#119141

Draft
trevor-e wants to merge 9 commits into
masterfrom
telkins/night-shift-verdict-unique-constraint
Draft

feat(seer): Scope night shift result dedupe to (run, kind, idempotency_key)#119141
trevor-e wants to merge 9 commits into
masterfrom
telkins/night-shift-verdict-unique-constraint

Conversation

@trevor-e

@trevor-e trevor-e commented Jul 7, 2026

Copy link
Copy Markdown
Member

Description

Stacked on #118976, which added a code-level recorded-rows check so redelivered shard results neither re-trigger autofix nor write duplicate result rows. This PR adds the DB backstop for the one residual race — two deliveries of the same shard processed concurrently, both passing the check before either writes.

Originally scoped to (run, group), but night shift is meant to support workflows beyond group-based triage, so results need a dedupe key that isn't always a group. Generalized to an idempotency_key field, unique per (run, kind):

  • Migration 0026 adds the idempotency_key column and a partial unique constraint on (run, kind, idempotency_key) WHERE idempotency_key IS NOT NULL (created CONCURRENTLY by the migration framework).

Rows written before this deploys keep idempotency_key = NULL — they're excluded from the constraint by its partial condition, so no backfill is needed. They'll never be redelivered at this point, so there's nothing to protect; leaving them null also keeps any past duplicates queryable for investigation.

delivery.py now sets idempotency_key = str(group_id) for triage results and checks it (instead of raw group_id) for redelivery dedupe. It already used bulk_create(..., ignore_conflicts=True), a no-op until the constraint exists.

Merge sequencing: land after #118976 has fully deployed. That way the recorded-rows check is live while the index build runs, minimizing the window where a fresh duplicate could fail the concurrent index creation.

trevor-e and others added 4 commits July 2, 2026 19:23
Night shift previously persisted SeerNightShiftRunResult rows only for
AUTOFIX verdicts whose trigger succeeded, and nothing at all for dry
runs. SKIP and ROOT_CAUSE_ONLY verdicts, failed triggers, and dry-run
verdicts existed only in log lines, making verdict-outcome analysis
impossible from the database.

Delivery now writes one result row per verdict regardless of action,
carrying the action, a capped reason, and a trigger_error marker for
autofix verdicts that failed to start. Row creation moves into
_process_verdicts in a single bulk_create; the autofix trigger loop
(previously _run_autofix_for_candidates in cron, its only caller being
delivery) moves into delivery.py and only triggers, returning run state
ids. Redelivered shard results are ignored via an existing-rows check
plus a partial unique constraint on (run, group), preceded by a dedupe
data migration.

The run row also gets num_eligible_projects and num_candidates stamped
into extras so zero-shard runs are distinguishable in analysis (no
eligible projects vs no scored candidates vs dispatch failure).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Landing the constraint after this code deploys shrinks the window where
old code without the recorded-rows check can mint a new duplicate
between the dedupe and the concurrent index build. The bulk_create keeps
ignore_conflicts, which is a no-op until the constraint lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stacked on the all-verdict result rows PR, which added a code-level
recorded-rows check for redelivered shard results. This adds the DB
backstop for the remaining race of two concurrent deliveries of the
same shard: dedupe existing duplicate rows (keeping the first of each
pair), then add a partial unique constraint on (run, group). Delivery
already bulk_creates with ignore_conflicts, so conflicting rows drop
silently once the constraint exists.

Landing this after the parent PR deploys means the recorded-rows check
is live while the dedupe and concurrent index build run, minimizing the
chance a fresh duplicate fails the migration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 7, 2026
@sentry

sentry Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Sentry Snapshot Testing

Name Added Removed Changed Renamed Unchanged Skipped Status
sentry-frontend
sentry-frontend
0 0 0 0 451 0 ✅ Unchanged

⚙️ sentry-frontend Snapshot Settings

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

This PR has a migration; here is the generated SQL for src/sentry/seer/migrations/0026_dedupe_night_shift_run_results.py src/sentry/seer/migrations/0027_unique_night_shift_result_run_group.py

for 0026_dedupe_night_shift_run_results in seer

--
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQL

for 0027_unique_night_shift_result_run_group in seer

--
-- Create constraint seer_nightshiftrunresult_unique_run_group on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_group" ON "seer_nightshiftrunissue" ("run_id", "group_id") WHERE "group_id" IS NOT NULL;

trevor-e added a commit that referenced this pull request Jul 8, 2026
…118976)

Night shift previously persisted `SeerNightShiftRunResult` rows only for
AUTOFIX verdicts whose trigger succeeded — SKIP and ROOT_CAUSE_ONLY
verdicts, failed triggers, and dry runs existed only in log lines, so
verdict outcomes couldn't be analyzed from the database.

Delivery now writes one row per verdict in a single `bulk_create`:
`extras` carries the action and a capped reason; autofix verdicts
additionally link `result_seer_run` on success or set
`extras["trigger_error"]` when the trigger raised. Dry runs persist rows
too.

Supporting changes:

- **Redelivery is now idempotent.** Delivery skips groups that already
have a row for the run. Seer re-delivers shard results whenever a deploy
kills a worker holding an unacked task, and each redelivery previously
re-triggered autofix for every AUTOFIX verdict and wrote duplicate rows.
The `bulk_create` passes `ignore_conflicts=True` for the `(run, group)`
unique constraint landing in #119141 (stacked; merges after this
deploys).
- The autofix trigger loop moved from `cron.py` into delivery's
`_process_verdicts` — delivery was its only caller, previously reached
through a circular-import workaround.
- The run's `extras` now stamp `num_eligible_projects` and
`num_candidates`, so zero-shard runs are distinguishable: no eligible
projects vs. no scored candidates vs. dispatch failure.

No API or frontend changes needed: the `issues` payload already exposes
`extras["action"]` and the UI has labels for every action, so the
Explorer now shows the full triage picture instead of only autofixed
issues.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Base automatically changed from telkins/night-shift-verdict-rows to master July 8, 2026 14:36
…-verdict-unique-constraint

# Conflicts:
#	src/sentry/seer/night_shift/delivery.py
#	tests/sentry/seer/night_shift/test_delivery.py
Night shift is meant to be a general agentic workflow, but the dedupe key was (run, group) — a shape that assumes every workflow's unit of work is a single group. Replace it with an idempotency_key charfield, unique per (run, kind), so a future workflow kind that isn't group-shaped (e.g. keyed by repo or PR) can still get redelivery-safe dedupe without touching the group FK's semantics.

AGENTIC_TRIAGE keeps working the same way today: it just sets idempotency_key to str(group_id) instead of relying on group_id directly for the uniqueness constraint and the recorded-rows check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

This PR has a migration; here is the generated SQL for src/sentry/seer/migrations/0026_add_night_shift_result_idempotency_key.py src/sentry/seer/migrations/0027_backfill_night_shift_result_idempotency_key.py src/sentry/seer/migrations/0028_unique_night_shift_result_run_kind_idempotency_key.py

for 0026_add_night_shift_result_idempotency_key in seer

--
-- Add field idempotency_key to seernightshiftrunresult
--
ALTER TABLE "seer_nightshiftrunissue" ADD COLUMN "idempotency_key" varchar(256) NULL;

for 0027_backfill_night_shift_result_idempotency_key in seer

--
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQL

for 0028_unique_night_shift_result_run_kind_idempotency_key in seer

--
-- Create constraint seer_nightshiftrunresult_unique_run_kind_idempotency_key on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_kind_idempotency_key" ON "seer_nightshiftrunissue" ("run_id", "kind", "idempotency_key") WHERE "idempotency_key" IS NOT NULL;

@trevor-e trevor-e changed the title feat(seer): Add unique constraint on night shift result (run, group) feat(seer): Scope night shift result dedupe to (run, kind, idempotency_key) Jul 8, 2026
Historical rows will never be redelivered, so there's nothing to protect by backfilling idempotency_key on them — leaving it null is fine, since the unique constraint's partial condition already excludes null keys. Those null rows stay queryable if we ever need to investigate a past duplicate.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit afc1266. Configure here.

Comment thread src/sentry/seer/night_shift/delivery.py Outdated
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

This PR has a migration; here is the generated SQL for src/sentry/seer/migrations/0026_add_night_shift_result_idempotency_key.py src/sentry/seer/migrations/0027_unique_night_shift_result_run_kind_idempotency_key.py

for 0026_add_night_shift_result_idempotency_key in seer

--
-- Add field idempotency_key to seernightshiftrunresult
--
ALTER TABLE "seer_nightshiftrunissue" ADD COLUMN "idempotency_key" varchar(256) NULL;

for 0027_unique_night_shift_result_run_kind_idempotency_key in seer

--
-- Create constraint seer_nightshiftrunresult_unique_run_kind_idempotency_key on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_kind_idempotency_key" ON "seer_nightshiftrunissue" ("run_id", "kind", "idempotency_key") WHERE "idempotency_key" IS NOT NULL;

No backfill sits between them anymore, so there's nothing forcing them apart — CheckedMigration is non-atomic by default, which is all CREATE UNIQUE INDEX CONCURRENTLY needs, and this codebase already has precedent for combining AddField with AddConstraint in one file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

This PR has a migration; here is the generated SQL for src/sentry/seer/migrations/0026_add_night_shift_result_idempotency_key.py

for 0026_add_night_shift_result_idempotency_key in seer

--
-- Add field idempotency_key to seernightshiftrunresult
--
ALTER TABLE "seer_nightshiftrunissue" ADD COLUMN "idempotency_key" varchar(256) NULL;
--
-- Create constraint seer_nightshiftrunresult_unique_run_kind_idempotency_key on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_kind_idempotency_key" ON "seer_nightshiftrunissue" ("run_id", "kind", "idempotency_key") WHERE "idempotency_key" IS NOT NULL;

Switching the redelivery-dedupe check to idempotency_key broke it for every row written before this deploys, since those rows have a null idempotency_key. That's a much bigger hole than the accepted gap in the DB constraint (which only guards a rare concurrent-write race): any redelivery of a pre-deploy run's shard would now re-trigger autofix.

group_id is always populated for AGENTIC_TRIAGE results, before and after this change, so match on it directly instead — idempotency_key only needs to exist for the DB constraint and future non-group workflow kinds.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@trevor-e

trevor-e commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

@sentry review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant