Skip to content

feat(seer): Record a night shift result row for every triage verdict#118976

Merged
trevor-e merged 6 commits into
masterfrom
telkins/night-shift-verdict-rows
Jul 8, 2026
Merged

feat(seer): Record a night shift result row for every triage verdict#118976
trevor-e merged 6 commits into
masterfrom
telkins/night-shift-verdict-rows

Conversation

@trevor-e

@trevor-e trevor-e commented Jul 2, 2026

Copy link
Copy Markdown
Member

Night shift previously persisted SeerNightShiftRunResult rows only for AUTOFIX verdicts whose trigger succeeded — SKIP and ROOT_CAUSE_ONLY verdicts, failed triggers, and dry runs existed only in log lines, so verdict outcomes couldn't be analyzed from the database.

Delivery now writes one row per verdict in a single bulk_create: extras carries the action and a capped reason; autofix verdicts additionally link result_seer_run on success or set extras["trigger_error"] when the trigger raised. Dry runs persist rows too.

Supporting changes:

  • Redelivery is now idempotent. Delivery skips groups that already have a row for the run. Seer re-delivers shard results whenever a deploy kills a worker holding an unacked task, and each redelivery previously re-triggered autofix for every AUTOFIX verdict and wrote duplicate rows. The bulk_create passes ignore_conflicts=True for the (run, group) unique constraint landing in feat(seer): Scope night shift result dedupe to (run, kind, idempotency_key) #119141 (stacked; merges after this deploys).
  • The autofix trigger loop moved from cron.py into delivery's _process_verdicts — delivery was its only caller, previously reached through a circular-import workaround.
  • The run's extras now stamp num_eligible_projects and num_candidates, so zero-shard runs are distinguishable: no eligible projects vs. no scored candidates vs. dispatch failure.

No API or frontend changes needed: the issues payload already exposes extras["action"] and the UI has labels for every action, so the Explorer now shows the full triage picture instead of only autofixed issues.

trevor-e and others added 2 commits July 2, 2026 19:23
Night shift previously persisted SeerNightShiftRunResult rows only for
AUTOFIX verdicts whose trigger succeeded, and nothing at all for dry
runs. SKIP and ROOT_CAUSE_ONLY verdicts, failed triggers, and dry-run
verdicts existed only in log lines, making verdict-outcome analysis
impossible from the database.

Delivery now writes one result row per verdict regardless of action,
carrying the action, a capped reason, and a trigger_error marker for
autofix verdicts that failed to start. Row creation moves into
_process_verdicts in a single bulk_create; the autofix trigger loop
(previously _run_autofix_for_candidates in cron, its only caller being
delivery) moves into delivery.py and only triggers, returning run state
ids. Redelivered shard results are ignored via an existing-rows check
plus a partial unique constraint on (run, group), preceded by a dedupe
data migration.

The run row also gets num_eligible_projects and num_candidates stamped
into extras so zero-shard runs are distinguishable in analysis (no
eligible projects vs no scored candidates vs dispatch failure).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

This PR has a migration; here is the generated SQL for src/sentry/seer/migrations/0026_dedupe_night_shift_run_results.py src/sentry/seer/migrations/0027_unique_night_shift_result_run_group.py

for 0026_dedupe_night_shift_run_results in seer

--
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQL

for 0027_unique_night_shift_result_run_group in seer

--
-- Create constraint seer_nightshiftrunresult_unique_run_group on model seernightshiftrunresult
--
CREATE UNIQUE INDEX CONCURRENTLY "seer_nightshiftrunresult_unique_run_group" ON "seer_nightshiftrunissue" ("run_id", "group_id") WHERE "group_id" IS NOT NULL;

Landing the constraint after this code deploys shrinks the window where
old code without the recorded-rows check can mint a new duplicate
between the dedupe and the concurrent index build. The bulk_create keeps
ignore_conflicts, which is a no-op until the constraint lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
trevor-e and others added 3 commits July 7, 2026 17:27
_process_verdicts filtered unknown/already-recorded groups, marked SKIPs,
and built the autofix candidate list in three separate loops over the
incoming verdicts. Merge them into one loop; no behavior change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The ROOT_CAUSE_ONLY test reassigned `result` (the request payload dict)
to the fetched SeerNightShiftRunResult row, failing mypy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_trigger_autofix_for_candidates took Sequence[TriageResult] but never
read the action field — the caller pre-filtered to AUTOFIX, so the type
implied a polymorphism that didn't exist. Inline the loop into
_process_verdicts and carry plain Groups, looking reasons up from the
verdicts. Drops delivery's TriageResult dependency; no behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@trevor-e trevor-e marked this pull request as ready for review July 7, 2026 22:34
@trevor-e trevor-e requested a review from a team as a code owner July 7, 2026 22:34
)
)
# ignore_conflicts: concurrent redeliveries can race past the recorded-rows check.
SeerNightShiftRunResult.objects.bulk_create(rows, ignore_conflicts=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: bulk_create with ignore_conflicts=True is used without a database unique constraint, allowing concurrent redeliveries to create duplicate rows due to a race condition.
Severity: MEDIUM

Suggested Fix

Deploy the database migration that adds the UniqueConstraint on (run, group) to the SeerNightShiftRunResult model. This will make the ignore_conflicts=True parameter effective and prevent duplicate row creation at the database level, resolving the race condition.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/sentry/seer/night_shift/delivery.py#L237

Potential issue: The code at `delivery.py:237` calls `bulk_create` with
`ignore_conflicts=True`, but the corresponding `(run, group)` unique constraint on the
`SeerNightShiftRunResult` model does not exist in the database yet. Without this
constraint, `ignore_conflicts=True` has no effect. An idempotency check on lines 134-138
is insufficient to prevent duplicates under concurrent redeliveries. A
time-of-check/time-of-use (TOCTOU) race condition exists where two workers can both
check for existing rows (finding none), and then both proceed to insert the same data,
resulting in duplicate `SeerNightShiftRunResult` rows.

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixing in a stacked PR follow-up

@chromy chromy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@trevor-e trevor-e merged commit 0bcf0d7 into master Jul 8, 2026
68 checks passed
@trevor-e trevor-e deleted the telkins/night-shift-verdict-rows branch July 8, 2026 14:36
trevor-e added a commit that referenced this pull request Jul 8, 2026
…loads (#119168)

Night shift shard dispatch crashed with
`night_shift.shard_dispatch_failed` when a candidate group's title
contained a literal NUL character (e.g. `System.FormatException: The
input string '\u0000...'`). The `SEER_RUN_CREATE` outbox payload is a
`JSONField`, and Postgres jsonb rejects `\u0000` — the whole shard
failed to enqueue.

The existing `clean_bad_params` execute wrapper doesn't help here: it
only scrubs plain string params, and a `JSONField` value reaches the
cursor wrapped in psycopg2's `Json` adapter. Group titles come from
`Group.data`, a compressed blob field with no such restriction, so NULs
survive in Sentry and only blow up when copied into jsonb.

The fix recursively strips NUL characters and lone surrogates (also
rejected by jsonb) from the payload in `enqueue_seer_run`, so any Seer
run body built from customer event data persists cleanly — not just
night shift. The regression test reproduces the production `DataError`
against a real outbox insert without the fix.

Stacked on #118976.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants