async delete: retry on deadlock

[valentijnscholten]

Sometimes deadlocks happen during async object deletion:

django.db.utils.OperationalError: deadlock detected
DETAIL:  Process 287 waits for ShareLock on transaction 4339; blocked by process 288.
Process 288 waits for ShareLock on transaction 4342; blocked by process 287.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (0,21) in relation "dojo_tagulous_finding_tags"

78 s; sync files=125, longest=0.010 s, average=0.001 s; distance=78579 kB, estimate=101709 kB; lsn=9/984D4118, redo lsn=9/961F9F30
postgres  | '2025-12-11 15:11:43.156 UTC [pid=27] LOG:  checkpoint starting: time
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'LOG:  process 493 detected deadlock while waiting for ShareLock on transaction 3200316 after 600000.324 ms
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'DETAIL:  Process holding the lock: 494. Wait queue: 495, 504.
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'CONTEXT:  while updating tuple (1,1) in relation "dojo_tagulous_finding_tags"
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'STATEMENT:  UPDATE "dojo_tagulous_finding_tags" SET "count" = ("dojo_tagulous_finding_tags"."count" +  -1) WHERE "dojo_tagulous_finding_tags"."id" = 351
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'ERROR:  deadlock detected
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'DETAIL:  Process 493 waits for ShareLock on transaction 3200316; blocked by process 494.
postgres  |     Process 494 waits for ShareLock on transaction 3200319; blocked by process 493.
postgres  |     Process 493: UPDATE "dojo_tagulous_finding_tags" SET "count" = ("dojo_tagulous_finding_tags"."count" +  -1) WHERE "dojo_tagulous_finding_tags"."id" = 351
postgres  |     Process 494: DELETE FROM "watson_searchentry" WHERE "watson_searchentry"."id" IN (58232)
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'HINT:  See server log for query details.
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'CONTEXT:  while updating tuple (1,1) in relation "dojo_tagulous_finding_tags"
postgres  | '2025-12-11 15:13:19.932 UTC [pid=493] [txid=3200319] dbusr@dojodb 'STATEMENT:  UPDATE "dojo_tagulous_finding_tags" SET "count" = ("dojo_tagulous_finding_tags"."count" +  -1) WHERE "dojo_tagulous_finding_tags"."id" = 351
postgres  | '2025-12-11 15:13:19.933 UTC [pid=494] [txid=3200316] dbusr@dojodb 'LOG:  duration: 599998.718 ms  statement: DELETE FROM "watson_searchentry" WHERE "watson_searchentry"."id" IN (58232)

This seemed to be caused by Django running multiple pre_delete signals inside a transaction leading to possible deadlocks:

• findings sharing the same tag being deleted in parallel
• watson somehow locking related rows or tables such as a finding

I tried various things to fix it, but none of them helped:

• clear tags before models are removed
• clear watson search entries before models are removed
• disconnect watson signal temporarily while deleting chunks
• ordering models by id and using distinct
• making sure all chunks (findings) are deleted before the parent (test) is deleted

I ended up adding a retry mechanism, and some small other improvements:

• wait for chunk deletion to be completed before deleting the parent, otherwise some double deletions could occur
• ensure ordering of models by asecnding id to have all parallel tasks use the same ordering

[dryrunsecurity]

🔴 Risk threshold exceeded.

This pull request modifies a sensitive file (dojo/utils.py) with multiple detected edits flagged by the configured codepaths checks; reviewers should verify these changes against project policy or update .dryrunsecurity.yaml to allow the authors if intended.

🔴 Configured Codepaths Edit in dojo/utils.py

Vulnerability	Configured Codepaths Edit
Description	Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.

🔴 Configured Codepaths Edit in dojo/utils.py

Vulnerability	Configured Codepaths Edit
Description	Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.

🔴 Configured Codepaths Edit in dojo/utils.py

Vulnerability	Configured Codepaths Edit
Description	Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.

🔴 Configured Codepaths Edit in dojo/utils.py

Vulnerability	Configured Codepaths Edit
Description	Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.

We've notified @mtesauro.

---

All finding details can be found in the DryRun Security Dashboard.

[mtesauro]

Approved