Implement graph-driven cascade delete and restrict on Diagram

[dimitri-yatsenko]

Summary

Replace the error-driven cascade in Table.delete() with graph-driven restriction propagation using Diagram. The Diagram is purely a graph computation and inspection tool — all mutation logic (transactions, SQL execution, prompts) lives in Table.delete() and Table.drop().

Resolves: #865 (applying restrictions to a Diagram), #1110 (cascade delete fails on MySQL 8 with limited privileges)

Architecture

Table.delete() calls Diagram.cascade(self) to compute the affected subgraph, then iterates reversed(diagram) to delete leaves first. Table.drop() builds a Diagram with all descendants and drops in reverse topological order. The Diagram never executes mutations — it computes the cascade graph and provides counts() for inspection.

Public API: Diagram.cascade() (classmethod)

# Preview cascade impact across all loaded schemas
dj.Diagram.cascade(Session & 'subject_id=1').counts()

# Inspect the cascade subgraph (in Jupyter)
dj.Diagram.cascade(Session & 'subject_id=1')

Diagram.cascade(table_expr) builds a complete cascade diagram from a table expression — including all descendants across all loaded schemas. It uses OR convergence: a descendant row is marked for deletion if any ancestor path reaches it.

Public API: diagram.restrict() (instance method)

export = (dj.Diagram(schema)
    .restrict(Subject & {'species': 'mouse'})
    .restrict(Session & 'session_date > "2024-01-01"')
    .prune())
export.counts()

restrict() propagates restrictions downstream with AND convergence — a row is included only when ALL ancestor conditions are satisfied. Chainable. prune() removes zero-count tables (only valid on restrict, not cascade).

The two modes are mutually exclusive — restrict() raises on a cascade Diagram.

Table.delete() and Table.drop() changes

• Table.delete() uses Diagram.cascade(self, ...) internally; all execution logic (transactions, SQL, prompts) stays in Table
• Table.drop() builds a Diagram with all descendants; drops in reverse topo order
• Part.delete() returns the delete count; Part.drop() passes part_integrity through
• part_integrity post-check: data-driven, avoids false positives when a Part table appears in the cascade but has zero affected rows
• Safe delete workflow: with safemode=True, delete previews all affected tables inside a transaction and asks "Commit deletes?" — declining rolls back everything

Cross-schema cascade discovery

Dependencies.load_all_downstream() iteratively discovers schemas that reference the loaded schemas via FK relationships, expanding the dependency graph until all downstream schemas are included. This ensures cascade delete and drop reach dependent tables even in schemas that haven't been explicitly activated.

Called by Diagram.cascade() and Table.drop().

Restriction propagation rules

For edge Parent→Child with attr_map:

Condition	Child restriction
Non-aliased AND parent_attrs subset of child.primary_key	Copy parent restriction directly
Aliased FK (fk_attrs != pk_attrs)	parent.proj(**{fk: pk for fk, pk in attr_map.items()})
Non-aliased AND parent_attrs not subset of child.primary_key	parent.proj()

Convergence: cascade uses OR (any path), restrict uses AND (all conditions).

Bug fixes

• Diagram.__init__: Custom __iter__ created chicken-and-egg when populating nodes_to_show. Fixed: use self.nodes() (inherited DiGraph iterator).
• Index declaration parsing (declare.py): Allow inline comments on index lines.
• Identifier quoting: add_parts() strips both backticks and double quotes for PostgreSQL.
• Name parsing: All name splitting uses adapter.split_full_table_name() — removed ad-hoc implementations.
• Error message: repr(source) instead of repr(source[0]) in connection-not-found error.
• CLI tests: Use sys.executable -m datajoint.cli instead of bare dj command.
• Dead code: Removed get_master() from utils.py (replaced by extract_master()) and dead _part_integrity attribute.
• Bare except: Changed to except Exception: in Table.delete().

Advantages over error-driven cascade

Scenario	Error-driven (prior)	Graph-driven (new)
MySQL 8 + limited privileges	Crashes (#1110)	Works — no error parsing needed
PostgreSQL	Savepoint overhead per attempt	No errors triggered
Multiple FKs to same child	One-at-a-time via retry loop	All paths resolved upfront
Part integrity enforcement	Post-hoc check after delete	Data-driven post-check (no false positives)
Unloaded schemas	Crash with opaque error	Discovered automatically via load_all_downstream()
Reusability	Delete-only	Delete, drop, export, prune
Inspectability	Opaque recursive cascade	counts() preview + safemode confirmation before commit

Files changed (14)

File	Change
diagram.py	Full rewrite: cascade() classmethod, restrict(), counts(), prune(), __iter__/__reversed__, all name parsing via adapter.split_full_table_name()
table.py	delete() and drop() own execution logic; FreeTable.__init__ uses adapter.split_full_table_name()
dependencies.py	load() accepts schema_names; new load_all_downstream()
adapters/base.py	New find_downstream_schemas_sql()
adapters/mysql.py	Implement find_downstream_schemas_sql()
adapters/postgres.py	Implement find_downstream_schemas_sql()
declare.py	Fix index declaration regex for inline comments
user_tables.py	Part.delete() returns value; Part.drop() passes part_integrity
utils.py	Remove dead get_master()
version.py	Bump to 2.2.0dev0
docs/design/thread-safe-mode.md	Removed (captured in datajoint-docs)
test_cascade_delete.py	Cascade preview test + cross-schema cascade discovery test
test_erd.py	Prune tests (restrict, unrestricted, idempotent, cascade raises)
test_cli.py	Use sys.executable -m datajoint.cli

Test plan

• All existing cascade delete tests pass
• New cascade preview test via Diagram.cascade().counts()
• Cross-schema cascade discovery test
• Prune tests for restrict mode
• Prune raises on cascade Diagram
• Unit tests pass (243 passed)
• Full integration suite (CI — both MySQL and PostgreSQL)

[mweitzel]

A few inline comments.

[mweitzel]

Can get_master be fully removed? Not just as an import, but is it dead code now?

[dimitri-yatsenko]

Yes — removed. get_master() in utils.py had no remaining callers after the switch to extract_master() from dependencies.py.

[mweitzel]

is 10 arbitrary?

[dimitri-yatsenko]

Replaced with a while any_new: loop. Termination is guaranteed because the dependency graph is a DAG and propagated_edges prevents re-processing the same edge. The loop only needs multiple passes when part_integrity='cascade' triggers upward propagation from a part to its master — in practice 2-3 passes at most.

[mweitzel]

Can this at least catch runtime errors instead of bare except?

[dimitri-yatsenko]

Fixed — changed to except Exception:. This lets KeyboardInterrupt and SystemExit propagate without attempting cancel_transaction on a dying process.

[mweitzel]

Can this scope instead of bare except?

[dimitri-yatsenko]

Addressed above — except Exception:.

[mweitzel]

inline imports?

[dimitri-yatsenko]

Intentional — diagram.py and table.py have a circular import dependency (Diagram uses FreeTable, Table.delete() uses Diagram). The inline from .table import FreeTable inside the method body is the standard pattern used throughout the codebase to break this cycle.

[mweitzel]

Awesome 😃