Skip to content

Add subcursor() for interleaved streaming + DML#397

Open
rustyconover wants to merge 1 commit intoduckdb:mainfrom
rustyconover:feature/subcursor
Open

Add subcursor() for interleaved streaming + DML#397
rustyconover wants to merge 1 commit intoduckdb:mainfrom
rustyconover:feature/subcursor

Conversation

@rustyconover
Copy link

@rustyconover rustyconover commented Mar 23, 2026

Summary

  • Adds subcursor() method that creates a cursor sharing the same Connection and transaction as the parent
  • Enables interleaved streaming SELECT + UPDATE/INSERT/DELETE within a single explicit transaction
  • Subcursors cannot manage transactions (begin/commit/rollback throw InvalidInputException)
  • Closing a subcursor cleans up any suspended query state without affecting the parent

Dependencies

This PR requires the companion core DuckDB PR: duckdb/duckdb#21569 — which adds the enable_suspended_queries setting and the suspend/resume mechanics in ClientContext. Without that PR, subcursor() will create a shared connection but suspension won't activate.

Usage

conn = duckdb.connect('my.db')
conn.execute("SET enable_suspended_queries = true")
conn.execute("BEGIN TRANSACTION")

scanner = conn.subcursor()
updater = conn.subcursor()

result = scanner.execute("SELECT rowid, col FROM t WHERE NOT processed")

while True:
    batch = result.fetchmany(1000)
    if not batch:
        break
    ids = ','.join(str(r[0]) for r in batch)
    updater.execute(f"UPDATE t SET processed=1 WHERE rowid IN ({ids})")

conn.execute("COMMIT")

Changes

  • ConnectionGuard: unique_ptr<Connection>shared_ptr<Connection> with ShareConnection()/GetSharedConnection() accessors
  • DuckDBPyConnection::Subcursor(): creates a cursor sharing the parent's Connection
  • is_subcursor flag guards Begin()/Commit()/Rollback()
  • pybind11 registration for subcursor method

Test plan

  • Basic interleaved scan + update pattern
  • Multi-table scan pattern (two tables, one updater)
  • Transaction guard (subcursor can't begin/commit/rollback)
  • Shared transaction visibility (subcursor sees uncommitted data)
  • Close doesn't destroy parent connection
  • Full stream consumption with periodic interleaved updates
  • Explicit transaction required for suspension to work
  • Close cancels suspended stream and frees resources
  • Close one of multiple suspended streams (others unaffected)
  • Abandoned subcursor (GC cleanup via destructor)
  • Existing cursor tests unaffected (13/13 pass, 1 pre-existing pytz skip)

🤖 Generated with Claude Code

Adds subcursor() method that creates a cursor sharing the same
underlying Connection and transaction as the parent. This enables
interleaved streaming SELECT + UPDATE/INSERT/DELETE within an
explicit transaction without destroying the streaming result.

Requires the companion duckdb core PR that adds the
enable_suspended_queries setting and suspend/resume mechanics.

Key changes:
- ConnectionGuard: unique_ptr<Connection> to shared_ptr<Connection>
- Add ShareConnection/GetSharedConnection accessors
- Add Subcursor() method that shares the parent's Connection
- Add is_subcursor flag with transaction management guard
- Subcursors cannot call begin/commit/rollback
- Closing a subcursor cleans up any suspended query state
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant