feat(export): implement chunked R2 database export with DO alarm cont…#102
Open
scrollingreel wants to merge 1 commit intoouterbase:mainfrom
Open
feat(export): implement chunked R2 database export with DO alarm cont…#102scrollingreel wants to merge 1 commit intoouterbase:mainfrom
scrollingreel wants to merge 1 commit intoouterbase:mainfrom
Conversation
…inuation - Replace in-memory dump with batched chunked export using R2 storage - Process rows in configurable batches (1000 rows) with 20s time limit - Use DO alarms to resume long-running exports that exceed 30s timeout - Store export state in DO storage for pause/resume across alarm cycles - Add breathing intervals between alarm cycles (2s gap) to prevent DB lock - Support optional callbackUrl for async completion notifications (POST) - Fast path: small exports (<20s) return file directly in response - Async path: large exports return 202 with exportId for polling - Add GET /export/status/:exportId endpoint to check progress - Add GET /export/download/:exportId endpoint to download completed dumps - Add POST /export/dump endpoint with callbackUrl query param support - Exclude internal tmp_ tables from exports - Handle NULL values correctly in SQL INSERT statements - Graceful error handling with callback notifications on failure - R2 bucket binding optional - falls back to legacy in-memory dump - Add R2 bucket configuration to wrangler.toml and Env interface - Add comprehensive test suite (15 tests) for all new functionality - Backwards compatible: existing GET /export/dump still works Closes outerbase#59
Author
Implementation SummaryProblemThe current
SolutionChunked R2-based export with DO alarm continuation:
New Endpoints
Tests15 new tests covering initialization, chunk processing, NULL handling, error handling, callbacks, status endpoint, and download endpoint. All existing tests unaffected. /claim #59 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Implement chunked R2-based database export to support any database size (up to 10GB).
Uses DO alarms for exports exceeding the 30-second Worker timeout, with breathing
intervals to prevent database lock contention. Includes callback URL notifications
and new status/download endpoints.
/claim #59
Tasks
POST /export/dumpwith optionalcallbackUrlparamGET /export/status/:exportIdendpointGET /export/download/:exportIdendpointVerify
pnpm vitest run src/export/chunked-dump.test.ts— 15 tests should passpnpm vitest run— all 166 existing tests still passGET /export/dumpon a small DB — should return .sql file directlyPOST /export/dump?callbackUrl=https://example.comon a large DB — should return 202 with exportIdGET /export/status/:exportId— should show progressGET /export/download/:exportId— should return the .sql fileBefore
After
#Short Demo video of changes
Recording.2026-03-02.201444.1.mp4