Serve the feed timeline from D1; Fly proxy becomes an ingest-push crawler - #348
Open
disnet wants to merge 4 commits into
Open
Serve the feed timeline from D1; Fly proxy becomes an ingest-push crawler#348disnet wants to merge 4 commits into
disnet wants to merge 4 commits into
Conversation
…wler A refresh was `1 + ceil((N-8)/50)` sequential batch calls, each hopping Worker -> Fly and then paying ~18 chunked D1 queries to annotate read state. Now the proxy pushes new/edited items into a central D1 archive and a refresh is one `GET /api/v2/timeline` — a single query that joins subscriptions and read state, so `getReadKeys` leaves the feed path entirely and reads never touch Fly. Backend: migration 0061 drops the dormant pre-Fly trio and creates `feeds` + `feed_items` (monotonic seq, unique (feed_url, guid), NOT NULL content_hash) plus the items_generation token; `routes/ingest.ts` adds the secret-authenticated, fail-closed `POST /api/internal/ingest` and `GET /api/internal/crawl-set` with an idempotent upsert, an 8 KB stored content cap and the 5,000-item per-feed sanity cap (the only pruning — D1 is an archive); `routes/timeline.ts` serves incremental drains and per-feed cold starts; `/api/v2/feeds/fetch` is re-backed with D1 plus a pull-through for a feed nobody has crawled yet. Proxy: the durable log is the outbox. `push_state` marks what reached D1, dirty rows drain in seq order every 15 s with capped backoff, the crawl set is pulled back every 5 min to keep feeds warm now that reads no longer stamp them, and push_state cascades on every feed_items delete. Everything is gated on INGEST_URL, so an environment without it crawls exactly as before. Frontend: one drain loop against a single global cursor in Dexie metadata, with the legacy batch path kept as a fallback for an old backend, a rollback, or an environment whose crawler isn't pushing yet. Also: a staging Fly proxy (fly.staging.toml + CI job + staging FEED_PROXY_URL) so the two environments stop sharing one machine, and admin feed health re-pointed at the archive with storage/churn alerts. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
…lout gate, and paged cold starts Ten findings from the review of the D1 timeline change, most consequential first. The subscribe-time pull-through ingested a proxy feed FORWARD. A proxy feed is newest-first and seq is assigned in insert order, so the newest item got the lowest seq — and since a re-push of an unchanged item is a no-op, that inversion never healed: every later per-feed cold start served the feed's oldest entries, and the sanity-cap trim would have deleted its newest. `ingestProxyFeed` now walks the array backwards, as the proxy's own `writeFeedItems` does. The rollout gate was inferred from an empty archive, which one pull-through write falsified: a client could commit a cursor against a D1 nothing was crawling and stop refreshing 40 feeds in silence. Both internal endpoints now stamp `sync_state.crawler_heartbeat_at` (the crawl-set pull runs every 5 minutes regardless of feed activity), the timeline reports `ingestActive` from it, and the client stays on the batch path until the crawler is demonstrably alive. Cold starts were bounded only by 500 feeds x 30 items — up to 15,000 rows buffered in one Worker response, for everyone at once after a generation bump. They now page: feeds in a stable order, an item budget per page, continuation via `cold_offset`/`nextColdOffset`. The client keeps the FIRST page's cursor (the head read before the slices, so concurrent ingest can't be skipped) and commits it only once the last page merges. Also: - A cursor above the archive head now cold-starts that client, the D1 twin of the proxy's snapshot-restore guard — a Time Travel restore no longer stalls every client forever pending a manual `UPDATE sync_state`. - `contentTruncated` is no longer inert: it rides into the Article row and ArticleCard extracts the original when such an article is expanded, so a full-text feed doesn't quietly degrade to a two-sentence RSS summary. - Subscriptions that arrive from another device sit below the global cursor, so each is backfilled once through the per-feed endpoint (<= 10 per sync, tracked in Dexie). - New-subscription backfill no longer forces a crawl per feed and is paced (3 at a time, 1 s apart); subscribe time crawls straight into the archive (`warmFeedIntoArchive` replaces the old warm-and-discard), and the endpoint gets its own light rate limit. A 250-feed OPML import stops 429ing halfway. - The pull-through requires the caller to subscribe to the feed, so the shared never-pruned archive isn't an open write surface. - Staging keeps pointing at the prod proxy until Phase 3 provisions its own Fly app, and the staging deploy job skips itself with a notice until then, instead of taking staging's extract/discovery down on merge. - The incremental drain's scan cost is documented where it lives. Co-Authored-By: claudebot.disnetdev.com (did:plc:n6ku5xddiuguwze3f356evla) <claudebot.disnetdev.com@noreply.radial>
Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) <codexbot.disnetdev.com@noreply.radial>
Co-Authored-By: codexbot.disnetdev.com (did:plc:hbonvqr5ysrscg5wdyb5klie) <codexbot.disnetdev.com@noreply.radial>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the D1-served feed timeline and Fly ingest-push crawler architecture, including all review follow-ups on rollout safety and recovery.
Delivered
Checks
Radial artifact