Open
Conversation
Outline for release post announcing Yjs durable streams as a managed service on Electric Cloud. Published: false. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add multiplayer territory game demo (built with Yjs CRDTs on Durable Streams) to the blog post. Players claim cells by moving over them, with Yjs LWW resolving contention. Embedded as an iframe. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VitePress dev server intercepts directory paths. Using the explicit filename bypasses VitePress routing and serves the static file. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
balegas
force-pushed
the
blog/yjs-durable-streams-on-electric-cloud
branch
from
e5a16f2 to
de80976
Compare
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
thruflo
requested changes
Mar 30, 2026
| published: true | ||
| --- | ||
|
|
||
| [Yjs](https://yjs.dev) is the de facto library for collaborative editing on the web. It's battle-proven, CRDT-based, and powers tools like [TipTap](https://tiptap.dev), [CodeMirror](https://codemirror.net), [BlockNote](https://www.blocknotejs.org/) and more. And it's not just collaboration between humans anymore — agents are increasingly editing documents, generating code, and filling in forms alongside users. Whether it's humans or agents collaborating, they need reliable, conflict-free sync. |
Contributor
There was a problem hiding this comment.
I would perhaps say what we're releasing here.
I.e.: "Yjs is X, it's increasingly for agents and <today we're releasing ...>."
What is it that the reader would be trying now when they "Try it now" below?! It's not Yjs, it's our new Durable Streams Yjs integration ...
Contributor
There was a problem hiding this comment.
Tweak the filename date to 31st March. N.b.: fixing any links referring to it.
| [Yjs](https://yjs.dev) is the de facto library for collaborative editing on the web. It's battle-proven, CRDT-based, and powers tools like [TipTap](https://tiptap.dev), [CodeMirror](https://codemirror.net), [BlockNote](https://www.blocknotejs.org/) and more. And it's not just collaboration between humans anymore — agents are increasingly editing documents, generating code, and filling in forms alongside users. Whether it's humans or agents collaborating, they need reliable, conflict-free sync. | ||
|
|
||
| >[!info] 🚀 Try it now | ||
| >Sign up to [Electric Cloud](https://dashboard.electric-sql.com), create a Yjs service, and connect your app. |
Contributor
There was a problem hiding this comment.
Maybe:
Create a Yjs service, see the integration docs, source code and demo app.
Contributor
There was a problem hiding this comment.
(The create link must use the intent link).
|
|
||
| Most Yjs setups are built on WebSockets to relay updates to clients. WebSockets are point-to-point connections with no fan-out distribution. They require sticky connections and content can't be cached at a CDN, which means there is a latency penalty for every user or agent that needs to retrieve the initial state of a document. | ||
|
|
||
| There is no standardized reference implementation if you want to implement this yourself. There are hosted services you can buy, but that means vendor lock-in and a new piece of infrastructure to add to your stack. |
| >Sign up to [Electric Cloud](https://dashboard.electric-sql.com), create a Yjs service, and connect your app. | ||
| >See the [`y-durable-streams` source](https://github.com/durable-streams/durable-streams/tree/main/packages/y-durable-streams) and [demo app](https://github.com/durable-streams/durable-streams/tree/main/examples/yjs-demo) on GitHub. | ||
|
|
||
| ## The problem with WebSockets |
Contributor
There was a problem hiding this comment.
The title seems a bit odd to me. This is an article about our Yjs integration. Where did WebSockets come from? Could this first para be mixed in below, so the body of the article leads with what we've built?
|
|
||
| ## Yjs on bare HTTP | ||
|
|
||
| We've built [`y-durable-streams`](https://github.com/durable-streams/durable-streams/tree/main/packages/y-durable-streams) — a new Yjs provider on [Durable Streams](/primitives/durable-streams), an open HTTP protocol for persistent, resumable, real-time streams. |
Contributor
There was a problem hiding this comment.
This is the sentence/para that I'd love to be in the intro above.
|
|
||
| We've built [`y-durable-streams`](https://github.com/durable-streams/durable-streams/tree/main/packages/y-durable-streams) — a new Yjs provider on [Durable Streams](/primitives/durable-streams), an open HTTP protocol for persistent, resumable, real-time streams. | ||
|
|
||
| Instead of WebSocket relay servers, document updates flow through plain HTTP. Clients POST edits and subscribe for real-time updates via SSE or long-polling — no persistent connections, no sticky sessions. Because it's standard HTTP, it works with the infrastructure you already have: load balancers, reverse proxies, CDNs. Snapshots are cacheable at the edge, so fan-out scales without extra effort. |
Contributor
There was a problem hiding this comment.
This focuses on HTTP and connection details. Is the lede not the para below? What do Durable Streams do -- they provide persistence, durability and sync. Plus with this impl also compaction etc.
I would prioritize what this does rather than tilting at the WebSocket straw man.
|
|
||
| For the full details, see the [Yjs Durable Streams Protocol](https://github.com/durable-streams/durable-streams/blob/main/packages/y-durable-streams/YJS-PROTOCOL.md) specification. | ||
|
|
||
| ## Demo |
Contributor
There was a problem hiding this comment.
The demo is totally awesome. It just needs a little more packaging. This is the main feedback I wanted to relay on the post -- we can get loads more value from the work with some simple tweaks.
Right now the demo is an interactive iframe embed in the page. However, it's not obvious that it's interactive or that it's multi-player or that it's an app at all.
I suggest:
- screen recording a 30 - 45 second screencast of using the game. Use Screen Studio so you record yourself as well and just talk about the game and how it's built whilst using it.
- publish this to the ElectricSQL YouTube and use the YoutubeEmbed component to embed the video below the warning box in the header above -- that way it's impossible to miss it and it's obvious it's a screencast video of a demo app
- here in the page if we want to embed the app then it needs clearer signposting that it is indeed an embedded interactive app, not just an embedded screenshot. I would also personally have a primary button that opens the app in target blank (or a popover window) because I think that will be easier for people to test the multiplayer and use as an app vs the embed here
- if possible, publish a /demos/territory-wars demo, using the markdown front matter of the demo page to link to the video, demo app and source then you can embed / link to that from the page here.
|
|
||
| ## Next steps | ||
|
|
||
| - sign up to [Electric Cloud](https://dashboard.electric-sql.com) |
| excerpt: >- | ||
| Sync Yjs documents over plain HTTP. y-durable-streams brings built-in compaction, real-time presence, and fan-out via CDN to collaborative apps and agentic systems. | ||
| authors: [balegas] | ||
| image: /img/blog/yjs-durable-streams-on-electric-cloud/header.png |
Contributor
There was a problem hiding this comment.
The image is just slightly not vertically centered. It needs the top padding cropped out slightly. Ideally exported as an 1536 x 947 pixel image.
commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4059 +/- ##
==========================================
- Coverage 88.67% 83.50% -5.18%
==========================================
Files 25 65 +40
Lines 2438 3850 +1412
Branches 613 611 -2
==========================================
+ Hits 2162 3215 +1053
- Misses 274 633 +359
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Remove embedded iframe, link to demo page instead - Add territory-wars demo to /demos with screenshot - Remove game rules from demo description - Add header image and SnapshotSyncDiagram component - Remove outdated outline file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
balegas
force-pushed
the
blog/yjs-durable-streams-on-electric-cloud
branch
from
fea0515 to
2dde8a8
Compare
thruflo
reviewed
Mar 31, 2026
| [Yjs](https://yjs.dev) is the de facto library for collaborative editing on the web — battle-proven, CRDT-based, and powering tools like [TipTap](https://tiptap.dev), [CodeMirror](https://codemirror.net), and [BlockNote](https://www.blocknotejs.org/). Today we're releasing [`y-durable-streams`](https://www.npmjs.com/package/@durable-streams/y-durable-streams) — a new Yjs provider built on [Durable Streams](/primitives/durable-streams), now live on [Electric Cloud](/cloud). It brings built-in persistence, compaction, and real-time presence to collaborative apps and agentic systems. | ||
|
|
||
| >[!info] 🚀 Try it now | ||
| >[Create a Yjs service](https://dashboard.electric-sql.cloud/?intent=create&serviceType=yjs), see the [integration docs](https://durablestreams.com/yjs), [source code](https://github.com/durable-streams/durable-streams/tree/main/packages/y-durable-streams), and [demo app](https://github.com/balegas/territory-wars). |
Contributor
There was a problem hiding this comment.
The demo can be the link to the demo page now. (Same for any similar links).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Some investigation shows that yes, shutting down a DynamicSupervisor with lots of children is **slow**. Calling `Process.kill(pid, :shutdown)` on a simple `DynamicSupervisor` with 200,000 child processes (`Agent`s) takes ~10 **minutes** to complete on my machine. This is despite the fact that `DynamicSupervisor`s terminate their children in parallel - sending `:EXIT` messages to all child processes without waiting for any to terminate. From the [erlang docs](https://www.erlang.org/doc/system/sup_princ.html#simplified-one_for_one-supervisors): > Because a simple_one_for_one supervisor can have many children, it > shuts them all down asynchronously. This means that the children will do > their cleanup in parallel and therefore the order in which they are > stopped is not defined. `DynamicSupervisor` inherited its shutdown behaviour from this supervisor strategy. Using a `PartitionSupervisor` helps and roughly reduces the time to shutdown by ~O(number of partitions) but this does not scale well with the number of running child processes. For instance with 200,000 children over 8 partitions the shutdown time is reduced to ~30s but if you increase the number of children to 500,000 there is a lower bound of ~7s below which you can never go, no matter how many partitions. The problem is that the `PartitionSupervisor` terminates its children sequentially. So as you increase the number of partitions, you're just increasing the number of children that are terminated sequentially, even if each child `DynamicSupervisor` terminates its children in parallel. This PR solves that by replacing the top-level `PartitionSupervisor` with another `DynamicSupervisor`. On shutdown all partition supervisors are terminated in parallel and the children of those partition supervisors are also terminated in parallel, removing the bottleneck. Here are some numbers from my benchmark showing the time required to shutdown a supervisor (tree) with 200,000 running processes. Our larger servers have 16 cores, so we're running a `PartitionSupervisor` with 16 partitions. ``` ========================================= Partitioned (PartitionSupervisor with 16 partitions) ========================================= 200000 processes memory: 72.7734375KiB start: 2.6s shutdown: 12.4s max queue len: 12480 ``` So 12 seconds even with a very simple process with no `terminate/2` callback. We could just increase the number of partitions... ``` ========================================= Partitioned (PartitionSupervisor with 50 partitions) ========================================= 200000 processes memory: 276.609375KiB start: 2.7s shutdown: 5.4s max queue len: 3936 ``` Which is better but we've nearly tripled the number of supervisors but only just over halved the shutdown time, so you start to see the tradeoff. Now with the new 2-tier `DynamicSupervisor`: ``` ========================================= DynamicPartitioned (DynamicSupervisor of 50 DynamicSupervisors) ========================================= 200000 processes memory: 180.84375KiB start: 2.8s shutdown: 0.5s max queue len: 3763 ``` So 10x improvement on the previous config and 25x on the current setup. The number of partitions can be set using a new env var `ELECTRIC_CONSUMER_PARTITIONS`. If that's not set then the partitions scale by `max_shapes` if that's known. If not we just use the number of cores. This is the shutdown time for our production stack with the fallback partition config, so nearly a 6x improvement. I've opted for a conservative default (could have gone with some multiple of the number of cores) but went with the lower-memory option. ``` ========================================= DynamicPartitioned (DynamicSupervisor of 16 DynamicSupervisors) ========================================= 200000 processes memory: 58.09375KiB start: 2.8s shutdown: 2.1s max queue len: 12555 ``` This also scales to 500,000 shapes, where it starts 125 partitions by default and shuts down in 1.5s (the original version took 70s): ``` 500000 processes memory: 435.21875KiB shutdown: 1.5s ``` This is the benchmarking script: https://gist.github.com/magnetised/af2379e9b2028d4e82f442e22d7d20d5
## Summary - Fixed `char(n)` (bpchar) column values being trimmed of trailing spaces in snapshot and subset queries - The `::text` cast in `pg_cast_column_to_text/1` was stripping space padding from `char(n)` columns, causing inconsistency with values from PG replication which correctly preserve padding - Uses `pg_typeof()` at runtime to detect bpchar columns and `concat()` to preserve padding without trimming ## Test plan - [x] Added test with `CHAR(8)` PK and `CHAR(10)` column verifying space padding is preserved - [x] Test includes NULL `char(n)` column to verify NULL handling is correct - [x] All existing querying tests pass (15/15) - [x] All shapes tests pass (501/501) - [x] All plug/router tests pass (131/131) Fixes #4039 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: erik-the-implementer <erik-the-implementer@users.noreply.github.com>
…eives commit fragment with no relevant changes (#4064) ## Summary Fix #4063 When a `{Storage, :flushed, offset}` message arrives at a Consumer process while it's in the middle of processing a multi-fragment transaction, the flush notification could be lost. This caused the FlushTracker in ShapeLogCollector to get stuck waiting for a flush that was already completed. ### Changes **Consumer (`consumer.ex`, `consumer/state.ex`):** - Defer flush notifications that arrive during a pending transaction by storing the max flushed offset in `pending_flush_offset` - Process deferred flush notifications when the pending transaction completes (commit, skip, or no relevant changes) **FlushTracker (`flush_tracker.ex`):** - Simplify to commit-only tracking: remove non-commit fragment handling since Consumer now defers flush notifications during pending transactions - Remove `shapes_with_changes` parameter from `handle_txn_fragment/3` — all affected shapes are tracked uniformly at commit time **ShapeLogCollector (`shape_log_collector.ex`):** - Only call `FlushTracker.handle_txn_fragment/3` on commit fragments - Remove `shapes_with_changes` computation that is no longer needed ## Test plan - [x] New regression test for stuck flush tracker scenario - [x] Stricter assertions in EventRouterTest for txn fragment reslicing - [x] Updated Consumer tests for deferred flush notification behavior - [x] Existing test suite passes
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`electric-cursor` is now required and validated by the client so ensure it's always there, even if the value is nonsense.
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @electric-sql/experimental@6.0.14 ### Patch Changes - Updated dependencies [deb7c32] - @electric-sql/client@1.5.14 ## @electric-sql/react@1.0.43 ### Patch Changes - Updated dependencies [deb7c32] - @electric-sql/client@1.5.14 ## @electric-sql/client@1.5.14 ### Patch Changes - deb7c32: Add move-in event support to the TypeScript client. Rename `MoveOutPattern` to `MovePattern` (with a deprecated alias for backwards compatibility), extend `EventMessage` to accept both `move-out` and `move-in` events, and add `active_conditions` field to `ChangeMessage` headers. ## @electric-sql/y-electric@0.1.40 ### Patch Changes - Updated dependencies [deb7c32] - @electric-sql/client@1.5.14 ## expo-db-electric-starter@1.0.15 ### Patch Changes - Updated dependencies [deb7c32] - @electric-sql/client@1.5.14 ## @core/electric-telemetry@0.1.10 ### Patch Changes - 0aa8c00: Extend top processes by memory metric to collect processes until the specified mem usage threshold is covered. `ELECTRIC_TELEMETRY_TOP_PROCESS_COUNT` has been renamed to `ELECTRIC_TELEMETRY_TOP_PROCESS_LIMIT` with a new format: `count:<N>` or `mem_percent:<N>`. The old env var is still accepted as a fallback. - 0aa8c00: Group request handler processes together to see their aggregated memory usage. ## @core/elixir-client@0.9.4 ### Patch Changes - cb2c45e: Include required headers in client mock responses ## @core/sync-service@1.4.16 ### Patch Changes - 6c5068a: Fix stuck flush tracker when storage flush notification arrives mid-transaction in Consumer - 93e5d40: Fix typo in source event name - 64a89a0: Fixed char(n) column values being trimmed of trailing spaces in snapshot and subset queries, causing inconsistency with values from PG replication. - 8919ca3: Reclassify `branch_does_not_exist` error as retryable. PlanetScale returns this error transiently during cluster maintenance, and classifying it as non-retryable caused sources to be permanently shut down requiring manual restart. - d89be52: Improve shutdown times by changing the consumer supervision strategy - 0af96e9: Make in-memory shape db instances isolated - 461576d: Add known errors for pg authorization failures ## @electric-sql/docs@0.0.8 ### Patch Changes - d89be52: Document new ELECTRIC_CONSUMER_PARTITIONS environment variable Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix bot color collisions (purple reserved for humans) - Win threshold 20% → 30% - Bot rematch support and rejoin improvements Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Local color assignment, enclosure steals, agent personalities - Rematch clears board, win overlay shows threshold Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Slower move speed (200ms), leader blink, bots never freeze Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
published: false— outline to be prosed up by authorTest plan
/blog-reviewbefore publishing🤖 Generated with Claude Code