feat: introduce MemWAL regional writer and MemTable reader

[touch-of-grey]

Based on draft shared from @jackye1995 , cleanup and publish for review.

Currently regional writer can reach 300MB/s performance on Amazon S3, around 50MB/s under high backpressure based on the benchmark.

Memtable read performance is inline with Lance table with a single fragment in memory, and much faster than using Lance table and append 1 fragment per batch.

[jackye1995]

Thanks for the fast turnaround! I will take a look tonight. Meanwhile, I think the code path deserves some benchmark, can you add that?

[codecov]

Codecov Report

❌ Patch coverage is 73.49033% with 2015 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ce/src/dataset/mem_wal/memtable/scanner/builder.rs	47.72%	362 Missing and 17 partials ⚠️
...rc/dataset/mem_wal/memtable/scanner/exec/vector.rs	0.94%	313 Missing ⚠️
rust/lance/src/dataset/mem_wal/index/ivf_pq.rs	64.58%	185 Missing and 8 partials ⚠️
rust/lance/src/dataset/mem_wal/index.rs	64.28%	155 Missing and 10 partials ⚠️
...e/src/dataset/mem_wal/memtable/scanner/exec/fts.rs	56.90%	143 Missing and 10 partials ⚠️
rust/lance/src/dataset/mem_wal/memtable.rs	78.51%	117 Missing and 13 partials ⚠️
rust/lance/src/dataset/mem_wal/manifest.rs	69.70%	84 Missing and 19 partials ⚠️
rust/lance/src/dataset/mem_wal/index/fts.rs	92.69%	95 Missing and 6 partials ⚠️
rust/lance/src/dataset/mem_wal/memtable/flush.rs	88.83%	47 Missing and 47 partials ⚠️
rust/lance/src/dataset/mem_wal/wal.rs	76.81%	69 Missing and 14 partials ⚠️
... and 6 more

📢 Thoughts on this report? Let us know!

[touch-of-grey]

All review comments have been addressed in commit cda38c0:

1. ✅ Removed rename_if_not_exists/copy_if_not_exists wrappers - using existing object_store APIs directly
2. ✅ Fixed "Arrow field metadata" typo
3. ✅ Changed region_specs: Vec<RegionSpec> to region_spec: Option<RegionSpec>, added TODO for add_region_spec() API
4. ✅ Moved load_vector_index_config and load_ivf_pq_components to standalone functions
5. ✅ Renamed indexed_writes to sync_indexed_write for clarity
6. ✅ Added comment clarifying that version hint failures are logged as warnings
7. ✅ Differentiated local vs cloud manifest writes (local uses temp file + atomic rename, cloud uses PUT-IF-NOT-EXISTS)
8. ✅ Made manifest_scan_batch_size configurable in RegionWriterConfig with default of 2

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[jackye1995]

thanks for bearing with all my comments and direct edits! I think this looks good now! We still need replay feature during initialization and also better alignment with the Lance scanner, but this is a great foundation. Pending CI to merge

[jackye1995]

Known CI failure, merging