fix: validate primary key nullability on every path that can change it

[xuanyu-z]

Schema::verify_primary_key rejects a primary key column that permits nulls, but its only caller is TryFrom<&ArrowSchema> for Schema — table creation and nothing else.

It matters because a multi-column key match treats a null as unequal to everything, including another null. A row holding a null in its key never matches its own earlier copy, so merge_insert inserts a second row rather than overwriting, and every repeat write adds another. MemWAL compaction folds flushed data in by matching on the key, so such a table duplicates rows silently.

Enforced where a schema becomes durable

The check goes in write_manifest_file, which every manifest write funnels through. Four kinds of path reach a manifest without the Arrow-schema conversion:

Path	Why it escapes
UpdateConfig field metadata	installs the key by writing metadata onto a field
ColumnAlteration::set_nullable(true)	changes an existing key column's nullability
Restore and clone	rebuild a manifest from a stored one; Operation::Clone carries no schema at all
Operation::Project / Merge / Overwrite	public, and validate_operation checks the schema only against fragments

Enumerating them per operation would leave the next schema-carrying operation uncovered. One check at the manifest boundary does not.

Early errors

Three API-level checks report the same violation before work is done — installing the key through field metadata, altering nullability, and initializing MemWAL on a table already carrying a bad key. InitializeMemWalBuilder::execute is also the gate for callers outside this crate, which is why it is checked there rather than by each caller.

Tests

Four, each verified to fail with its own check removed:

• write_manifest_file_rejects_a_nullable_primary_key — forges a manifest, since a stored one can no longer be produced. Restore and clone are covered by routing through this call.
• test_unenforced_primary_key_rejects_a_nullable_column
• test_alter_columns_cannot_make_a_primary_key_nullable — also alters a non-key column, so the rejection is specific to the key
• test_initialize_mem_wal_rejects_a_nullable_primary_key

Four tests in dataset/metadata.rs installed a key on a gen_batch column, which is nullable. Their subject is metadata translation and key immutability, so they now build from an explicit non-nullable schema. Clustering-key tests are untouched — verify_primary_key does not govern them.

Full lance --lib suite: 2630 pass.

Not addressed here

Casting a primary key column rebuilds the field from a freshly constructed Arrow field carrying no metadata, which drops the key marker. Unchanged by this; verify_primary_key then passes because the schema has no key.

Related

#6324 — a sibling symptom of the same gap, about protobuf population on the metadata path rather than validation.

[codecov]

Codecov Report

❌ Patch coverage is 97.24771% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/schema_evolution.rs	91.66%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!