Skip to content

fix(storage): add explicit split-store consolidation#425

Merged
ScriptedAlchemy merged 5 commits into
masterfrom
codex/explicit-store-consolidation
Jul 10, 2026
Merged

fix(storage): add explicit split-store consolidation#425
ScriptedAlchemy merged 5 commits into
masterfrom
codex/explicit-store-consolidation

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Summary

  • add an explicit offline dry-run/apply workflow for consolidating two non-empty profile shards without dropping either input
  • freeze SQLite families without source mutation, block open holders, hold write reservations, back up both stores, and cut markers over only after exhaustive verification
  • preserve sessions, LCM payloads, facts, feedback, branches, dirty sentinels, and forensic artifacts across moves and symlinked checkouts
  • surface the exact recovery command from identity split diagnostics

Safety

  • deterministic confirmation token is recomputed under locks
  • ledger and staging writes are atomic and restartable at every durable state
  • unsupported open-holder discovery fails closed
  • current schema tables have explicit merge/rebuild/reject dispositions

Verification

  • cargo fmt --check
  • cargo clippy --lib -- -D warnings
  • cargo test --lib migrate::consolidate::tests
  • cargo test --bin tracedecay migrate_commands_parse_manifest_scaffolding_flags
  • cargo test --test storage_suite corrupt_nonempty_cutover_store_reports_both_shards_without_switching
  • cargo test --test storage_suite verify_manifest_accepts_logically_equal_sqlite_artifacts_with_different_bytes

@changeset-bot

changeset-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d3bb28b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e9756090e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/migrate/consolidate/sqlite.rs Outdated
@ScriptedAlchemy ScriptedAlchemy force-pushed the codex/explicit-store-consolidation branch from 3e97560 to 2297c45 Compare July 10, 2026 06:10
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2297c454f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/migrate/consolidate/sqlite.rs Outdated
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 82cfa9b9bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +169 to +173
trust_score = CASE WHEN COALESCE((
SELECT s.last_feedback_at FROM source.memory_facts s WHERE s.content = t.content
), -1) > COALESCE(t.last_feedback_at, -1) THEN (
SELECT s.trust_score FROM source.memory_facts s WHERE s.content = t.content
) ELSE t.trust_score END,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve newer explicit trust updates

When the duplicate source fact was changed through MemoryStore::update_fact with a new trust but no feedback, last_feedback_at stays null while updated_at advances. This merge only imports source trust when the source feedback timestamp is newer, so it keeps the target's stale trust even though the source is the newer fact; that silently drops legitimate trust edits during consolidation.

Useful? React with 👍 / 👎.

Comment on lines +245 to +249
INSERT OR IGNORE INTO memory_fact_entities(fact_id, entity_id)
SELECT fm.target_id, em.target_id
FROM source.memory_fact_entities sfe
JOIN consolidation_fact_map fm ON fm.source_id = sfe.fact_id
JOIN consolidation_entity_map em ON em.source_id = sfe.entity_id;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recompute fact vectors after entity merges

When source-only entity links are inserted for an overlapping fact, the fact's hrr_vector remains whatever the target shard had. The later bank rebuild uses the existing vector bytes rather than recomputing them, so semantic/vector retrieval ignores the source entities after consolidation; this affects overlaps where the two shards learned different entities for the same fact content.

Useful? React with 👍 / 👎.

@ScriptedAlchemy ScriptedAlchemy merged commit de3d05d into master Jul 10, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant