Skip to content

fix(destination): recreate an index with the source's own prefix lengths - #215

Merged
abnegate merged 2 commits into
mainfrom
fix/restore-index-lengths
Aug 5, 2026
Merged

fix(destination): recreate an index with the source's own prefix lengths#215
abnegate merged 2 commits into
mainfrom
fix/restore-index-lengths

Conversation

@abnegate

@abnegate abnegate commented Aug 5, 2026

Copy link
Copy Markdown
Member

The defect

validateFieldsForIndexes() derives every index length from the destination columnnull, or MAX_ARRAY_INDEX_LENGTH for an array column. The source's recorded prefix lengths are parsed into the Index resource and then never used: the class had no getLengths() at all.

An index often only fits under an adapter's byte limit because of that prefix. Two large string columns capped at 100 and 20 bytes are a perfectly valid index on the source and a 767-byte violation on the destination — so restoring a healthy database fails to recreate its own indexes:

Invalid index: Index length is longer than the maximum: 767

Why this matters

Measured against Appwrite Cloud production across all six regions: 3 separate customers in 3 regions hit exactly this signature, the most recent four days ago. One of them issued nine in-place restore attempts in eighty minutes, every one failing, before working around it (DAT-2113).

The fix

Lengths come from the resource being recreated. A recorded 0 — how a source records "no prefix" for a position, e.g. a short column that needs none — stays no-prefix rather than becoming a zero-length one. Array columns keep their MAX_ARRAY_INDEX_LENGTH override, which is a destination-side constraint rather than a source fact.

Tests

AppwriteIndexLengthsTest drives the real Appwrite destination through Transfer:

  • an index with source prefixes [100, 20] is recreated carrying them;
  • the same index without source lengths still fails the adapter's byte limit — the control case, proving the test exercises a genuinely limit-bound index rather than one that would fit anyway;
  • a recorded 0 stays no-prefix (asserted in the shape a live production row for this exact case carries, [100, 0], since the metadata collection types lengths as an integer array).

Verified red against main: 2 of the 3 fail without the change. Full unit suite green (67 tests), Pint and PHPStan clean.

One note for reviewers: the test transfers the schema and the index in two run() passes, because Transfer::GROUP_DATABASES_RESOURCES orders TYPE_INDEX before TYPE_COLUMN — the reverse of what a real source emits. A single pass would offer the index against a table with no columns yet and test the mock's ordering rather than production's.

🤖 Generated with Claude Code

validateFieldsForIndexes() derived every length from the destination column
instead of the index it was recreating: null, or the array cap for an array
column. The source's recorded prefix lengths were read into the Index resource
and then never used — the class had no getter for them at all.

An index only fits under an adapter's byte limit BECAUSE of that prefix. Two
large string columns capped at 100 and 20 bytes are a valid index on the
source and a 767-byte violation on the destination, so restoring a healthy
database failed to recreate its own indexes with 'Invalid index: Index length
is longer than the maximum: 767'.

Measured on production: 3 customers across 3 regions hit exactly this, most
recently four days ago, and one retried nine times in eighty minutes before
giving up (DAT-2113).

Lengths now come from the resource, with zero — how a source records 'no
prefix' for a position — staying no-prefix rather than becoming a zero-length
one. Array columns keep their MAX_ARRAY_INDEX_LENGTH override.

DAT-2113

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR now preserves source index prefix lengths during destination validation and compares those lengths when reconciling an existing index.

  • Adds Index::getLengths() so destinations can consume recorded source prefixes.
  • Uses positive source prefix lengths while retaining the destination-specific array-column override.
  • Adds migration tests for explicit prefixes, absent prefixes, zero sentinels, and overwrite reconciliation.

Confidence Score: 4/5

The PR is not yet safe to merge because unchanged array-column indexes can still be deleted and recreated on every eligible overwrite migration.

Index creation replaces an absent source prefix with MAX_ARRAY_INDEX_LENGTH for array columns, while the new comparator continues comparing persisted lengths against the unnormalized source value; the same index therefore appears different on every later overwrite, and recreation failure after deletion leaves the destination without it.

Files Needing Attention: src/Migration/Destinations/Appwrite.php

Important Files Changed

Filename Overview
src/Migration/Destinations/Appwrite.php Applies source prefix lengths during index creation and compares them during overwrite reconciliation, but array-column normalization remains inconsistent between creation and comparison.
src/Migration/Resources/Database/Index.php Exposes the existing lengths field through a typed getter for destination consumers.
tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php Covers source prefixes, missing and zero lengths, and ordinary length-only overwrite mismatches, but does not cover repeated reconciliation of array-column indexes.

Reviews (3): Last reviewed commit: "fix(destination): compare index prefix l..." | Re-trigger Greptile

Comment thread src/Migration/Destinations/Appwrite.php
…isting index

Greptile on this PR, and correct: indexSpecMatches() omitted lengths, so an
Overwrite migration onto an index whose ONLY difference was its prefix
treated it as already matching and skipped it, silently keeping the
destination's lengths -- reintroducing the very defect this PR fixes, on the
overwrite path instead of the create path.

The exclusion was reasonable when it was written: lengths were derived
per-column from the adapter and not carried on the resource, so comparing
them compared nothing. The source's own lengths drive them now, so a
length-only difference is a genuine mismatch. Positions are compared as
integers so 'no prefix' matches however it is spelled -- the resource records
0, the metadata collection reads back 0 or null.

The test asserts the decision rather than the recreate: the in-memory adapter
cannot drop an index the destination believes it holds, and that is an
adapter limit rather than the behaviour under test.

DAT-2113

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abnegate

abnegate commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

Greptile's finding is correct and now fixed.

indexSpecMatches() omitted lengths, so an Overwrite migration onto an index whose only difference was its prefix treated it as already matching and skipped it — silently keeping the destination's lengths, which reintroduces exactly the defect this PR fixes, on the overwrite path instead of the create path.

The exclusion was reasonable when it was written, and its comment said so: lengths were derived per-column from the adapter and not carried on the resource, so comparing them compared nothing. That is no longer true — the source's own lengths drive them now — so the comment and the behaviour both change. Positions compare as integers so "no prefix" matches however it is spelled (the resource records 0; the metadata collection reads back 0 or null).

On the test: it asserts the decision — that a length-only difference is no longer reported as STATUS_SKIPPED — rather than the full drop-and-recreate. The in-memory adapter cannot drop an index the destination believes it holds, which is an adapter limit rather than the behaviour under test; asserting around it would have been asserting an artifact. Verified red against the unfixed comparison.

Full unit suite green (68 tests), Pint and PHPStan clean.

Comment thread src/Migration/Destinations/Appwrite.php
@abnegate

abnegate commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

@greptileai review

@abnegate
abnegate merged commit e545aed into main Aug 5, 2026
4 checks passed
@abnegate
abnegate deleted the fix/restore-index-lengths branch August 5, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant