Skip to content

feat(fts): compose boosted compound scorers - #8094

Open
BubbleCal wants to merge 1 commit into
mainfrom
yang/oss-1598-composable-fts-scorer
Open

feat(fts): compose boosted compound scorers#8094
BubbleCal wants to merge 1 commit into
mainfrom
yang/oss-1598-composable-fts-scorer

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Feature

Stacked on #8093 (which is stacked on #8092). Review this PR as the Boost increment only.

What is the new feature?

Add posting-backed BoostQuery composition, including nested Boost nodes inside same-column Boolean/Phrase scorer trees.

Why do we need this feature?

Boost scoring subtracts a weighted negative score from the positive score. This can produce signed results, so ordinary non-negative upper-bound propagation is insufficient for exact bounded top-k execution.

How does it work?

  • BoostScorer is driven by the positive scorer and lazily confirms/scores the negative side on matching documents.
  • Signed score bounds conservatively subtract the negative range without underestimating the parent upper bound.
  • Competitive thresholds propagate to the positive side only when the negative scorer is known to be non-negative.
  • Match and Boost multipliers are rejected unless finite and non-negative.
  • Nested Boolean + Phrase + Boost execution is covered by the same-column end-to-end test.
  • Cross-column, incomplete-index, and stale-overlay shapes retain the exact fallback.

No performance improvement is claimed here; this PR validates exact bounded execution semantics.

Related: https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts

Validation

  • cargo check -p lance-index -p lance --lib
  • cargo test -p lance-index scalar::inverted::compound::tests --lib
  • cargo test -p lance dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded --lib
  • cargo test -p lance dataset::tests::dataset_index::test_nested_multimatch_limit_propagation --lib
  • cargo test -p lance io::exec::fts::tests::test_boolean_query_parts_searched_metrics --lib
  • cargo test -p lance dataset::scanner::test::test_fts_query_contract_rejects_invalid_values --lib
  • cargo clippy -p lance-index -p lance --lib --tests -- -D warnings
  • cargo fmt --all

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Jul 30, 2026
This was referenced Jul 30, 2026
Merged
Merged
@BubbleCal
BubbleCal force-pushed the yang/oss-1598-composable-fts-scorer branch from a1062d5 to 341d0ea Compare July 30, 2026 16:43
BubbleCal added a commit that referenced this pull request Jul 31, 2026
@BubbleCal Yang Cen
feat(fts): add composable scorer core for multimatch (#8092)
18d36a2 
## Feature

### What is the new feature?

Introduce the posting-backed compound FTS scoring core and use it for
bounded same-column `MultiMatchQuery` execution. The change adds the
internal `ComposableScorer` protocol, a cross-partition `TopKCollector`,
incremental `WandCursor` support, and `CompoundQueryExec` integration.

### Why do we need this feature?

Compound FTS currently falls back to DataFusion plans that remove the
query limit and materialize intermediate results. Leaf FTS already has
WAND/MAXSCORE bounds; this PR establishes the composable scorer and
collector layer needed to preserve exact top-k semantics while applying
the limit inside posting-backed execution.

### How does it work?

- Scorers expose ordered document iteration, exact scores, shallow score
bounds, competitive-score feedback, and two-phase confirmation.
- One collector owns the competitive threshold across all index
partitions and applies deterministic score/row-id tie ordering.
- Same-column, fully indexed, fresh `MultiMatchQuery` shapes use DisMax
over posting-backed leaves.
- Cross-column, incomplete-index, stale-overlay, and unsupported query
shapes retain the existing exact fallback.

## Stack

1. This PR: scorer core + same-column MultiMatch
2. #8093: Boolean + Phrase
3. #8094: Boost + nested composition

Related:
https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts

## Validation

- `cargo check -p lance-index -p lance --lib`
- `cargo test -p lance-index scalar::inverted::compound::tests --lib`
- `cargo test -p lance
dataset::tests::dataset_index::test_same_column_multimatch_uses_compound_scorer
--lib`
- `cargo test -p lance
dataset::tests::dataset_index::test_nested_multimatch_limit_propagation
--lib`
- `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings`
- `cargo fmt --all`

---------

Co-authored-by: Yang Cen <yang@lancedb.com>
@BubbleCal
BubbleCal force-pushed the yang/oss-1598-composable-fts-scorer branch from 341d0ea to 1650f6b Compare July 31, 2026 09:25
@BubbleCal
BubbleCal force-pushed the yang/oss-1598-composable-fts-scorer branch from 1650f6b to 664ba62 Compare July 31, 2026 10:05
Base automatically changed from yang/oss-1598-composable-fts-boolean-phrase to main July 31, 2026 11:19
BubbleCal added a commit that referenced this pull request Jul 31, 2026
@BubbleCal Yang Cen
feat(fts): compose boolean and phrase scorers (#8093)
662659f 
## Feature

Review this PR as the Boolean/Phrase increment only.

### What is the new feature?

Extend the composable FTS scorer tree with same-column `BooleanQuery`
and `PhraseQuery` execution.

### Why do we need this feature?

Boolean clauses need exact composition of required, optional, and
prohibited matches without materializing join intermediates. Phrase
clauses additionally require an approximation/confirmation split so
posting iteration remains cheap while positional checks stay exact.

### How does it work?

- Boolean SHOULD clauses use score sums; MUST clauses intersect
membership while preserving the existing first-MUST scoring contract;
MUST NOT clauses filter confirmed matches.
- Phrase leaves use the scorer's two-phase `matches()` hook for
positional confirmation.
- Sum and Boolean score bounds remain conservative before propagating
competitive thresholds.
- Phrase-containing compound queries require an index with positions.
- Cross-column, incomplete-index, stale-overlay, and Boost-containing
shapes continue to use the exact fallback in this layer.

## Stack

1. #8092: scorer core + same-column MultiMatch
2. This PR: Boolean + Phrase
3. #8094: Boost + nested composition

Related:
https://linear.app/lancedb/issue/OSS-1598/introduce-a-composable-scorer-interface-for-compound-fts

## Validation

- `cargo check -p lance-index -p lance --lib`
- `cargo test -p lance-index scalar::inverted::compound::tests --lib`
- `cargo test -p lance
dataset::tests::dataset_index::test_same_column_compound_scorer_is_exact_and_bounded
--lib`
- `cargo test -p lance
dataset::tests::dataset_index::test_nested_multimatch_limit_propagation
--lib`
- `cargo test -p lance
io::exec::fts::tests::test_boolean_query_parts_searched_metrics --lib`
- `cargo test -p lance
dataset::scanner::test::test_fts_query_contract_rejects_invalid_values
--lib`
- `cargo clippy -p lance-index -p lance --lib --tests -- -D warnings`
- `cargo fmt --all`

Co-authored-by: Yang Cen <yang@lancedb.com>
@BubbleCal
BubbleCal force-pushed the yang/oss-1598-composable-fts-scorer branch from 664ba62 to 43d0edf Compare July 31, 2026 11:19
@BubbleCal
BubbleCal marked this pull request as ready for review July 31, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

A-index Vector index, linalg, tokenizer enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@BubbleCal