Skip to content

MB-65018 add custom_filter/custom_score query nodes with context-driven callback hooks#2289

Merged
abhinavdangeti merged 19 commits intomasterfrom
MB-65018
Apr 7, 2026
Merged

MB-65018 add custom_filter/custom_score query nodes with context-driven callback hooks#2289
abhinavdangeti merged 19 commits intomasterfrom
MB-65018

Conversation

@maneuvertomars
Copy link
Copy Markdown
Member

This PR adds two new query node types in Bleve: custom_filter and custom_score, both wrapping an inner query.
Execution is embedding-driven via request-scoped context hooks (CustomFilterFactory / CustomScoreFactory) so Bleve remains engine-agnostic.
custom_filter applies per-hit keep/drop logic through FilteringSearcher; custom_score applies per-hit score mutation through ScoreMutatingSearcher.

@coveralls
Copy link
Copy Markdown

coveralls commented Feb 22, 2026

Coverage Status

coverage: 52.25% (-0.4%) from 52.691% — MB-65018 into master

Comment thread docs/custom_query_hooks.md Outdated
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_filter.go
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_hooks.go Outdated
Comment thread search/query/custom_score.go Outdated
@abhinavdangeti abhinavdangeti added this to the v2.6.0 milestone Feb 24, 2026
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_score.go Outdated
@abhinavdangeti
Copy link
Copy Markdown
Member

@maneuvertomars Let's make sure you scrub the proposed code once again for any nil handling, especially around pointer dereferencing.

Copy link
Copy Markdown
Member

@abhinavdangeti abhinavdangeti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maneuvertomars Let's do a quick walk through tomorrow.

Comment thread search_test.go Outdated
Comment thread search/query/custom_hooks.go Outdated
abhinavdangeti
abhinavdangeti previously approved these changes Mar 27, 2026
Copy link
Copy Markdown
Member

@abhinavdangeti abhinavdangeti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please wait for review from @CascadingRadium as well.

Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_hooks.go Outdated
Comment thread search/query/custom_hooks.go Outdated
Comment thread search/query/custom_hooks.go Outdated
Comment thread search/query/custom_hooks.go Outdated
Comment thread search/searcher/search_custom_score.go Outdated
@capemox
Copy link
Copy Markdown
Member

capemox commented Mar 30, 2026

@maneuvertomars please add functions to create new custom_filter/custom_score queries in query.go in bleve's root, and also add end to end tests in search_test.go to validate it works with some non-trivial custom filters and custom scorers.

Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_score.go Outdated
Comment thread docs/custom_query_hooks.md Outdated
Comment thread docs/custom_query_hooks.md Outdated
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_score.go Outdated
Comment thread search/query/custom_payload.go
Comment thread search/query/query.go Outdated
Comment thread search/searcher/search_score_mutator.go Outdated
Comment thread query.go Outdated
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_score.go Outdated
@abhinavdangeti
Copy link
Copy Markdown
Member

@maneuvertomars lets also capture the behavior in the .md when the user does not enable docValues for the necessary field mappings.

@abhinavdangeti
Copy link
Copy Markdown
Member

use doc values instead of stored fields for custom query field loading

Looks like you already covered this with your last patch 👍🏼

abhinavdangeti
abhinavdangeti previously approved these changes Apr 6, 2026
Comment thread search/query/custom_filter.go Outdated
Comment thread search/query/custom_score.go Outdated
Comment thread search/searcher/search_custom_score.go Outdated
Comment thread docs/custom_query.md
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class custom_filter and custom_score query nodes that allow embedders to inject per-hit filtering and score mutation logic, while keeping Bleve core engine-agnostic and enabling optional doc-values-backed field access within callbacks.

Changes:

  • Introduces CustomFilterQuery / CustomScoreQuery JSON node support (via pluggable parsers) and corresponding searchers.
  • Implements doc value loading + field-type-aware decoding to populate DocumentMatch.Fields before invoking callbacks.
  • Adds unit/integration tests and user-facing documentation for custom query behavior and requirements.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
search/searcher/search_custom_score.go Adds a searcher wrapper that loads doc values and mutates hit scores via a callback.
search/searcher/search_custom_filter.go Adds a searcher wrapper that loads doc values and applies a per-hit accept/drop callback.
search/searcher/search_custom_fields.go Adds shared helpers to load doc values and decode them into typed DocumentMatch.Fields.
search/query/query.go Extends ParseQuery to recognize custom_filter / custom_score nodes and delegate to embedder-registered parsers.
search/query/custom_score.go Adds CustomScoreQuery type, JSON marshaling, validation, and searcher construction.
search/query/custom_filter.go Adds CustomFilterQuery type, JSON marshaling, validation, and searcher construction.
search/query/custom_payload.go Adds shared payload unmarshaling and mapping-type resolution utilities.
search/query/custom_query_test.go Adds JSON marshal/unmarshal tests for both custom query types.
search_test.go Adds end-to-end tests verifying filtering/scoring behavior and doc-values field decoding.
docs/custom_query.md Adds documentation describing custom query nodes, payload semantics, and doc-values requirements.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread search/searcher/search_custom_fields.go Outdated
Comment thread search/searcher/search_custom_fields.go Outdated
Comment thread search/query/custom_filter.go
Comment thread search/query/custom_score.go
Comment thread docs/custom_query.md Outdated
Comment thread search/query/custom_filter.go Outdated
Comment thread docs/custom_query.md Outdated
Comment thread search/searcher/search_custom_score.go Outdated
@capemox
Copy link
Copy Markdown
Member

capemox commented Apr 7, 2026

Few more tiny changes @maneuvertomars, otherwise looks alright to me!

@capemox
Copy link
Copy Markdown
Member

capemox commented Apr 7, 2026

Also @maneuvertomars please do click "resolve conversation" for the comments that you've taken care of, it's easier for everyone

@abhinavdangeti abhinavdangeti dismissed CascadingRadium’s stale review April 7, 2026 16:50

Addressed comments, approved by others.

@abhinavdangeti abhinavdangeti merged commit 8a44c37 into master Apr 7, 2026
10 checks passed
@abhinavdangeti abhinavdangeti deleted the MB-65018 branch April 7, 2026 16:50
@github-project-automation github-project-automation Bot moved this from Todo to Done in Script Queries Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants