This is a tracking issue for adding skipping indexes on top of the existing zoned layout, using the per-zone summary machinery already in vortex-layout (layouts/zoned) and vortex-array (aggregate_fn, stats/rewrite).
Motivation
Parent epic: #8900
Design
ZonedLayout already pairs a data child with a per-zone stats table and prunes by running Expression::falsify(predicate) over that table (ZoneMap::prune), then expanding the per-zone skip mask in ZonedReader::pruning_evaluation. This should be sufficient for a skipping index.
Adding an index on this path is two registrations, plus a third only when the summary needs custom test logic:
- A serializable aggregate for the per-zone summary (
AggregateFnVTable).
- A
StatsRewriteRule that lowers the query predicate into a proof over the summary.
- If the test is not expressible with builtin comparisons (e.g. a bloom membership test), a probe scalar function (
ScalarFnVTable) that the rule applies to the stored summary. Min/max-style indexes don't need this.
Scans consume these indexes automatically: the pushed-down filter drives ZonedReader::pruning_evaluation in the existing prune/filter/project loop, so no query plan changes are needed.
Note that on read, an unknown aggregate ID currently hard-errors (zoned/schema.rs::to_aggregate_fn, no allows_unknown fallback, unlike aggregate_fn/proto.rs::from_proto), so a custom index is currently a hard read-dependency rather than an ignorable hint.
A later phase should bundle these registrations behind a single interface so that adding a skipping index is one implementation instead of several coordinated registrations that are easy to half-wire (store the aggregate but forget the rewrite rule, and it silently prunes nothing). The exact shape (one trait vs. composable seams) is the epic's open interface question (#8900); this issue tracks implementing whatever is chosen on top of the zoned layout.
Steps
Unresolved questions
Implementation history
This is a tracking issue for adding skipping indexes on top of the existing zoned layout, using the per-zone summary machinery already in
vortex-layout(layouts/zoned) andvortex-array(aggregate_fn,stats/rewrite).Motivation
Parent epic: #8900
Design
ZonedLayoutalready pairs a data child with a per-zone stats table and prunes by runningExpression::falsify(predicate)over that table (ZoneMap::prune), then expanding the per-zone skip mask inZonedReader::pruning_evaluation. This should be sufficient for a skipping index.Adding an index on this path is two registrations, plus a third only when the summary needs custom test logic:
AggregateFnVTable).StatsRewriteRulethat lowers the query predicate into a proof over the summary.ScalarFnVTable) that the rule applies to the stored summary. Min/max-style indexes don't need this.Scans consume these indexes automatically: the pushed-down filter drives
ZonedReader::pruning_evaluationin the existing prune/filter/project loop, so no query plan changes are needed.Note that on read, an unknown aggregate ID currently hard-errors (
zoned/schema.rs::to_aggregate_fn, noallows_unknownfallback, unlikeaggregate_fn/proto.rs::from_proto), so a custom index is currently a hard read-dependency rather than an ignorable hint.A later phase should bundle these registrations behind a single interface so that adding a skipping index is one implementation instead of several coordinated registrations that are easy to half-wire (store the aggregate but forget the rewrite rule, and it silently prunes nothing). The exact shape (one trait vs. composable seams) is the epic's open interface question (#8900); this issue tracks implementing whatever is chosen on top of the zoned layout.
Steps
allows_unknown/foreign into the zoned read path, or require ignorable third-party indexes to be their own layout: Bypass unknown aggregates inZonedLayout#8904Unresolved questions
allows_unknown/foreign intozoned/schema.rs::to_aggregate_fn, and what state dtype would a foreign aggregate present?falsifyrewrite, and thus needs a different hook?Implementation history
ZonedLayout#8904