Skip to content

Bloom-filter fast-path for the dense tag store (O(1) insertion) (phase 2)#11900

Draft
dougqh wants to merge 1 commit into
dougqh/dense-storefrom
dougqh/dense-bloom-filter
Draft

Bloom-filter fast-path for the dense tag store (O(1) insertion) (phase 2)#11900
dougqh wants to merge 1 commit into
dougqh/dense-storefrom
dougqh/dense-bloom-filter

Conversation

@dougqh

@dougqh dougqh commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What

Adds a per-map bloom-style presence filter (one long, knownBloom) to OptimizedTagMap's dense known-tag store. A clear bit means a known tag is definitely absent, so an append skips the O(n) knownIndexOf linear scan — the common per-span-build case (each tag set once) becomes O(1).

Correctness — superset invariant

The filter is a superset of the present ids' bits: set on every add, never cleared on remove (a stale-set bit only costs an unnecessary scan, never a wrong answer). The linear scan stays authoritative — a set bit means "maybe present, scan to confirm." So correctness is independent of the position→bit collision rate; only the fast-path hit rate depends on it. All 339 TagMap + dense fuzz tests pass.

Position→bit mapping

Crude 1L << (fieldPos & 63) to start. fieldPos (not globalSerial) is deliberate: it's the axis a future per-type graph-coloring optimizes — coloring co-occurring tags to distinct positions raises the fast-path hit rate without touching correctness.

Why (evidence)

On the stacked id-comparison benchmark, the bloom brings dense id-insertion to HashMap parity at typical tag counts (0.99× at 7 tags, up from 0.91× without it), while dense still allocates ~half of HashMap. Cost: +8 B/op (the single long field). The demonstrating benchmark lands in the stacked id PR.

Stacking

Dense-store id line: #11814 (Entry-less storage) → this (bloom fast-path) → id API + comparison benchmark. Based on dougqh/dense-store.

Follow-ups

Per-type graph coloring (raises hit rate); multi-level transitive prune for read-through (skip parents whose bit is clear).

🤖 Generated with Claude Code

@dougqh dougqh added comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring labels Jul 9, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.00 s 13.90 s [-0.0%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent 12.88 s 12.94 s [-1.2%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 16.86 s 16.83 s [-0.8%; +1.1%] (no difference)
startup:petclinic:iast:Agent 16.91 s 16.96 s [-1.2%; +0.5%] (no difference)
startup:petclinic:profiling:Agent 16.32 s 16.87 s [-7.8%; +1.2%] (no difference)
startup:petclinic:sca:Agent 16.43 s 16.91 s [-7.2%; +1.5%] (no difference)
startup:petclinic:tracing:Agent 16.17 s 16.04 s [-0.1%; +1.8%] (no difference)

Commit: 75950bee · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jul 15, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 92.86%
Overall Coverage: 57.32% (-0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 75950be | Docs | Datadog PR Page | Give us feedback!

@dougqh
dougqh force-pushed the dougqh/dense-store branch from a5e12e4 to 6c70848 Compare July 15, 2026 21:23
@dougqh
dougqh force-pushed the dougqh/dense-bloom-filter branch from 267ebad to 56550cb Compare July 15, 2026 21:23
@dougqh
dougqh force-pushed the dougqh/dense-store branch from 6c70848 to 0c0d740 Compare July 20, 2026 16:49
…scan)

A per-map long bitmask (knownBloom) over the dense store: a set bit means a
tagId MAY be present (scan to confirm), a clear bit means DEFINITELY absent —
so the common per-build insert skips the linear knownIndexOf scan and appends
in O(1). Crude position->bit map (fieldPos & 63); a collision-minimizing
per-type coloring later only raises the hit rate — correctness never depends
on it because the scan stays authoritative. Superset semantics: set on add,
never cleared on remove (a stale bit costs a scan, never a wrong answer).

Alloc-neutral (one long field, no extra allocation); the win is insertion CPU,
moving the dense store toward HashMap insertion parity without the scan.

Reconciled onto the folded-class dense store (#11814).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/dense-bloom-filter branch from 56550cb to 75950be Compare July 20, 2026 16:51
@dougqh dougqh changed the title Bloom-filter fast-path for the dense tag store (O(1) insertion) Bloom-filter fast-path for the dense tag store (O(1) insertion) (phase 2) Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant