Add off-graph KV-cache ring policy

[kiymetakdemir]

Summary

Adds the sliding-window (ring) layout policy. RingPolicy keeps a layer's last window tokens in a fixed ring of window + max_write − 1 slots — oversized so a multi-token prefill step fits without clobbering context earlier queries in the step still need — and evicts the oldest. SequenceCache dispatches windowed layers to it, so a mixed model (gemma4's alternating full/sliding-window layers) shares one logical length. Backend- and tensor-free: it emits integer write/read runs; the per-query window mask stays in the backend.

Files

• extension/llm/cache/cache.h — LayerPolicy::Kind::Ring; max_write on CacheConfig (max tokens/step; default 1 = decode-only ring).
• extension/llm/cache/sequence_cache.h — split_ring_runs (wrap into ≤2 runs), RingPolicy over the oversized ring (retained_from still window-based), the
  make_policy ring branch, and a ring-specific T ≤ max_write guard.
• extension/llm/cache/test/cache_test.cpp — ring wrap/evict, mixed flat/ring shared length, rewind bounded by the window, and step-over-max_write rejected.

Testing

C++ unit tests for the ring path (wrap, eviction, mixed flat/ring, window-bounded rewind, max_write guard). Built and ran via the standard flow, all pass:

cmake -B cmake-out -DEXECUTORCH_BUILD_EXTENSION_LLM=ON -DEXECUTORCH_BUILD_TESTS=ON
cmake --build cmake-out --target extension_llm_cache_test -j
ctest --test-dir cmake-out -R extension_llm_cache --output-on-failure

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21447

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 984f7b2 with merge base 8001512 ():

NEW FAILURES - The following jobs have failed:

• pull / unittest / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 65ee4b2bb370071ec1c730253a4457737671aa4e15b1eb3d7414085ceba97c6b /exec failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  RuntimeError: Command docker exec -t c00e922038125b3c390f03f6cb616d3fc3c5bb40f82f6b0e4bb3132e766b6c74 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[metascroy]

This should probably default to something bigger for prefill

[kiymetakdemir]

It is window size in MLX (the field is optional; unset defaults to window). I can change to that or let me know if you have any other suggestion.

[metascroy]

Private method on policy?

[kiymetakdemir]

Makes sense, changing it.