feat: offload idle encoder from VRAM to RAM after configurable timeout#227
Merged
Conversation
The cached search encoder pinned multiple GB of VRAM indefinitely, blocking other GPU workloads from sharing the device while PhotoMap idled. Move the model to system RAM after a period of inactivity so the next query can reload it in well under a second instead of paying the Hub fetch / tensor allocation cost of a fresh build. - New ``encoder_idle_timeout_seconds`` Config field (default 30s, 0 disables) - ``ImageTextEncoder.offload()`` / ``_ensure_on_device()`` with a per-instance RLock so the watcher can't yank weights mid-encode - Daemon-thread watcher started/stopped via FastAPI lifespan; CLI tools and tests aren't burdened with it Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
encoder_idle_timeout_seconds(default30, set to0to disable) of search inactivity, a background watcher moves the model from VRAM to system RAM. The next query auto-reloads it onto the GPU in well under a second — no Hub fetch, no tensor re-allocation.ImageTextEncoder.offload()/_ensure_on_device()are guarded by a per-instanceRLockso the watcher can't yank weights out from under an in-flightencode_*call.lifespan, so CLI tools and tests aren't burdened with a background thread they don't need. Indexing path is unchanged — it already builds and closes its own short-lived encoder.Test plan
pytest tests/backend— 207 passed (12 new tests covering offload/reload, watcher behavior,timeout=0short-circuit, cache-timestamp refresh, config field default + YAML round-trip)ruff check photomap tests— cleannvidia-smithat VRAM usage drops ~30s after the last query and is reclaimed by the next query🤖 Generated with Claude Code