server: auto-cache slot KV for repeated system prompts — worth a PR?

[Tongas]

Adds automatic disk-based KV cache for repeated system prompts in
llama-server: on first request the slot KV is saved to --kv-cache-path;
on subsequent requests it is restored before dispatch, skipping the
system-prompt prefill entirely.

condition	prompt_ms	speedup
cold prefill	3267 ms	—
cache hit	97 ms	33.5×

RTX 3090, Qwen3.6-27B Q4_K_M, 4185-token system prompt.
Cache file ~415 MB, restore overhead ~37 ms.

• opt-in via --kv-cache-path PATH, off by default
• no new dependencies (pure C++ SHA-256, no OpenSSL)
• zero impact on existing users if flag not set
• graceful fallback to normal request if no idle slot available
• works for both streaming and non-streaming requests

I have a working implementation (6 files, 359 lines added).
Happy to open a PR if this direction is welcome.

— Gastón Parravicini

[black-snow]

I'm looking for that!

[zacaj]

This sounds handy, but how does your impl know what portion is the system prompt? Is it just literally the portion marked system?

I often see cases where there's lots more to the 'standard' prompt (eg, AGENTS.md) that is categorized as user, not system, but I'd really want to cache it too. my system prompt is only 1-2k tokens but if you add in all the other boilerplate it's more like 15k, and it'd be useful to cache all of it. It can sometimes take over a minute for that all to process before the model even starts generating its response, and all of it is lost if you change folders, restart llama.cpp, or invalidate your KV cache somehow.

Is your impl posted somewhere? Would like to take a look and maybe try it out on my local fork.

[MatthewLeibel]

On the AGENTS.md question: the stock --slot-save-path option with the /slots/{id}?action=save and action=restore endpoints already saves and restores the whole cached prefix of a slot, system prompt and user content alike, so a long first message is covered too. The catch is that it is per slot and by hand: nothing calls it for you, and a file only works with the same model and settings.

Related open work that would automate it upstream: #29738 (session-keyed save and restore) and #28092 (--cache-dir).

Disclosure: I make RAMen (https://getramenai.com), a layer in front of llama-server that does the saving automatically. It finds a session by its system prompt plus first user message, so AGENTS.md-style content in the first message is included.