RAMen: automatic slot save and restore for every chat and agent session, on top of llama-server

[MatthewLeibel]

Hi all. I built a small layer on top of llama-server and wanted to share how it works, since it leans on llama.cpp features that most chat apps never call.

What it does

• Saves the KV cache of every conversation to disk with llama-server's slot save and restore, so a long chat or agent session comes back after a restart instead of being read again from the first token.
• Finds a conversation by an x-session-id header, or by its start (model, tools, system prompt, first user message).
• Checks that a saved file matches the model and settings before restoring it, and refuses otherwise.
• Lets many chats share a few live slots: idle sessions wait on disk until their next message arrives.
• For long chats, it also uses a selective attention path in the engine that reads only the parts of the cache that matter for each new token.

It sits under anything that speaks the OpenAI API (Cline, Continue, Aider, OpenCode, Open WebUI and others); ramen connect <app> prints the setup for each.

The free way to do the first part by hand, no RAMen needed: start llama-server with --slot-save-path ./slots --port 8080, then after a long chat

curl -X POST "http://127.0.0.1:8080/slots/0?action=save" -H "Content-Type: application/json" -d '{"filename": "chat.bin"}'

and the same call with action=restore after a restart, before the next message. Same model and settings only, one slot per file.

Limits, honestly

• Only OpenAI-style /v1/chat/completions and /v1/completions requests are saved. Anthropic-style /v1/messages (Claude Code) and /v1/responses (Codex) pass through unsaved.
• Hybrid and sliding-window models: see Misc. bug: /slots restore yields no KV reuse on hybrid/recurrent and SWA models (context checkpoints are not persisted) #28194. Our tests so far are Qwen3 4B, 32B and 30B-A3B plus one sliding-window model.
• NVIDIA builds start at RTX 20 (Turing); AMD GPUs run on the CPU for now.
• It is closed source, built on llama.cpp (MIT). The free plan keeps the last 3 chats; Pro is $3.99 a month. I'm the developer.

It helps most on long chats; short chats see little difference. Method and measurements: https://getramenai.com/how-it-works.html

Site and install: https://getramenai.com

Happy to answer questions, and thank you to everyone who works on llama.cpp.