You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
server: id_slot pinned to an empty slot skips the prompt cache #34
Pinning a request to an empty slot bypasses an exact prompt-cache match and reprocesses the entire prompt.
Environment: current main 465e49b9, Linux aarch64, Gemma 4 12B Q4_K_M, --parallel 2 --slots --cache-ram 4096.
Reproduction:
Send prompt A with cache_prompt:true,id_slot:0.
Send prompt B with the same options, moving A to the prompt cache.
Send A with cache_prompt:true,id_slot:1.
Observed on step 3: cache_n:0, prompt_n:290.
For an empty pinned slot, f_keep = 0 / 0 becomes NaN, so f_keep < 0.5f is false and the cache is not loaded. The host-memory prompt-cache path dates to d00cbea6.
Activity