You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
server: concurrent router loads can leave one model queued forever #36
With router mode and --models-max 1, two simultaneous requests for unloaded models can leave one request queued indefinitely.
Environment: current main 465e49b9, Linux aarch64, CUDA. Three presets A/B/C used the same Gemma 4 model file to isolate scheduler behavior.
Reproduction:
Request A and let it become idle.
Request B and C simultaneously.
Apply a 25-second client timeout.
Observed: B completed in 3.06 seconds; C timed out after 25 seconds with zero bytes. Both waiters selected and attempted to unload A, but only B claimed the freed slot and C remained queued.
The affected LRU scheduler path was introduced in dff15d4a.
Activity