Skip to content

server: concurrent router loads can leave one model queued forever #36

Closed

Description

@am17an

Upstream: ggml-org#28443

With router mode and --models-max 1, two simultaneous requests for unloaded models can leave one request queued indefinitely.

Environment: current main 465e49b9, Linux aarch64, CUDA. Three presets A/B/C used the same Gemma 4 model file to isolate scheduler behavior.

Reproduction:

  1. Request A and let it become idle.
  2. Request B and C simultaneously.
  3. Apply a 25-second client timeout.

Observed: B completed in 3.06 seconds; C timed out after 25 seconds with zero bytes. Both waiters selected and attempted to unload A, but only B claimed the freed slot and C remained queued.

The affected LRU scheduler path was introduced in dff15d4a.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingserver

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions