Skip to content

server: multi-choice completions report usage for only one choice #55

Open

Description

@am17an

Upstream: ggml-org#29451

With n > 1, chat completions return all choices but count only one choice in usage.completion_tokens.

Confirmed on Spark Linux ARM64, CPU Release, upstream 95887577ab5fead779581a7030a83c7752ff3234, Qwen2.5-0.5B-Instruct Q4_K_M.

llama-server -m qwen2.5-0.5b-instruct-q4_k_m.gguf -ngl 0 -c 2048 -np 4 -t 2 --host 127.0.0.1 --port 18927
curl http://127.0.0.1:18927/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"x","messages":[{"role":"user","content":"Write a long, detailed paragraph about the ocean and its many creatures."}],"max_tokens":8,"temperature":0.8,"seed":1,"n":4}'

Observed: four distinct choices, all finish_reason:length; server logs confirm 8 generated tokens each (32 total). Response reports completion_tokens:8, prompt_tokens:43, total_tokens:51. Controls with n=1/2/4 all report completion_tokens=8.

Cause: non-streaming result aggregation appends choices into arr[0] without summing usage. Introducing commit not bisected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingserver

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions