You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
server: multi-choice completions report usage for only one choice #55
With n > 1, chat completions return all choices but count only one choice in usage.completion_tokens.
Confirmed on Spark Linux ARM64, CPU Release, upstream 95887577ab5fead779581a7030a83c7752ff3234, Qwen2.5-0.5B-Instruct Q4_K_M.
llama-server -m qwen2.5-0.5b-instruct-q4_k_m.gguf -ngl 0 -c 2048 -np 4 -t 2 --host 127.0.0.1 --port 18927
curl http://127.0.0.1:18927/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"x","messages":[{"role":"user","content":"Write a long, detailed paragraph about the ocean and its many creatures."}],"max_tokens":8,"temperature":0.8,"seed":1,"n":4}'
Observed: four distinct choices, all finish_reason:length; server logs confirm 8 generated tokens each (32 total). Response reports completion_tokens:8, prompt_tokens:43, total_tokens:51. Controls with n=1/2/4 all report completion_tokens=8.
Cause: non-streaming result aggregation appends choices into arr[0] without summing usage. Introducing commit not bisected.
Activity