Repository navigation
Add opt-in request batching to the inference runtime - #6
Conversation
max_batch_size (--max-batch-size, JEVANY_MAX_BATCH_SIZE) lets concurrent text requests share one forward pass. Batching is group commit: whichever caller holds the model lock scores every queued request, so an idle server still scores each request alone and adds no waiting time. A batch never allocates more padded tokens than max_packed_tokens. Media and prefix-cache requests run alone; a failed batch is retried per request so an error stays with the request that caused it. The default of 1 keeps the existing single-request path. On one H200 with 32 concurrent requests to a Qwen3.5-0.8B checkpoint, batching raised throughput 2.3x in FP32 (max probability change 1.5e-5, no decision changed) and 4.6x in BF16, where batch-shape rounding changed decisions on 2 of 32 requests; the docs recommend the default for reproducing metrics.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Human Review
Human Review: see the findings and review evidence. AI semantic review has not completed.
Reviewed head |
|
This still conflicts in |
Preserve context-length errors and tokenizer/cache locking while combining queued text requests. Cover mixed Python/HTTP callers, batch and token budgets, per-request failure recovery and multimodal fallback. Co-authored-by: Xuying Ning <xuying@meta.com>
ZhiningLiu1998
left a comment
ZhiningLiu1998
left a comment
There was a problem hiding this comment.
Integrated batching with current context-length errors and cache/tokenizer locking. The 85 focused offline checks pass, including concurrent Python/HTTP parity and failure isolation; both full CPU CI jobs and site checks pass.
2 participants
Concurrent text requests can share a forward pass when
max_batch_sizeis greater than 1. Requests are grouped within the batch-size and padded-token limits; an idle runtime adds no collection delay. Prefix-cache requests and multimodal checkpoints retain serial scoring.The revision preserves current
ContextLengthErrorresponses and tokenizer/cache locking, and retries failed batches per request. Tests cover mixed Python/HTTP callers, packed and row backbones, batch and token limits, failure isolation, cache recovery, and multimodal fallback.Validation: 85 offline checks passed; 7 CUDA cases skipped on the local CPU host. Both full CPU CI jobs and the website checks pass on the revised head. Original contributor commits are retained.