Find out what slowed down your vLLM requests, test a change, and compare the results.
llmtrace records requests, scheduler steps and GPU telemetry. It helps you see where requests waited, what work they shared, and whether a configuration change helped across repeated runs.
Use it for research and testing before production. It targets vLLM 0.11.0.
Full scheduler tracing needs the engine core in the same process:
VLLM_ENABLE_V1_MULTIPROCESSING=0. With the default multiprocess core or
AsyncLLM, fewer signals are available. llmtrace doctor explains which ones.
From a local checkout:
pip install -e .
llmtrace workload template --output w.json
llmtrace run --workload w.json --engine fake --out ./runs/base --repeat 2
llmtrace findings ./runs/base/r0 --verbose
llmtrace visualize ./runs/base/r0 --html-out report.htmlThe fake engine demonstrates the workflow. Its timings and power values are invented, so they tell you nothing about real hardware.
Follow the quickstart to test a configuration change or run on a GPU.
- Record:
runreplays a workload and saves traces, settings and health records. - Investigate:
findingsshows evidence for possible bottlenecks. - Test:
plansuggests changes;run --plantries them in fresh engines. - Compare:
decidechecks your latency target across independent repeats.
For example, a bursty Qwen2.5-7B workload was waiting behind a limit on concurrent requests. Doubling that limit reduced p95 time to first token from 6.4 s to 2.2 s, with 47% lower energy per output token, across four repeats. These are results for that workload, not a general speedup claim. See the recorded GPU results.
| Term | Meaning |
|---|---|
| TTFT | Time from submitting a request to observing its first output token, including queue wait |
| TPOT | Average time per output token after the first observation |
| p95 | The latency at the 95th percentile; a way to track slower requests |
| Goodput | Share of selected requests that meet every latency limit in their SLO |
| SLO | The latency limits you set for a request class |
| GPU span | Elapsed time between CUDA events around a step; includes gaps between kernel launches |
| J/token | An estimated share of device energy per output token |
A finding names its evidence, assumptions and missing data. A supported finding is a hypothesis to test, not proof of a cause.
decide requires clean tracer health, compatible manifests and at least two
eligible repeats by default. Three or more repeats give a better view of
variation. It checks the workload, model, seed, intended arrivals and
per-request token counts before ranking. Missing evidence stays exploratory;
it cannot qualify a run for a recommendation. < and <= keep their meaning.
The report separates variation between runs from a bootstrap interval over requests. Requests sharing engine steps are related, so that interval is not a substitute for more runs. Recommendations are advisory; they do not change a running server.
LLM.generate() in vLLM 0.11.0 returns final outputs only, so llmtrace cannot
observe TTFT or TPOT through it. Use the runner or the timing helper in the
quickstart. Engine timings are measured
at step boundaries.
CUDA-event spans include launch gaps. They do not directly measure GPU busy time or separate kernel execution from host stalls.
On a multi-GPU host, select every physical GPU used by the engine with
--gpu-id, repeated for each device. These are NVML indices, not remapped
CUDA indices. The manifest records the selection and physical UUIDs.
Energy comes from sampled device power, then a stated allocation policy
splits it among requests. Other processes on a selected GPU can affect the
reading. decide withholds J/token when coverage or allocations are
insufficient, and explains why. A lone power sample never means zero energy.
See the energy method.
GPU sessions cover opt-125m and Qwen2.5-7B on four GPU models. They include long-prompt interference, queue overload, KV-cache pressure, tracing overhead and a comparison of CUDA spans with Nsight Systems.
Models above 7B, speculative decoding, multiple outputs per request and the OpenAI-compatible server process itself have not been validated. Distributed tracing and attaching to an existing process are not implemented.
CPU tests check llmtrace's logic. They do not establish compatibility with a new vLLM version. See status and limitations for the details.
| Read this | For |
|---|---|
| Quickstart | Install, record a run and compare changes |
| Status | What works, what was tested and what is missing |
| GPU results | Recorded measurements and their limits |
| Raw evidence | Download traces and verify checksums |
| Development guide | Architecture, methods and contributing |
| Design review | Integration risks and next steps |
| Implementation summary | A short technical overview and migration notes |
| Mixed-prompt experiment | A worked example of finding and testing a bottleneck |
MIT