TileRT: Tile-Based Runtime for
Ultra-Low-Latency LLM Inference

Overview · Generation · MTP Generation · Installation · News

📰 News

🏆 2026-09-24 · TileRT Takes the Top Spot on AgentX. On SemiAnalysis's InferenceX AgentX leaderboard, TileRT powers GLM-5.3 to 469 tok/s of single-user generation on million-token agent sessions with 8× AMD Instinct MI355X — #1 in single-user performance. Running the original FP8 weights, it leads the second-place NVIDIA GB300 NVL72 FP4 submission by more than 100 tok/s. v0.1.6 brings ROCm support to the public release: GLM-5.2/5.3 on AMD Instinct MI350X/MI355X.
🔀 2026-07-14 · v0.1.5 Released. Introduce PD (prefill–decode) disaggregation — vLLM prefill + TileRT decode, behind an OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2.
💥 2026-06-08 · Breaking 1000 TPS on a 1T Model. In collaboration with Xiaomi MiMo, TileRT pushes MiMo-V2.5-Pro-UltraSpeed past 1000 tokens/s on a 1-trillion-parameter model through extreme model–system co-design — a first without custom silicon, all on a single 8-GPU node.
🚀 2026-06-01 · v0.1.4 Released. A major performance upgrade for both DeepSeek-V3.2 and GLM-5, with model quality unchanged. See the benchmark charts for details.
🏭 2026-05-22 · TileRT in Production. GLM-5.1-highspeed is now live on Z.ai, powered by TileRT — from experimental prototype to real production.

Key Milestones

🔥 2026-02-14 · Try the Online Demo. Our online demo is now live! Experience ultra-low-latency inference with GLM-5 and DeepSeek-V3.2. Try it now !
🎉 2026-02-14 · v0.1.3 Released. The v0.1.3 release introduces full support for the latest GLM-5 model, achieving up to 500 tokens/s on GLM-5-FP8 and up to 600 tokens/s on DeepSeek-V3.2.
🚀 2026-01-26 · v0.1.2-alpha.1. Multi-Token Prediction (MTP) is now available in TileRT! With mtp=3, we achieve decoding rates of up to 590 tokens/s under synthetic workloads.
⚡ 2025-12-23 · v0.1.1. Achieved ~35% further reduction (3 ~ 4x speedup over baseline) in end-to-end token generation latency on a single node with 8× NVIDIA B200.
🚀 2025-11-20 · v0.1.0-alpha.1. Initial public release for DeepSeek-V3.2-Exp, targeting ultra-low-latency inference. Available on PyPI and HuggingFace.

TileRT is a project designed to serve large language models (LLMs) in ultra-low-latency scenarios. Its goal is to push the latency limits of LLMs without compromising model size or quality—enabling models with hundreds of billions of parameters to achieve millisecond-level time per output token (TPOT).

Unlike traditional inference systems optimized for high-throughput batch processing, TileRT prioritizes responsiveness, which is critical for applications such as high-frequency trading, interactive AI, real-time decision-making, long-running agents, and AI-assisted coding, where the latency of individual requests matters most.

To achieve this, TileRT introduces a tile-level runtime engine. Leveraging a compiler-driven approach, LLM operators are decomposed into fine-grained tile-level tasks, while the runtime dynamically reschedules computation, I/O, and communication across multiple devices in a highly overlapped manner. This design minimizes idle time and improves hardware utilization.

The project is actively evolving, and the underlying compiler techniques will be gradually shared with the community as they are integrated into TileLang and TileScale.

_{GLM-5.2/5.3-FP8 token generation speed on 8× AMD Instinct MI350X with TileRT v0.1.6 (greedy decoding, bf16 KV and indexer caches). Output length 1K, input length 1K–1M. Bars compare TileRT without MTP, with MTP at depth 3 and average acceptance length 3.2, and the peak under best-case MTP acceptance (4.0).}

Installation

Important

TileRT v0.1.6 is distributed as a pre-built binary wheel that carries three backends: two CUDA backends (DeepSeek-V3.2, GLM-5/5.1) and one ROCm backend (GLM-5.2/5.3). Each backend is linked against the exact PyTorch ABI listed below, and tilert.load_backend() refuses a PyTorch of the wrong version or flavor (CUDA vs ROCm). Other combinations of Python, CUDA/ROCm, or PyTorch versions are untested and not guaranteed to work — please reproduce this environment for a supported setup.

Build environment of the v0.1.6 wheel

The official tilert==0.1.6.post2 wheel on PyPI was compiled against the following stacks. Treat these as hard requirements, not lower bounds (transformers / tokenizers are lower bounds).

Component	NVIDIA — DeepSeek-V3.2, GLM-5/5.1	AMD — GLM-5.2/5.3
GPU	8× NVIDIA B200	8× AMD Instinct MI350X / MI355X
Driver / runtime	NVIDIA driver supporting the CUDA 13.2 runtime	ROCm 7.14.0
Operating System	Linux x86_64, glibc ≥ 2.28 (manylinux_2_28)	Linux x86_64, glibc ≥ 2.28 (manylinux_2_28)
Python	3.12	3.12
PyTorch	`torch==2.11.0+cu130`	`torch==2.12.0+rocm7.14.0`
`transformers`	`>= 4.46.3`	`>= 4.46.3`
`tokenizers`	`>= 0.20.3`	`>= 0.20.3`

Recommended: pre-built Docker images

The pinned build environments above are preinstalled in our official images — this is the recommended way to run v0.1.6 and avoids any version drift on the host.

NVIDIA (B200). The image is mirrored to two registries; pull from whichever is reachable, then launch a container with all 8 B200 GPUs attached:

# GitHub Container Registry
docker pull ghcr.io/tile-ai/tilert:cu132-latest

# Docker Hub
docker pull tileai/tilert:cu132-latest

docker run --rm -it --gpus all --ipc=host \
    -v "$PWD":/workspace -w /workspace \
    ghcr.io/tile-ai/tilert:cu132-latest

AMD (MI350X / MI355X). Pull the ROCm image and launch a container with the GPUs attached:

docker pull ghcr.io/tile-ai/tilert-rocm-decode:0.1.6

docker run --rm -it --ipc=host \
    --device=/dev/kfd --device=/dev/dri --group-add video \
    -v "$PWD":/workspace -w /workspace \
    ghcr.io/tile-ai/tilert-rocm-decode:0.1.6

The ROCm image sets HF_HUB_OFFLINE=1; run unset HF_HUB_OFFLINE before downloading a checkpoint from inside the container.

Inside either container, install the wheel. It accepts both PyTorch builds (torch>=2.11,<2.13), so pip keeps the one preinstalled in the image:

# From PyPI:
pip install tilert==0.1.6.post2

# Or pin the exact wheel from the GitHub Release page directly
# (same artifact, useful when PyPI is unreachable):
pip install https://github.com/tile-ai/TileRT/releases/download/v0.1.6.post2/tilert-0.1.6.post2-cp312-cp312-manylinux_2_28_x86_64.whl

Verify the install:

python -c "import tilert, torch; print('tilert', tilert.__version__, '/ torch', torch.__version__)"
# NVIDIA — expected: tilert 0.1.6.post2 / torch 2.11.0+cu130
# AMD    — expected: tilert 0.1.6.post2 / torch 2.12.0+rocm7.14.0

Proceed to Getting Started to download and convert model weights.

Getting Started

Step 1: Download Official Model Weights

Starting from release v0.1.3, TileRT no longer requires downloading pre-converted weights from Hugging Face. Instead, you can download the official model weights directly from the model's source (e.g., Hugging Face), and then convert them using the weight converter script included with the latest TileRT release.

Step 2: Shard Weights with `weight_converter`

The converters ship inside the tilert wheel. For the NVIDIA models, weight_converter rewrites the official HF checkpoint into TileRT's per-device layout — 8 shards, one per B200, with keys suffixed *_dev_{0..7} and a fresh model.safetensors.index.json. The runtime loads these shards directly; the original checkpoint is no longer needed after conversion.

For DeepSeek-V3.2:

python -m tilert.models.preprocess.weight_converter \
  --model_type deepseek-v32 \
  --model_dir "/path/to/DeepSeek-V3.2" \
  --save_dir "/path/to/DeepSeek-V3.2-TileRT"

For GLM-5/5.1:

python -m tilert.models.preprocess.weight_converter \
  --model_type glm-5 \
  --model_dir "/path/to/GLM-5-FP8" \
  --save_dir "/path/to/GLM-5-FP8-TileRT"

For GLM-5.2/5.3 (AMD MI350X / MI355X), the ROCm port has its own converter:

python -m tilert.models.glm_5_2_rocm.weight_converter \
  --model_dir "/path/to/GLM-5.2-FP8" \
  --save_dir "/path/to/GLM-5.2-FP8-TileRT" \
  --num_mtp 1

--num_mtp 1 also converts the MTP module, which MTP decoding needs; leave it out to convert the base model only.

--model_dir is the directory of the downloaded HF checkpoint; --save_dir is where the sharded TileRT-format weights will land.

Running the Generation Example

The simplest entry point is the bundled CLI. Pick --model deepseek_v3_2, --model glm5, or --model glm5_2_rocm, and point --model-weights-dir (required) at the converted weights:

python -m tilert.generate --model glm5_2_rocm \
    --model-weights-dir /path/to/GLM-5.2-FP8-TileRT \
    --prompt "Tell me three jokes." --max-new-tokens 1000

On AMD, --max-seq-len (default 8192) sets the KV-cache length; the NVIDIA models take it from their model args.

Warning

In the published tilert==0.1.6.post2 wheel, the CLI fails for the NVIDIA models (--model deepseek_v3_2 / --model glm5) once the weights are loaded, with TypeError: generate() got an unexpected keyword argument 'max_new_tokens'. The fix is in this repository's tilert/generate.py but not yet in a published wheel; until it is, drive the NVIDIA models through the Python API below.

Note

v0.1.6 ships three independent backend libraries — libtilert_dsv32.so and libtilert_glm5.so (CUDA), and libtilert_glm52_rocm.so (ROCm) — and loads exactly one per Python process via tilert.load_backend(model_type), which also checks the running PyTorch's version and flavor (CUDA vs ROCm). Run different models in separate processes — they cannot coexist in a single interpreter.

To drive generation programmatically, load the backend first, then build the matching generator:

import tilert
from tilert.models.deepseek_v3_2.generator import DSAv32Generator
from tilert.models.deepseek_v3_2.model_args import ModelArgs

tilert.load_backend("deepseek_v3_2")

generator = DSAv32Generator(
    model_args=ModelArgs(),
    max_new_tokens=1000,
    model_weights_dir="/path/to/DeepSeek-V3.2-TileRT",
    with_mtp=False,
)
generator.from_pretrained()

prompt = (
    "Tell me three jokes:\n\n"
    "1. A dad joke,\n"
    "2. A programmer joke,\n"
    "3. A joke that only makes sense if you've ever tried "
    "to train a large language model.\n"
    "Keep each joke under 15 words."
)

print("Prompt:", prompt)
print("Completion:")
completion = generator.generate(prompt)

(For GLM-5, swap in tilert.load_backend("glm5") and from tilert.models.glm_5.generator import GLM5Generator with ModelArgsGLM5. For GLM-5.2/5.3 on AMD, use tilert.load_backend("glm5_2_rocm") and from tilert.models.glm_5_2_rocm.generator import Glm52Generator with ModelArgsGlm52; it takes num_mtp= (0, 1 or 3) instead of with_mtp=.)

For example, TileRT may generate:

Sample output (click to expand)

1. I'm afraid for the calendar. Its days are numbered.
2. There are only 10 kinds of people: those who understand binary and those who don't.
3. My model's loss is low, but its answers are still nonsense. Overfitting.

This example demonstrates basic single-step autoregressive generation using the precompiled model.

Running the Generation Example with Multi-Token Prediction (MTP)

TileRT also supports Multi-Token Prediction (MTP), which allows the model to generate multiple tokens per forward pass and reduces sequential decoding depth. Enable it from the CLI with --num-mtp, the draft depth (0, the default, disables it):

python -m tilert.generate --model glm5_2_rocm \
    --model-weights-dir /path/to/GLM-5.2-FP8-TileRT \
    --num-mtp 3 --max-new-tokens 1000

GLM-5.2/5.3 run MTP at depth 1 or 3 and need weights converted with --num_mtp 1; the NVIDIA models run any --num-mtp above 0 at their fixed depth of 3.

Or programmatically, pass with_mtp=True to the generator:

import tilert
from tilert.models.deepseek_v3_2.generator import DSAv32Generator
from tilert.models.deepseek_v3_2.model_args import ModelArgs

tilert.load_backend("deepseek_v3_2")

generator = DSAv32Generator(
    model_args=ModelArgs(),
    max_new_tokens=1000,
    model_weights_dir="/path/to/DeepSeek-V3.2-TileRT",
    with_mtp=True,
)
generator.from_pretrained()
prompt = "Tell me 10 jokes, keep them all under 100 words."

print("Prompt:", prompt)
print("Completion:")
completion = generator.generate(prompt)

When MTP is enabled, TileRT may report statistics similar to the following during generation:

Accepted length: mean=2.77, min=1, max=4

This indicates that, on average, multiple tokens are accepted per decoding step under MTP.

Sample output (click to expand)

Of course! Here are 10 short jokes for you.

1. I told my wife she was drawing her eyebrows too high. She looked surprised.

2. I invented a new word: Plagiarism.

3. Why don't scientists trust atoms? Because they make up everything.

4. I'm reading a book on anti-gravity. It's impossible to put down.

5. What's the best thing about Switzerland? I don't know, but the flag is a big plus.

6. I told my computer I needed a break, and now it won't stop sending me vacation ads.

7. Why did the scarecrow win an award? He was outstanding in his field.

8. What do you call a fake noodle? An impasta.

9. I told my suitcase there's no vacation, and now it has a lot of baggage.

10. Why don't skeletons fight each other? They don't have the guts.

This example highlights how MTP enables TileRT to efficiently generate longer outputs by accepting multiple tokens per decoding step, while preserving the same Python API interface.

For the full list of CLI flags (prompt files, sampling, MTP depth, …), run python -m tilert.generate --help.

Disaggregated Serving: vLLM Prefill + TileRT Decode

TileRT can run as the decode engine behind a vLLM prefill, integrated through vLLM's V1 KVConnector interface. The connector, decode server, and router all ship inside the tilert wheel under tilert.pd_vllm — no vLLM fork or patch is needed (the connector loads via vLLM's standard kv_connector_module_path). Latency-critical requests are routed to the TileRT decode pool; other traffic can stay on native vLLM decode.

Prerequisites

Convert the model weights for TileRT decode (see Step 2).
On the prefill node, a vLLM build with V1 disaggregation and support for the GLM-5/5.1 / DeepSeek-V3.2 (DSA) model and the fp8_ds_mla KV-cache dtype. Install tilert in the same environment so the connector plugin is importable.
On the decode and router hosts, install tilert with the pd extra: pip install "tilert[pd]==0.1.6.post2" pulls in FastAPI, Uvicorn, HTTPX, Requests and Pydantic.
The KV-cache dtype must match on both ends. These examples use fp8: --kv-cache-dtype fp8_ds_mla on the vLLM prefill and --kv-cache-dtype fp8 on the TileRT decode (a mismatch is rejected at the connector handshake).
The examples use the NIXL transfer engine. On multi-NIC hosts, pin NIXL to the RDMA NICs via UCX_NET_DEVICES (otherwise UCX may pick the wrong interface). Mooncake is also supported (--transport mooncake on the decode, "tilert_transport": "mooncake" on the prefill).

Commands below use GLM-5/5.1. For DeepSeek-V3.2, use --model deepseek_v3_2, the DeepSeek-V3.2-TileRT weights, and --parser none. For GLM-5.2/5.3 on AMD MI350X / MI355X, the decode server's --model glm5_2 profile (glm5_3 is accepted as an alias) runs the ROCm engine; start that decode node from the ROCm image.

Topology A: vLLM prefill → TileRT decode

Three processes — a TileRT decode server, a stock vLLM prefill, and an OpenAI-compatible router:

# 1) TileRT decode node
python -m tilert.pd_vllm.decode_server \
    --engine tilert --model glm5 \
    --model-weights-dir /path/to/GLM-5.1-FP8-TileRT \
    --with-mtp --max-seq-len 202752 \
    --kv-cache-dtype fp8 --transport nixl \
    --ctrl-port 5556 --http-port 5557

# 2) vLLM prefill (stock vLLM; the TileRT connector loads as a plugin).
#    The MTP speculative config is required: the prefill populates the
#    draft-layer KV that decode-side speculation resumes from.
export UCX_NET_DEVICES=mlx5_1:1,mlx5_2:1,...   # pin NIXL to the RDMA NICs (multi-NIC hosts)
vllm serve /path/to/GLM-5.1-FP8 \
    --served-model-name glm5 --port 8000 \
    --tensor-parallel-size 8 --enforce-eager --trust-remote-code \
    --return-tokens-as-token-ids --gpu-memory-utilization 0.75 \
    --kv-cache-dtype fp8_ds_mla \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
    --kv-transfer-config '{
        "kv_connector": "TileRTConnector",
        "kv_connector_module_path": "tilert.pd_vllm.prefill_connector",
        "kv_role": "kv_producer",
        "kv_connector_extra_config": {
            "tilert_host": "<TILERT_DECODE_IP>", "tilert_ctrl_port": 5556,
            "tilert_model": "glm5", "tilert_max_seq_len": 202752,
            "tilert_transport": "nixl"}}'

# 3) Router — OpenAI-compatible ingress for the TileRT pool
python -m tilert.pd_vllm.pd_router \
    --vllm-url http://<PREFILL_IP>:8000 \
    --decode <TILERT_DECODE_IP>:5556:5557 \
    --model-path /path/to/GLM-5.1-FP8 \
    --parser glm47 --port 23333

Send OpenAI requests to http://<router>:23333/v1/chat/completions. The router runs the prefill on vLLM (first token), hands the attention state to the TileRT decode node over RDMA, and streams the completion back.

A decode engine serves one sequence at a time, so the router reserves a node per request and answers 429 while they are all busy. Add --queue-timeout <seconds> to make a request wait for a free node instead of failing: useful when a single client fans out into concurrent sub-conversations — an agentic session spawning sub-agents, say — and the burst is wider than the pool but short-lived. Waits longer than 0.1 s are logged. The default, 0, keeps the fail-fast behaviour.

The router resolves sampling defaults the way vLLM does: --generation-config auto (the default) reads generation_config.json under --model-path, --generation-config vllm uses neutral defaults, and --default-temperature / --default-top-p / --default-top-k override single values. The resolved values are sent explicitly to both the prefill and the decode leg, so the two cannot disagree. Decode nodes advertise what they support (logprobs, penalties, ignore_eos), and a request that asks for something its node cannot honour is rejected with an error.

On the decode node, --pd-buffer-device cpu keeps the PD receive buffer in pinned host memory registered with the transport: it frees VRAM at the cost of one extra host-to-device copy per request. The default is cuda; TILERT_PD_BUFFER_DEVICE sets it too.

Topology B: shared prefill → TileRT decode and native vLLM decode

One prefill pool feeds two decode pools side by side, composed under vLLM's MultiConnector. Each request is claimed by exactly one connector — the TileRT connector claims requests marked with tilert_host, and vLLM's native connector handles the rest — so latency-critical traffic goes to TileRT while general traffic stays on native vLLM decode, behind the same OpenAI surface.

# 1) TileRT decode node (identical to Topology A)
python -m tilert.pd_vllm.decode_server --engine tilert --model glm5 \
    --model-weights-dir /path/to/GLM-5.1-FP8-TileRT --with-mtp \
    --max-seq-len 202752 --kv-cache-dtype fp8 --transport nixl \
    --ctrl-port 5556 --http-port 5557

# 2) Native vLLM decode node — vLLM's standard disaggregation (NixlConnector consumer)
export UCX_NET_DEVICES=mlx5_1:1,mlx5_2:1,...
vllm serve /path/to/GLM-5.1-FP8 --served-model-name glm5 --port 8001 \
    --tensor-parallel-size 8 --enforce-eager --trust-remote-code \
    --return-tokens-as-token-ids --kv-cache-dtype fp8_ds_mla \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
    --kv-transfer-config '{"kv_connector": "NixlConnector", "kv_role": "kv_consumer"}'

# 3) Shared vLLM prefill — MultiConnector[ NixlConnector + TileRTConnector ]
export UCX_NET_DEVICES=mlx5_1:1,mlx5_2:1,...
vllm serve /path/to/GLM-5.1-FP8 --served-model-name glm5 --port 8000 \
    --tensor-parallel-size 8 --enforce-eager --trust-remote-code \
    --return-tokens-as-token-ids --gpu-memory-utilization 0.75 \
    --kv-cache-dtype fp8_ds_mla \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
    --kv-transfer-config '{
        "kv_connector": "MultiConnector", "kv_role": "kv_producer",
        "kv_connector_extra_config": {"connectors": [
            {"kv_connector": "NixlConnector", "kv_role": "kv_producer"},
            {"kv_connector": "TileRTConnector",
             "kv_connector_module_path": "tilert.pd_vllm.prefill_connector",
             "kv_role": "kv_producer",
             "kv_connector_extra_config": {
                 "tilert_host": "<TILERT_DECODE_IP>", "tilert_ctrl_port": 5556,
                 "tilert_model": "glm5", "tilert_max_seq_len": 202752,
                 "tilert_transport": "nixl"}}]}}'

# 4a) TileRT router — latency-critical traffic → TileRT pool
python -m tilert.pd_vllm.pd_router --vllm-url http://<PREFILL_IP>:8000 \
    --decode <TILERT_DECODE_IP>:5556:5557 --model-path /path/to/GLM-5.1-FP8 \
    --parser glm47 --port 23333

# 4b) General traffic → native vLLM decode pool, via vLLM's standard NixlConnector
#     disaggregation proxy, pointing prefill :8000 → native decode :8001.

Note. Running NIXL end to end (both the native and TileRT connectors in NIXL mode) lets the shared prefill use a single transfer library. Only the prefill's --kv-transfer-config differs from Topology A; the TileRT decode node is unchanged, and the native decode instance plus its proxy follow vLLM's usual NixlConnector disaggregation setup.

Status & Future Work

TileRT is currently offered as a preview release, and we’re just getting started. We are continuously improving the installation experience and enhancing end-to-end performance. Future releases will keep pushing the boundaries of low-latency generation.

Thank you for your interest and support — stay tuned, even faster token generation is on the way!

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
assets		assets
scripts		scripts
tilert		tilert
.flake8		.flake8
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt

Navigation Menu

Repository files navigation

TileRT: Tile-Based Runtime for
Ultra-Low-Latency LLM Inference

📰 News

Installation

Build environment of the v0.1.6 wheel

Recommended: pre-built Docker images

Getting Started

Step 1: Download Official Model Weights

Step 2: Shard Weights with `weight_converter`

Running the Generation Example

Running the Generation Example with Multi-Token Prediction (MTP)

Disaggregated Serving: vLLM Prefill + TileRT Decode

Topology A: vLLM prefill → TileRT decode

Topology B: shared prefill → TileRT decode and native vLLM decode

Status & Future Work

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

TileRT: Tile-Based Runtime forUltra-Low-Latency LLM Inference

📰 News

Installation

Build environment of the v0.1.6 wheel

Recommended: pre-built Docker images

Getting Started

Step 1: Download Official Model Weights

Step 2: Shard Weights with weight_converter

Running the Generation Example

Running the Generation Example with Multi-Token Prediction (MTP)

Disaggregated Serving: vLLM Prefill + TileRT Decode

Topology A: vLLM prefill → TileRT decode

Topology B: shared prefill → TileRT decode and native vLLM decode

Status & Future Work

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

TileRT: Tile-Based Runtime for
Ultra-Low-Latency LLM Inference

Step 2: Shard Weights with `weight_converter`