Skip to content
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@ telefuser/
|----------|------|-------|
| `Qwen-Image` | T2I, Edit | [examples/qwen_image/README.md](examples/qwen_image/README.md) |
| `Z-Image` | T2I | [examples/z_image/README.md](examples/z_image/README.md) |
| `PAN 2` | T2V, I2V | NVFP4 + block CPU offload on Blackwell; [PAN 2 guide](docs/en/pan2.md) |
| `Flux2 Klein` | T2I | [examples/flux2_klein/README.md](examples/flux2_klein/README.md) |

See [examples/README.md](examples/README.md) for the example runner and baseline comparison workflow.
Expand Down
1 change: 1 addition & 0 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ telefuser/
|----------|------|------|
| `Qwen-Image` | T2I, Edit | [examples/qwen_image/README.md](examples/qwen_image/README.md) |
| `Z-Image` | T2I | [examples/z_image/README.md](examples/z_image/README.md) |
| `PAN 2` | T2V、I2V | Blackwell 上的 NVFP4 + 分块 CPU offload,见 [PAN 2 指南](docs/zh/pan2.md) |
| `Flux2 Klein` | T2I | [examples/flux2_klein/README.md](examples/flux2_klein/README.md) |

[examples/README.md](examples/README.md) 中提供了统一的 example runner 与 baseline 对比流程说明。
Expand Down
66 changes: 66 additions & 0 deletions docs/en/pan2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# PAN 2 on Blackwell

TeleFuser exposes the public `IFM/PAN2` Diffusers modular pipeline through
`telefuser.pipelines.pan2`. PAN 2 is a 29.2B-parameter world model. BF16 does
not fit on a 24 GB RTX 5090; the supported single-card profile uses TorchAO
NVFP4 for the diffusion transformer and Diffusers block-level CPU offload.

Install `diffusers>=0.43.0`, `torchao>=0.17`, and a PyTorch build with CUDA 13.
The GPU must report compute capability SM100 or newer. The first run downloads
the PAN 2 transformer, Qwen3.5-9B text encoder, and Wan 2.2 VAE, so plan for
more than 100 GB of host storage and RAM.

```bash
python examples/pan2/pan2_t2v.py \
--nvfp4 \
--steps 4 \
--prompt "A small red car drives through a rainy city street." \
--output pan2.mp4
```

The same entrypoint accepts `--image` for image-to-video. NVFP4 quantization
is applied only to Linear layers in the PAN 2 transformer; the text encoder
and VAE stay in BF16/FP32 and are also offloaded. This profile trades latency
for fitting the model on a single 24 GB Blackwell card.

MiniMax-H3-Image is not enabled by this profile. Its 33.1B transformer is
released as approximately 66 GB of BF16 weights and requires a separate
multi-component loader; it does not fit as a resident single-card TeleFuser
pipeline on a 24 GB RTX 5090.

## BF16 vs NVFP4 benchmark

Run the matched benchmark after the PAN 2 weights are cached locally:

```bash
python tools/validation/benchmark_pan2_telefuser.py \
--num-frames 49 --steps 4 \
--output-dir outputs/pan2-benchmark
```

The command writes `official-bf16.mp4`, `telefuser-nvfp4.mp4`, and
`report.json`. `generated_video_fps / 24` is the realtime ratio; values above
`1.0` mean the run produces video faster than playback. Both runs use identical
CPU offload, seed, prompt, resolution, frame count and scheduler steps.

For the full transformer, convert blocks one at a time to avoid a resident
BF16 copy during quantization:

```bash
python tools/quantization/quantize_pan2_layerwise.py \
/path/to/IFM/PAN2 /path/to/pan2-nvfp4-layerwise
```

The converter is resumable and writes one `block-XX.pt` file at a time.

Run the example against the converted blocks with:

```bash
python examples/pan2/pan2_t2v.py \
--model-id /path/to/IFM/PAN2 \
--layerwise-nvfp4-dir /path/to/pan2-nvfp4-layerwise \
--steps 4 --output pan2-nvfp4.mp4
```

The measured 5090 smoke result and chart are in
`benchmarks/pan2/report.json` and `benchmarks/pan2/pan2-nvfp4-results.png`.
1 change: 1 addition & 0 deletions docs/en/supported_models.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ by the selected guide before changing optimization settings.
| [LingBot-World v2](/TeleFuser/cookbook/lingbot-world/) | Bidirectional world-model streaming | Standalone and LiveKit |
| [LingBot-World-Fast](/TeleFuser/cookbook/lingbot-world/) | Causal interactive streaming | Standalone and LiveKit |
| [ABot-World-0-5B-LF](/TeleFuser/cookbook/abot-world/) | Interactive generation | HTTP and LiveKit |
| [PAN 2](pan2.md) | T2V and I2V | NVFP4 with block CPU offload on Blackwell |

## Video, Audio, and Restoration

Expand Down
62 changes: 62 additions & 0 deletions docs/zh/pan2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# 在 Blackwell 上运行 PAN 2

TeleFuser 通过 `telefuser.pipelines.pan2` 接入公开的 `IFM/PAN2`
Diffusers 模块化流水线。PAN 2 有 292 亿参数,BF16 无法放入 24 GB 的
RTX 5090;单卡配置使用 TorchAO NVFP4 量化扩散 Transformer,并启用
Diffusers block-level CPU offload。

安装 `diffusers>=0.43.0`、`torchao>=0.17` 和 CUDA 13 PyTorch。GPU 必须是
SM100 或更新架构。首次运行需要下载 PAN 2 Transformer、Qwen3.5-9B 文本
编码器和 Wan 2.2 VAE,请预留超过 100 GB 的磁盘和主机内存。

```bash
python examples/pan2/pan2_t2v.py \
--nvfp4 \
--steps 4 \
--prompt "A small red car drives through a rainy city street." \
--output pan2.mp4
```

加入 `--image` 可执行 image-to-video。NVFP4 只量化 PAN 2 Transformer 中
的 Linear 层,文本编码器和 VAE 保持 BF16/FP32 并参与 offload;代价是更高
的延迟,以换取单张 24 GB Blackwell 显卡上的可装载性。

MiniMax-H3-Image 暂未启用。它的 331 亿参数 Transformer 约 66 GB BF16,
需要独立的多组件加载器,不能作为常驻单卡 TeleFuser pipeline 放入 24 GB
RTX 5090。

## BF16 与 NVFP4 对比

PAN 2 权重已缓存后,运行匹配基准:

```bash
python tools/validation/benchmark_pan2_telefuser.py \
--num-frames 49 --steps 4 \
--output-dir outputs/pan2-benchmark
```

命令会生成 `official-bf16.mp4`、`telefuser-nvfp4.mp4` 和 `report.json`。
`generated_video_fps / 24` 是实时倍率,大于 `1.0` 表示生成速度超过播放速度。
两次运行使用相同的 CPU offload、seed、prompt、分辨率、帧数和采样步数。

完整 Transformer 的量化应逐个 block 执行,避免量化过程中同时驻留完整
BF16 模型:

```bash
python tools/quantization/quantize_pan2_layerwise.py \
/path/to/IFM/PAN2 /path/to/pan2-nvfp4-layerwise
```

该工具支持中断后继续,并按 block 写出 `block-XX.pt` 文件。

使用已转换的 block 运行示例:

```bash
python examples/pan2/pan2_t2v.py \
--model-id /path/to/IFM/PAN2 \
--layerwise-nvfp4-dir /path/to/pan2-nvfp4-layerwise \
--steps 4 --output pan2-nvfp4.mp4
```

5090 实测报告和图表位于 `benchmarks/pan2/report.json` 与
`benchmarks/pan2/pan2-nvfp4-results.png`。
1 change: 1 addition & 0 deletions docs/zh/supported_models.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ ModelScope 地址、所需文件、支持任务、硬件假设及可运行命令
| [LingBot-World v2](/TeleFuser/zh/cookbook/lingbot-world/) | 双向世界模型流式推理 | 单机和 LiveKit |
| [LingBot-World-Fast](/TeleFuser/zh/cookbook/lingbot-world/) | 因果交互式流推理 | 单机和 LiveKit |
| [ABot-World-0-5B-LF](/TeleFuser/zh/cookbook/abot-world/) | 交互式生成 | HTTP 和 LiveKit |
| [PAN 2](pan2.md) | T2V 和 I2V | Blackwell 上的 NVFP4 与分块 CPU offload |

## 视频、音视频与修复

Expand Down
42 changes: 42 additions & 0 deletions examples/pan2/pan2_t2v.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
"""Run PAN 2 through TeleFuser's NVFP4 loader."""

from __future__ import annotations

import argparse
from pathlib import Path

from diffusers.utils import export_to_video

from telefuser.pipelines.pan2 import PAN2Pipeline, PAN2PipelineConfig


def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--model-id", default="IFM/PAN2")
parser.add_argument("--prompt", default="A small red car drives through a rainy city street.")
parser.add_argument("--image", type=Path)
parser.add_argument("--steps", type=int, default=4)
parser.add_argument("--output", type=Path, default=Path("pan2.mp4"))
parser.add_argument("--nvfp4", action="store_true")
parser.add_argument("--layerwise-nvfp4-dir")
args = parser.parse_args()

pipe = PAN2Pipeline.from_pretrained(
PAN2PipelineConfig(
model_id=args.model_id,
nvfp4=args.nvfp4,
cpu_offload=True,
layerwise_nvfp4_dir=args.layerwise_nvfp4_dir,
)
)
request = {"prompt": args.prompt, "num_inference_steps": args.steps, "output_type": "np"}
if args.image:
request["image"] = str(args.image)
result = pipe(**request)
frames = getattr(result, "frames", getattr(result, "videos", result))
export_to_video(frames[0] if isinstance(frames, list) else frames, str(args.output), fps=24)
print(args.output)


if __name__ == "__main__":
main()
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ nav:
- Basic Inference: quickstart.md
- Models & Cookbook:
- Supported Models: supported_models.md
- PAN 2 NVFP4 on Blackwell: pan2.md
- Serving & APIs:
- Overview: serving.md
- Batch Service and API Reference: service.md
Expand Down
6 changes: 6 additions & 0 deletions telefuser/pipelines/pan2/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""PAN 2 video generation pipeline integration."""

from .pipeline import PAN2Pipeline, PAN2PipelineConfig
from .layerwise import load_layerwise_transformer

__all__ = ["PAN2Pipeline", "PAN2PipelineConfig", "load_layerwise_transformer"]
134 changes: 134 additions & 0 deletions telefuser/pipelines/pan2/layerwise.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
"""Layerwise loader for the CPU-produced PAN2 NVFP4 block checkpoint."""

from __future__ import annotations

import json
import copy
from pathlib import Path

import torch
from accelerate import init_empty_weights
from accelerate.utils import set_module_tensor_to_device
from diffusers.models.transformers.transformer_pan2 import PAN2Transformer3DModel, PAN2TransformerBlock
from torchao.prototype.mx_formats import NVFP4DynamicActivationNVFP4WeightConfig
from torchao.prototype.mx_formats import nvfp4_tensor as _nvfp4
from torchao.quantization import quantize_


_ORIGINAL_NVFP4_QUANTIZE = _nvfp4.nvfp4_quantize


def _chunked_nvfp4_quantize(data_hp: torch.Tensor, block_size: int = 16, per_tensor_scale=None):
"""Bound activation quantization peak memory without changing NVFP4 layout."""
if data_hp.ndim < 2 or data_hp.shape[-2] <= 256:
return _ORIGINAL_NVFP4_QUANTIZE(data_hp, block_size, per_tensor_scale)
leading = data_hp.shape[:-2]
rows, width = data_hp.shape[-2:]
flat = data_hp.reshape(-1, width)
q_parts, scale_parts = [], []
for start in range(0, flat.shape[0], 256):
scales, q = _ORIGINAL_NVFP4_QUANTIZE(flat[start : start + 256], block_size, per_tensor_scale)
q_parts.append(q)
scale_parts.append(scales)
return (
torch.cat(scale_parts, dim=0).reshape(*leading, rows, width // block_size),
torch.cat(q_parts, dim=0).reshape(*leading, rows, width // 2),
)


_nvfp4.nvfp4_quantize = _chunked_nvfp4_quantize


def _load_non_block_weights(model: torch.nn.Module, model_dir: Path, device: torch.device) -> None:
from safetensors import safe_open

index = json.loads((model_dir / "diffusion_pytorch_model.safetensors.index.json").read_text())["weight_map"]
for shard in sorted(set(index.values())):
keys = [key for key, value in index.items() if value == shard and not key.startswith("transformer_blocks.")]
if not keys:
continue
with safe_open(str(model_dir / shard), framework="pt", device="cpu") as handle:
for key in keys:
value = handle.get_tensor(key)
if value.is_floating_point():
value = value.to(torch.bfloat16)
set_module_tensor_to_device(model, key, device, value=value)


def load_layerwise_transformer(model_dir: str | Path, block_dir: str | Path, device: str = "cuda") -> PAN2Transformer3DModel:
model_dir = Path(model_dir)
block_dir = Path(block_dir)
config = PAN2Transformer3DModel.load_config(str(model_dir))
target = torch.device(device)
with init_empty_weights():
model = PAN2Transformer3DModel.from_config(config)
model = model.to_empty(device="meta").to(dtype=torch.bfloat16)
_load_non_block_weights(model, model_dir, target)
template = PAN2TransformerBlock(4096, 32, 128, 8.0, 4.0)
quantize_(template, NVFP4DynamicActivationNVFP4WeightConfig(use_triton_kernel=True))
for index, empty_block in enumerate(model.transformer_blocks):
print(f"restoring block {index}", flush=True)
state_path = block_dir / f"block-{index:02d}.pt"
if not state_path.exists():
raise FileNotFoundError(state_path)
block = copy.deepcopy(template)
block.load_state_dict(
torch.load(state_path, map_location="cpu", weights_only=False), assign=True
)
for parameter in block.parameters():
tensor = parameter.data
if hasattr(tensor, "act_quant_kwargs") and tensor.act_quant_kwargs is not None:
tensor.act_quant_kwargs.use_triton_kernel = True
model.transformer_blocks[index] = block
del empty_block
del template
if target.type == "cuda":
# TorchAO NVFP4 tensors do not implement every tensor hook operation
# used by Diffusers group offload. Explicit hooks keep only one block
# on the GPU and use the supported module.to() path.
for block in model.transformer_blocks:
def onload(module: torch.nn.Module, _args: tuple[object, ...], d: torch.device = target) -> None:
with torch.inference_mode(False):
module.to(d)

def offload(module: torch.nn.Module, _args: tuple[object, ...], output: object) -> object:
with torch.inference_mode(False):
module.to("cpu")
return output

block.register_forward_pre_hook(onload)
block.register_forward_hook(offload)

# The PAN2 checkpoint keeps the input projection in fp32 while the
# pipeline normally creates BF16 latents. Match the projection dtype so
# PyTorch selects the CUDA convolution kernel instead of slow_conv3d.
def cast_inputs(module: torch.nn.Module, args: tuple[object, ...], kwargs: dict[str, object]):
if len(args) >= 3:
hidden, timestep, context = args[:3]
hidden = hidden.to(target, dtype=module.x_embedder.weight.dtype)
context = context.to(target, dtype=module.context_embedder[1].weight.dtype)
timestep = timestep.to(target)
return (hidden, timestep, context, *args[3:]), kwargs
if "hidden_states" in kwargs:
kwargs = dict(kwargs)
kwargs["hidden_states"] = kwargs["hidden_states"].to(target, dtype=module.x_embedder.weight.dtype)
if "encoder_hidden_states" in kwargs:
kwargs["encoder_hidden_states"] = kwargs["encoder_hidden_states"].to(
target, dtype=module.context_embedder[1].weight.dtype
)
if "timestep" in kwargs:
kwargs["timestep"] = kwargs["timestep"].to(target)
return args, kwargs

model.register_forward_pre_hook(cast_inputs, with_kwargs=True)

def restore_output_device(_module: torch.nn.Module, _args: tuple[object, ...], output: object) -> object:
if hasattr(output, "sample"):
output.sample = output.sample.to("cpu")
return output
if isinstance(output, torch.Tensor):
return output.to("cpu")
return output

model.register_forward_hook(restore_output_device)
return model
Loading
Loading