Asynchronous GPU-to-CPU tensor reads for CPU MoE offloading

[KaiFelixBennett]

Asynchronous GPU-to-CPU tensor reads for CPU MoE offloading

Motivation

When experts are kept in system RAM using --n-cpu-moe or -ot exps=CPU, the scheduler needs to copy tensors from GPU memory to CPU memory before executing CPU splits.

Currently, this path performs synchronous tensor copies. For backends such as Vulkan, each individual readback can involve a separate GPU submission and fence wait.

I investigated this overhead using a local profiling patch and found that these synchronization costs can become a significant part of token generation time.

Current implementation

In ggml/src/ggml-backend.cpp, ggml_backend_sched_copy_input() handles the transfer of inputs to CPU splits.

When the destination is the CPU backend, there is no cpy_tensor_async implementation available. The current path therefore synchronizes the relevant backends and eventually calls:

ggml_backend_tensor_copy(input, input_cpy)

This results in a synchronous readback for each tensor.

Interestingly, Vulkan already implements get_tensor_async in ggml/src/ggml-vulkan/ggml-vulkan.cpp. It records the readback into the current compute context, with the actual memory copy completed during synchronization.

Measurements

Tested with Vulkan on both AMD and NVIDIA hardware:

	AMD Radeon AI PRO R9700	NVIDIA RTX 5080
CPU MoE experts	20 / 40	20 / 40
Readbacks per token	40 / 80	40 / 80
Average time per readback (before)	~74 µs	~61 µs
Average time per readback (prototype)	~35 µs	~28–29 µs

On the R9700, the readback overhead accounted for approximately 15.5% and 18.6% of token generation time with 20 and 40 CPU MoE layers, respectively.

Prototype results

I implemented a prototype that queues asynchronous tensor reads for CPU splits in ggml_backend_sched_compute_splits() and synchronizes once per source backend.

GPU	CPU MoE	Before	Prototype	Speedup
R9700	20	46.7 t/s	52.8 t/s	1.13x
R9700	40	29.7 t/s	35.0 t/s	1.18x
RTX 5080	20	51.8 t/s	57.1 t/s	1.10x
RTX 5080	40	32.5 t/s	38.2 t/s	1.17x

The improvement is visible on both GPUs, despite their very different PCIe configurations.

Correctness

I compared llama-perplexity results with and without the prototype using -ncmoe 40 -c 512.

The reported perplexity values were identical at all reported positions, including every chunk:

• ub=512: 6 chunks, PPL 1.6315 ± 0.06915
• ub=1: 2 chunks, PPL 1.9163 ± 0.16872

This does not constitute a bitwise comparison of logits, but no difference in the reported perplexity results was observed.

Possible implementation approaches

I see two possible directions:

A. Queue asynchronous reads in the scheduler

Use ggml_backend_tensor_get_async() for inputs that need to be transferred from GPU memory to CPU splits, then synchronize once per source backend before executing the CPU split.

This is what my prototype currently does.

B. Extend the backend copy interface

Provide asynchronous copy support for the CPU backend as a destination, potentially through cpy_tensor_async or a public API that could also address the existing TODO in ggml_backend_sched_copy_input().

Questions for the maintainers

• Would asynchronous GPU-to-CPU readbacks be a reasonable optimization in the scheduler?
• Would approach A be acceptable, or should this functionality be exposed through a more general backend API?
• Are there existing design considerations around asynchronous reads into CPU buffers that I should account for?

I would appreciate feedback on the preferred direction before preparing a PR.

Test environment

• Windows 11
• Intel Core Ultra 7 265KF
• 64 GB RAM
• AMD Radeon AI PRO R9700 (gfx1201, PCIe 4.0 x4, Adrenalin 26.9.2)
• NVIDIA RTX 5080 (PCIe 4.0 x16)
• Vulkan Release build
• Qwen3.6-35B-A3B UD-Q4_K_XL
• 41 layers, 20.1 GB of 22.8 GB experts

The measurements were performed using llama-bench with -ngl 99 -ncmoe N -fa 1 -p 0 -n 64 -r 3, with only one GPU visible per run.

Limitations

So far, I have only tested Vulkan as the source backend, on AMD and NVIDIA GPUs.

CUDA, HIP, Metal, SYCL, Linux/RADV and mixed CUDA/Vulkan configurations have not been tested.

There is also a potential edge case with mixed backends: the CPU compute buffer is allocated using the host buffer of the first device. Depending on the device order, this may prevent Vulkan from using its asynchronous readback path and cause it to fall back to staging and immediate synchronization. This is currently a code-based observation, not a measured result.

AI disclosure

Claude (Claude Code) assisted with the research, profiling scripts, scheduler instrumentation and prototype implementation. I ran the benchmarks on my hardware, reviewed the results and made the technical decisions.