Repository navigation
Description
Status (2026-11-20, after the investigation). Of the +3..5% cycles on identical source, most turned out to be redundant work in the kernel that DSL <= 4.5.3 optimised away and DSL >= 4.6.2 no longer does: (1) the causal mask applied on every m_block (only the diagonal tiles need it), (2) the cluster leader rank / pair masks re-derived at every pipeline arrival. Both are now removed at the source: Dao-AILab/flash-attention#3200 (mask gate + constant destinations) and #3909 here (constant rank / masks in the SM100 pipelines, for every DSL kernel with a 2CTA-pair cluster). With #3200 the causal backward on 4.8.0 is back at the 4.5.3 level (32K causal 1293 vs 1298 TFLOPS).
What remains is pure lowering, on identical source with #3200 applied, 4.5.3 vs 4.8.0: +1.5% kernel cycles (compute-warp loop body 376 -> 427 SASS instructions per iteration, +38 MOV / +17 IMAD re-materialising one uniform base before each use; dQ-drain stage loop +2 BAR.SYNC +1 FENCE.VIEW.ASYNC +1 MEMBAR per iteration), plus a second, simpler instance in an elementwise kernel (+45%, lost memory-level parallelism, 112 -> 86 registers). The refined ask is in the first comment below; the "Ask" at the end of this report is the original one and is superseded by it. The measurements above are kept as reported.
Summary. The FlashAttention-4 SM100 backward kernel (flash_attn/cute/flash_bwd_sm100.py, Dao-AILab/flash-attention) compiled from one unchanged source gets slower with newer CuTe DSL releases, causal path only. ncu --clock-control none, sm__cycles_elapsed.avg (repeat noise 0.02%), B200, hdim 128, 16 heads, seqlen 8k, batch 4, 2CTA kernel (cluster 2, 4096 CTAs):
| DSL | 4.5.3 | 4.6.2 | 4.6.3 | 4.7.0 | 4.7.1 | 4.8.0 |
|---|---|---|---|---|---|---|
| 8k causal, kernel cycles (M) | 3.366 | 3.465 (+2.9%) | 3.543 (+5.2%) | 3.464 | 3.546 | 3.532 |
| 32k causal, batch 1 (M) | 11.89 | 12.23 | 12.52 | 12.23 | 12.53 | 12.49 |
| 8k non-causal (M) | 6.157 | 6.167 | 6.165 | – | 6.145 | 6.167 |
embedded CUDA compiler (strings in _cutlass_ir*.so) |
12.9 / NVVM 20 | 13.3 build 37800683 | same | same | same | 13.4 build 38657139 |
Source for every column: flash-attention 82d6441e plus a 7-line shim that accepts both nvvm.fmax signatures (the only API difference between 4.5 and 4.6 this file hits). cu12 and cu13 wheel variants give the same numbers. DSL 4.6.0 (stable) deadlocks this kernel (nested elect_one, known). The kernel source is not the variable: with the compiler fixed, every flash-attention commit from 02-27 to main gives the same cycles (4.5.3 for 02-27..06-25 incl. the paper-era commits on 4.4.0 / 4.4.1, 4.8.0 for 06-25..main; details in Dao-AILab/flash-attention#3187). End to end this is 3-7% causal backward throughput at every sequence length (8k x 4: 1192 -> 1151 TFLOPS, 32k: 1298 -> 1251), non-causal unchanged at 8k+. flash-attention main requires >= 4.6.2 (4.8 APIs), so it cannot pin back.
What changed in the generated code.
- 4.7.0 and 4.7.1 embed the same CUDA 13.3 compiler build, so the +2.2% between them is the DSL lowering. Their PTX (
CUTE_DSL_KEEP=ptx) differs in ~45 lines out of 3800: 4.7.1 (and 4.6.3, 4.8.0, same PTX family) removes theadd.s32chains that 4.7.0 (and 4.6.2) hoist before the pipeline loops (precomputed mbarrier / stage byte offsets:add.s32 %r, %r, 8 / 16 / 24 / ... / 304 / 312 / 320 / 328) and re-derives the addresses at the use sites (add.s32 %r, %r, 128,setp.ne.b32 %r, %r, -1, extrabra.uni). In SASS this isUMOV450 -> 296 andMOV195 -> 149 static (nvdisasm of the dumped cubins): the uniform-register precomputation of the pipeline addresses is gone and address/phase arithmetic now sits next to eachmbarrier.try_wait/arrive. Assembling both PTX files with one externalptxas(13.1) reproduces the static SASS difference (+22 MOV, +21 IMAD, +8 UIADD3), so it is not ptxas. - 4.5.3 -> 4.6.2 is a lowering rewrite (3676 -> 3317 PTX instructions;
mov.b32634 -> 263,mov.b64399 -> 580,setp.*.s32->setp.*.b32,st.shared.v4.u32->.b32,mbarrier.arrive.expect_tx.release.cta-> non-release forms) together with the embedded NVVM 20 / ptxas 12.9 -> NVVM 23 / ptxas 13.3 switch; I cannot separate those two from outside (the 4.5.3 PTX does not assemble with ptxas 13.1). - ncu per-warp view (4.5.3 vs 4.6.3, 8k causal): the kernel issues 4% fewer instructions (712 M -> 683 M) but takes 5% more cycles; 128 registers in both, no spills (4.8.0 adds 131k local loads, 4.6.3 none, so spills are not it). The warpgroup that drains dQ from TMEM to smem and issues the bulk reduce (
LDTM/STS.128/FENCE.VIEW.ASYNC/BAR.SYNC) goes from 194 to 155 static instructions per iteration but its stall samples rise 13% (mbarrier try_wait, BAR.SYNC, FENCE); the softmax warpgroup +6%; the MMA warp is unchanged. Non-causal shows the same per-warp shifts with flat kernel time because it is MMA-bound (96k cycles per m-block vs 104k in causal); causal is bound by those warps, so it loses the 5%.
Repro.
# DSL version under test: 4.6.2 / 4.6.3 / 4.7.0 / 4.7.1 / 4.8.0 (4.5.3 needs 82d6441e + the fmax shim from the gist)
pip install nvidia-cutlass-dsl==4.7.1 quack-kernels==0.5.3 einops torch-c-dlpack-ext
git clone https://github.com/Dao-AILab/flash-attention && cd flash-attention
git checkout 5835c733 # fa4-v4.0.0.beta21, same cycles as 82d6441e
pip install --no-deps -e flash_attn/cute # package flash_attn.cute
# kernel cycles, independent of the SM clock
ncu --clock-control none \
--metrics sm__cycles_elapsed.avg,sm__cycles_elapsed.avg.per_second \
-k regex:BackwardSm100 --launch-skip 2 --launch-count 3 \
python bwd_once.py --seqlen 8192 --causal
# PTX / cubin of the backward kernel
CUTE_DSL_KEEP=ptx,cubin CUTE_DSL_DUMP_DIR=./dump python bwd_once.py --seqlen 8192 --causalbwd_once.py (one forward + 3 backward, hdim 128, 16 heads, 8k x 4):
import argparse, torch
from flash_attn.cute.interface import flash_attn_func
p = argparse.ArgumentParser()
p.add_argument("--seqlen", type=int, default=8192)
p.add_argument("--causal", action="store_true")
a = p.parse_args()
bs = 32768 // a.seqlen
torch.manual_seed(0)
q, k, v = [torch.randn(bs, a.seqlen, 16, 128, device="cuda", dtype=torch.bfloat16, requires_grad=True) for _ in range(3)]
g = torch.randn_like(q)
y = flash_attn_func(q, k, v, causal=a.causal)
y = y[0] if isinstance(y, tuple) else y
for _ in range(3):
for x in (q, k, v):
x.grad = None
y.backward(g, retain_graph=True)
torch.cuda.synchronize()PTX + SASS for all six versions, the 4.7.0 / 4.7.1 PTX diff, the ncu data and the scripts: https://gist.github.com/Johnsonms/46828fff34c33dd0915869e42cf8255e
Ask. Is the removal of the hoisted pipeline-address precomputation in 4.6.3+ intentional (register pressure?), and is there a pass option (CUTE_DSL_COMPILER_OPT) that restores the 4.6.2 / 4.7.0 behaviour? And can you separate the 4.5.3 -> 4.6.2 step (lowering rewrite vs NVVM 23) on your side? Cross-reference: Dao-AILab/flash-attention#3187.
Activity
Johnsonms commentedon Nov 20, 2026
Update after the source-side work (Dao-AILab/flash-attention#3200 and #3909 here).
Attribution. With the compiler fixed, every flash-attention commit from the paper to
maingives the same cycles; with the source fixed, the steps are 4.5.3 -> 4.6.2 (+2.9%) and 4.6.2 -> 4.6.3 (+2.2%, same embedded NVVM build in 4.7.0 / 4.7.1, so DSL lowering). The effect is per iteration and causal-only: a two-shape fit (8k / 32k) gives 3206 -> 3370 cycles per m_block iteration for causal from 4.5.3 to 4.8.0, 3264 -> 3273 for non-causal.What the newer lowering does, same source, 8k causal, ncu source pages per warp role. 4.6.2 -> 4.6.3: the 8 compute (softmax) warps' loop body gains 29 static instructions per iteration, all
IMAD.U32 R, RZ, RZ, URx/MOV R, URxcopies of one uniform base (the LDS / LDTM address) re-materialised before each use; the causal body carries the mask's live values, the non-causal body does not, which is why only causal pays. 4.5.3 -> 4.6.2: the 4 dQ-drain warps' stage loop loses 39 uniform-datapath instructions (UIADD3 / USHF / ULOP3 / ULEA) but gains 2BAR.SYNC+ 1FENCE.VIEW.ASYNC+ 1MEMBARper iteration, with +11% stall samples. Register budget is not the lever (compute warps 128 .. 160 registers: no gain, 128 is 3.5% worse).Residual after removing the redundant work on the FA side (mask only on the diagonal tiles, constant cluster destinations: flash-attention#3200, pipelines in #3909). Same gated source on 4.5.3 vs 4.8.0: compute loop body 376 -> 427 SASS instructions per iteration (+38 MOV, +17 IMAD, nothing else), dQ drain +2 BAR +1 FENCE +1 MEMBAR, kernel 3.389 -> 3.4385 M cycles (+1.5%). That is the part only the lowering can address; the refined ask is whether the lost uniform-register precomputation / the per-use UR->R re-materialisation in the compute warps is intentional, and whether a pass option restores the 4.5.3 behaviour.
Johnsonms commentedon Nov 20, 2026
Two additions from the follow-up measurements (flash-attention
main, B200, DSL 4.8.0 unless noted).A second, simpler instance of the lowering change: the old backward postprocess kernel (fp32 dQaccum -> bf16 dQ, flash-attention 06-25 tree, one elementwise pass over 32k tokens x 16 heads x 128, 384 MB of traffic). Same source: 73.7 us on DSL 4.5.3 vs 106.7 us on 4.8.0 (+45%), i.e. 5.2 TB/s vs 3.6 TB/s. ncu: identical grid (4096 x 128 threads), 64 KB dynamic SMEM per block, theoretical occupancy 18.75% on both; registers 112 -> 86; executed instructions 14.3 M -> 11.7 M; stall samples 4.8k -> 7.0k with the whole increase in long-scoreboard (3.3k -> 5.4k) at the
STS.128that consume theld.global.v4data. The 4.8 schedule keeps fewer global loads in flight, so at 12 warps/SM the HBM latency is exposed: lost memory-level parallelism. (flash-attentionmainhas since rewritten that kernel and runs 57 us on 4.8.0, so this is a characterisation, not a current-release problem.) PTX of both versions is in the gist directory of the original report; happy to add a standalone reproducer if useful.#3909 and flash-attention#3200 measured (main code, DSL 4.8.0; flash-attention's dense hdim 128 / 256 causal + non-causal test matrix on the patched package: 6232 passed, the 8 failures are pre-existing and identical on the stock package): the constant pair rank / mask alone gives 8k causal -1.7%, 32k -1.7%, non-causal -1.0%, hdim 256 -4.7% / -3.4%; with the flash-attention-side mask change the causal backward is back within ~1% of the DSL-4.5.3 build. The residual on identical source (compute warp loop body 376 -> 427 SASS instructions per iteration, +38 MOV +17 IMAD UR->R re-materialisation; dQ-drain +2 BAR +1 FENCE +1 MEMBAR per iteration) stands as the open item here.
[-][CuTe DSL][sm_100] Same kernel source, +3..5% cycles on DSL >= 4.6.2 vs 4.5.3 (FlashAttention-4 causal backward); 4.6.3 / 4.7.1 lowering drops hoisted mbarrier address precomputation[/-][+][CuTe DSL][sm_100] Lowering since 4.6.2 re-materialises uniform bases per use and adds BAR/FENCE in pipeline loops: +1.5% residual on FlashAttention-4 causal backward (was +3..5% before source-side fixes), +45% on an elementwise kernel[/+]