Skip to content

[CuTe DSL] SM100 pipelines: constant leader rank and pair mask when the cluster is one 2CTA pair - #3909

Open
Johnsonms wants to merge 1 commit into
NVIDIA:mainfrom
Johnsonms:dsl-2cta-constant-cluster-targets
Open

Johnsonms wants to merge 1 commit into
NVIDIA:mainfrom
Johnsonms:dsl-2cta-constant-cluster-targets

Conversation

@Johnsonms

@Johnsonms Johnsonms commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PipelineTmaUmma, PipelineAsyncUmma, PipelineUmmaAsync (and PipelineTmaMultiConsumersAsync through the shared helper) derive the leading CTA rank and the multicast / peer masks from block_idx_in_cluster at kernel start and keep them in registers for every arrival. When the cluster is exactly one 2CTA pair, cluster shape (2, 1, 1, 1), both CTAs get the same values: leader rank 0 and pair mask 0b11. This PR returns those constants in that case (_is_single_pair_cluster), so the arrivals carry no rank arithmetic; other cluster shapes keep the generic path.

Measured on the FlashAttention-4 SM100 backward (Dao-AILab/flash-attention main, B200, DSL 4.8.0 package with this change applied, ncu --clock-control none): seven mapa computations fold to a literal and the stack allocation drops 16 -> 0 bytes; kernel cycles 8k causal 3.548 M -> 3.486 M (-1.7%), 32k causal 12.53 M -> 12.32 M (-1.7%), 8k non-causal 6.216 M -> 6.151 M (-1.0%), hdim 256 causal 10.09 M -> 9.61 M (-4.7%), non-causal 18.26 M -> 17.64 M (-3.4%). Outputs and dK / dV bitwise identical, dQ within its fp32 reduce-order noise; flash-attention's tests/cute/test_flash_attn.py hdim 128 / 256 subsets pass. examples/.../fp16_gemm_4.py (preferred cluster (2, 4, 1): generic path, fallback (2, 1, 1): constant path) passes verification.

Context: this is one of the two source-side items behind the DSL >= 4.6 causal-backward regression in #3907 / Dao-AILab/flash-attention#3187.

Scope: this is the DSL-side counterpart of the constant-cluster-destination half of Dao-AILab/flash-attention#3200 (its other half, masking only the diagonal tiles, is flash-attention-only). It removes work at the source; it does not address the residual lowering change tracked in #3907 (per-use re-materialisation of uniform bases in the compute warps, extra BAR / FENCE in the dQ-drain loop), which stays open there.

…he cluster is one 2CTA pair

PipelineTmaUmma, PipelineAsyncUmma, PipelineUmmaAsync and PipelineTmaMultiConsumersAsync
derive the leading CTA rank and the multicast / peer masks from the CTA rank at runtime. When
the cluster is exactly one 2CTA pair (cluster shape (2, 1, 1, 1)) both CTAs get the same
values, rank 0 and mask 0b11, so return those constants instead. Every arrival on these
pipelines then carries no rank arithmetic (seven mapa computations fold to a literal in the
FlashAttention-4 SM100 backward, 8k causal -1.7% kernel cycles, non-causal -1.1%).
Other cluster shapes keep the generic path.
@Johnsonms

Copy link
Copy Markdown
Contributor Author

Validation after opening the PR (B200, DSL 4.8.0 package with this change applied, FlashAttention-4 SM100 backward on Dao-AILab/flash-attention main 68bd095c, code unchanged; details in #3907).

Kernel cycles (ncu --clock-control none): 8k causal 3.548 M -> 3.486 M (-1.7%), 32k causal 12.53 M -> 12.32 M (-1.7%), 8k non-causal 6.216 M -> 6.151 M (-1.0%), hdim 256 causal 10.09 M -> 9.61 M (-4.7%), hdim 256 non-causal 18.26 M -> 17.64 M (-3.4%). The FA-side copy of the same constants (Dao-AILab/flash-attention#3200) gives the same numbers, so the gain is fully explained by the constant rank / masks.

Paper-style TFLOPS (-lgc 1965, hdim 128, 16 heads, 32k tokens per shape), causal 1k / 2k / 4k / 8k / 16k / 32k: 550 / 796 / 1009 / 1166 / 1279 / 1247 -> 555 / 807 / 1014 / 1175 / 1276 / 1259.

Tests: flash-attention tests/cute/test_flash_attn.py dense hdim 128 + 256, causal + non-causal, all variants on the patched package: 6232 passed, 3744 skipped, 8 failed; the 8 failures (learnable_sink + gqa/mqa) are identical on the stock package. Outputs and dK / dV bitwise identical to the stock package. examples/python/CuTeDSL/cute/blackwell/tutorial/tutorial_gemm/fp16_gemm_4.py passes with preferred cluster (2, 4, 1) (generic path) and fallback (2, 1, 1) (constant path).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant