Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #874 +/- ##
==========================================
+ Coverage 1.96% 80.49% +78.52%
==========================================
Files 24 26 +2
Lines 1980 2210 +230
==========================================
+ Hits 39 1779 +1740
+ Misses 1941 431 -1510 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vchuravy
force-pushed
the
vc/dgsem-benchmarks-tiles
branch
from
October 11, 2026 12:07
db555f2 to
96ef92d
Compare
Contributor
Benchmark ResultsShow table
Benchmark PlotsA plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. |
vchuravy
changed the base branch from
vc/dgsem-benchmarks-localmem-layout
to
vc/dgsem-benchmarks-layout-switch
October 11, 2026 12:16
vchuravy
added this pull request to stack #873
October 11, 2026 12:20
vchuravy
force-pushed
the
vc/dgsem-benchmarks-tiles
branch
3 times, most recently
from
October 11, 2026 15:58
a5c8758 to
74f036b
Compare
With a KernelAbstractions that has tiles (#864), add DGSEM kernels in which the N² work-items of an element (2D) or of a slab (3D) form a tile and exchange values with `tile_shfl` instead of `@localmem` and `@synchronize`: the 2D weak form, the 2D half-sweep flux differencing (each two-point flux computed once, cyclically, for uniform control flow) and the 3D slab kernel. "weak/node localmem tile_barrier" keeps the local memory and replaces `@synchronize` with `tile_barrier`. The kernels live in benchmark/dgsem_tiles.jl, which is only included if KernelAbstractions defines `tile_shfl`, so the suite still runs on versions without tiles. Assisted-by: Claude Code (Opus 5.5)
vchuravy
force-pushed
the
vc/dgsem-benchmarks-tiles
branch
from
October 11, 2026 15:58
74f036b to
e133135
Compare
Open
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #875 (← #871 ← #870 ← #866). Uses the tiles of #864.
Restacked on #875 (the per-backend local-memory layout switch):
weak/node localmem tile_barriernow takes the layout too, and gets a variant in the backend's other layout. The timings below were taken before the restack, when the unsuffixed local-memory variants used the variable index first on every backend. On CUDA the two layouts are within 5% of each other in these kernels (see #875), so the comparisons still hold.This adds DGSEM kernels with tiles. The
N² = 16work-items of an element (2D) or of a slab (3D) form a tile, and they exchange values withtile_shflinstead of@localmemand@synchronize. The kernels are in a newbenchmark/dgsem_tiles.jl, whichdgsem.jlonly includes ifKernelAbstractionsdefinestile_shfl, so the suite still runs without #864. The variants:weak/node tile E=…(2D)weak/node localmem: the fluxes of the other nodes on a line come fromtile_shflweak/node localmem tile_barrier E=…(2D)weak/node localmemwithtile_barrierinstead of@synchronize, since each element reads only its own local memory; in the backend's default layout and the other one (#875)fluxdiff/node halfsweep tile E=…(2D)tile_shfl, as are the partner's state and metric terms. For control flow that is uniform over the tile, nodeicomputes the fluxes withi + o(cyclically) foro in 1:N÷2and gets those withi - ofrom those nodes; for evenNthe pair ato = N÷2is computed by both, with half the weightweak/slab tile E=…(3D)weak/slabwith the slab's fluxes fromtile_shfl: no local memory and no barrierThey use the N-D
ndrange(N, N, nelem)and narrow tiles that share a sub-group, from #864's latest commit (214d8c3). The lane of node(i, j)isi + N(j - 1).E(tiles per workgroup) ismin(128 ÷ N², tiles_per_workgroup(backend, N²)): 8 on CUDA, 1 on PoCL. Tile kernels are only added whentiles_per_workgroupallows the width. This adds 7 benchmarks per precision on backends with tiles.The design follows a local study of #864 for Trixi-style DGSEM kernels (posted as feedback on #864), but uses the N-D
ndrangeinstead of mapping lanes to nodes by hand.On PoCL,
tile_shflof any tuple of two or more elements fails at kernel compilation, and the process exits:#864 includes 4694613 ("POCL: shuffle packed structs with vector shuffles"), which shuffles packed structs with
__spirv_SubgroupShuffleINTELon devices withcl_intel_subgroups. Its commit message says PoCL's CPU device accepts these since pocl/pocl#2394, but that PR is still open. The releasedpocl_standalone_jll7.2.2+4 advertisescl_intel_subgroups, but rejects the module. Reproducer, with KA at 214d8c3 andpocl_standalone_jll7.2.2+4:Scalar shuffles and
NTuple{1}work. So until that is fixed in #864 (or PoCL), three of the four tile variants crash the CPU benchmark run: all excepttile_barrier. To check these kernels on PoCL, I swappedtile_shflof a tuple for a shuffle per element in a scratch copy. That copy is what the PoCL numbers below measure.Correctness
I compared every variant with
weak/elementorfluxdiff/elementin 2D and 3D (including a partially filled last workgroup), in Float32 and Float64: 92 of 92 within 100 eps on CUDA, and on PoCL with the per-element shuffles. Without #864 (KAmainfrom #866), the 82 non-tile checks still pass, so the include guard works. The tile versions of the weak form and the slab kernel give exactly the same results as their local-memory counterparts.Timings
Minimum / median in µs, at the suite's mesh sizes (2D 256² elements and 3D 32³ on CUDA).
CUDA, Quadro RTX 4000, CUDA.jl
vc/ki-subgroup-ops(CUDACore) and KA at 214d8c3. Both runs waited for an idle GPU; in both, another process had started by the time they ended.PoCL, 1 thread, with the per-element shuffle workaround (
E = 1: PoCL can't put narrow tiles side by side, so each tile is a workgroup of 16):Observations:
SHFL.tile_barriercosts the same as@synchronize.tile_barrieris 1.7× slower than@synchronize, with the same 16-work-item workgroups. With working vector shuffles (Accept SPV_INTEL_subgroups SPIR-V on the CPU device again pocl/pocl#2394) the shuffle kernels should improve: 4694613 measured 2.7× forSVector{4, Float32}.🤖 Generated with Claude Code