SPIRVIntrinsics: sub-group votes and collectives, unchecked shuffle lanes

[vchuravy]

Adds the sub-group votes and collectives that KernelInterface's sub-group operations need on SPIR-V back-ends (JuliaGPU/KernelAbstractions.jl#831), and fixes a miscompile of the existing sub-group shuffles.

Convergent builtins

The shuffles were declared without the convergent attribute, so LLVM was free to make them control-dependent on additional values. For

x = i <= m ? in[i] : 0
r = sub_group_shuffle(x, 1)

jump threading duplicated the shuffle into both arms of the branch. The work-items of a sub-group then executed different calls, which is undefined behavior; on PoCL every work-item got 0.

@builtin_ccall now takes a convergent = true option, used for the shuffles, the new votes and collectives, and the barriers (which had their own convergent_call! helper before).

Shuffle lanes

sub_group_shuffle converted the lane with UInt32(i - 1), which throws an InexactError (and leaves an error branch in the kernel) for an out-of-range lane, e.g. in a shuffle-up written as sub_group_shuffle(x, lane - offset). The lane is now passed modulo UInt32, and so is the mask of sub_group_shuffle_xor. An out-of-range lane gives an undefined value, as with the OpenCL C built-ins.

Votes

sub_group_any(::Bool) and sub_group_all(::Bool) (cl_khr_subgroups), and sub_group_ballot(::Bool)::NTuple{4, VecElement{UInt32}} (cl_khr_subgroup_ballot, OpenCL's uint4 mask). These call the OpenCL C built-ins, which the SPIR-V back-end lowers, because @builtin_ccall can't mangle the bool arguments of __spirv_GroupNonUniformAny/All/Ballot.

Collectives

The cl_khr_subgroups collectives for Int32, UInt32, Int64, UInt64, Float16, Float32 and Float64:

• sub_group_reduce_{add,min,max}
• sub_group_scan_inclusive_{add,min,max}
• sub_group_scan_exclusive_{add,min,max}
• sub_group_broadcast(x, lane), with a 1-based lane like sub_group_shuffle

The non-uniform arithmetic of cl_khr_subgroup_non_uniform_arithmetic (mul, and, or, xor, …) and the char/short types of cl_khr_subgroup_extended_types aren't included, because PoCL doesn't support them.

Tests

• The divergent shuffle and reduction tests fail without convergent.
• The bounds-checked divergent reduction needs pocl_jll 7.2.1+1, which includes WorkitemLoops: share local memory across region replicas pocl/pocl#2239.
• spirv2clc doesn't implement the vote and collective instructions, so those tests are skipped on the OpenCL C program back-end.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.29%. Comparing base (b6b2429) to head (2e42134).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #526   +/-   ##
=======================================
  Coverage   86.29%   86.29%           
=======================================
  Files          19       19           
  Lines        1642     1642           
=======================================
  Hits         1417     1417           
  Misses        225      225           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[maleadt]

LGTM. Reworked a bit, e.g., applying the convergent fix to other intrinsics and testing using Float16, but kept the fundamentals.

[christiangnrd]

LGTM