Declare indirect access on compiled kernels

[michel2323]

Fixes #661.

Reductions of SharedBuffer-backed arrays intermittently returned 0. This isn't a synchronization race, isn't caused by #654, and isn't specific to the LTS stack or Julia 1.13. It reproduces on an Arc A750 with the rolling stack and Julia 1.12, with one thread or several, and with or without ONEAPI_SYNC_EACH_SUBMISSION. In some processes 30–80% of the reductions fail.

Cause

It only happens when a shared allocation reuses the address of one that was freed. Looping the testset with a GC.gc() every 10 iterations, 1511 of 1512 failures had a reused input address. Keeping every array alive gave no failures at all. Simple kernels that only read or only write a shared array were fine, and HostBuffer/DeviceBuffer never failed.

Kernels get their arrays as oneDeviceArray structs, so to Level Zero every USM pointer they dereference is an indirect access. We never declared that (zeKernelSetIndirectAccess was wrapped but unused). The driver therefore only maps allocations we explicitly made resident with zeContextMakeMemoryResident, and release frees without evicting. A new shared allocation at a freed address then isn't visible to the kernel, which reads zeros. Removing make_resident for shared buffers raises the failure rate to 100%, which is consistent with this.

That this showed up as a 1.13-only flake on Aurora is probably GC timing: whether a shared array from an earlier test was collected and its address reused before this testset runs.

Fix

Set the DEVICE | HOST | SHARED indirect-access flags once, when a compiled kernel is linked. As far as I know, SYCL's Level Zero adapter does the same for every kernel. Kernels built directly through oneL0 are unaffected.

I preferred this to evicting before free, which also fixes the reproducer. Eviction isn't queue-ordered, so evicting a buffer from a finalizer while in-flight work still uses it risks a GPU page fault on the rolling stack, as the existing comment in release notes.

Results on the A750, 2000 iterations with GC.gc() every 10:

Variant	Failures
current main	1627
evict shared buffers before free	0
indirect access, make_resident kept (this PR)	0
indirect access, make_resident removed	0
no make_resident for shared buffers	2000

Launch overhead with 600 extra live allocations: about +0.2–0.35 µs per async launch (7.6 → 7.9 µs), and no measurable difference for launch+sync.

Test

The new reusing freed $B allocations testset allocates two arrays per iteration, frees them in alternating order and runs periodic GCs. Whether a reused allocation reads as zeros depends on allocator state: the existing testset passes in a fresh process, which is why #649 didn't catch this. With this much churn, the test failed in 8 of 8 processes without the fix (73–100 of 100 iterations bad) and passes with it.

Locally (A750, Julia 1.12) array, execution and level-zero pass. Still to do:

• ALCF LTS CI (1.10 and 1.13)
• Possible follow-up: with indirect access declared, the per-allocation make_resident (and the host-buffer "NotPresent pagefault" workaround in pool.jl, which looks like the same bug) may no longer be needed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.36%. Comparing base (f3c41ae) to head (da8277f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #662      +/-   ##
==========================================
+ Coverage   79.35%   79.36%   +0.01%     
==========================================
  Files          57       57              
  Lines        4083     4085       +2     
==========================================
+ Hits         3240     3242       +2     
  Misses        843      843              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[michel2323]

Going to look into removing the LTS workarounds once the ALCF CI is back online.