Skip to content

fix(array): give Array::iter/rev_iter/iter2 live semantics - #3972

Open
bobzhang wants to merge 1 commit into
mainfrom
array-iter-live-semantics
Open

fix(array): give Array::iter/rev_iter/iter2 live semantics#3972
bobzhang wants to merge 1 commit into
mainfrom
array-iter-live-semantics

Conversation

@bobzhang

@bobzhang bobzhang commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Problem

Array::iter delegates to self[:].iter(), which freezes (buf, len) in a view at iterator creation. Shrink operations (remove, pop, truncate, clear, drain) go through unsafe_truncate_to_length, which nulls the vacated slots for GC. A later iterator step then reads a nulled slot through unsafe_get against the stale bounds and hands out an invalid value of type T — a null deref/trap on native and wasm-gc, undefined leaking in on JS. See #2195.

Growth has the dual problem: a push that reallocates detaches the iterator onto the old buffer.

Fix

Array::iter, Array::rev_iter, and Array::iter2 now close over the array itself and re-read the live length and buffer at every step, with the element access adjacent to the bounds test. unsafe_get remains sound under any structural mutation because the bounds evidence never outlives the step that produced it:

  • elements appended during iteration are visited;
  • removing an element shifts its successors left, so the element moving into the current slot is skipped (the classic live-iteration wart — but type-safe and deterministic);
  • iteration stops as soon as the array shrinks to the cursor; rev_iter clamps its cursor to the live length.

Cost: two dependent loads per step (self.len, self.buf) instead of two registers. When the loop body provably cannot mutate the array, both loads are loop-invariant and hoisting recovers the old codegen exactly; when the body is opaque, the loads are precisely the price of soundness.

Intentionally unchanged: ArrayView::iter (a view's contract is a frozen (buf, start, len) snapshot — xs[:].iter() stays the explicit opt-in for snapshot iteration with the hoisted codegen) and FixedArray::iter (buffer can never move or resize, so the hoisted form is already sound).

Caveat: for x in xs is compiler-specialized

for x in xs over an Array does not go through Array::iter — the compiler specializes the lowering with the same frozen-snapshot pattern, so it keeps the old behavior until the compiler is updated to match (tests in this PR drive the iterators explicitly for exactly this reason; internal issue compiler/ideas#2130 tracks the compiler side). Helpers written with for .. in (each, eachi, fold) therefore also keep snapshot behavior for now; their docs now say so explicitly instead of claiming UB territory.

No .mbti changes — the public interface is untouched. Tests pass on wasm-gc, native, and js targets.

Follow-ups (out of scope here)

  • Compiler: switch the for .. in Array specialization to the same check-adjacent live loop, at which point each/eachi/fold inherit live semantics for free.
  • Optional fail-fast: compare the live length against the creation-time length each step and abort on mismatch ("array modified during iteration"). Catches the common accident cheaply; full precision needs a mod-count word in the Array header — separate discussion.
  • Views + shrink is a remaining (narrower) soundness hole independent of iteration: let b = a[:]; a.pop() |> ignore; b[2] reads a nulled slot through the view's stale bounds.

🤖 Generated with Claude Code

Array::iter previously delegated to self[:].iter(), freezing (buf, len)
in a view at iterator creation. Because shrink operations (remove, pop,
truncate, clear, drain) null out vacated slots for GC, a later iterator
step would read a nulled slot through unsafe_get and hand out an invalid
value of type T: a null deref / trap on native and wasm-gc, undefined
on JS (see #2195).

The iterators now close over the array itself and re-read the live
length and buffer at every step, with the element access adjacent to the
bounds test, so unsafe_get stays sound under structural mutation:
appended elements are visited, removals skip the shifted successor, and
iteration stops as soon as the array shrinks below the cursor.
rev_iter clamps its cursor to the live length before each step.

ArrayView::iter and FixedArray::iter are intentionally unchanged: a
view's contract is a frozen (buf, start, len) snapshot, and a
FixedArray's buffer can never move or resize. xs[:].iter() remains the
explicit opt-in for snapshot iteration with today's hoisted codegen.

Note: `for x in xs` does not go through Array::iter - the compiler
specializes that lowering with the same frozen-snapshot pattern, so it
keeps the old (unsound) behavior until the compiler is updated to match.
Docs on each/eachi/fold, which are written with `for .. in`, now say so
explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 4, 2026 03:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes unsound behavior in Array::iter, Array::rev_iter, and Array::iter2 when an Array is structurally mutated during iteration (shrink operations that null vacated slots for GC and growth operations that may reallocate), addressing the backend inconsistencies described in #2195.

Changes:

  • Reimplemented Array::iter, Array::rev_iter, and Array::iter2 to use live semantics by re-reading the array’s current length/buffer on each iterator step.
  • Updated docs for Array::each/Array::fold to accurately describe that for .. in over Array is currently compiler-lowered with frozen traversal bounds.
  • Added tests covering clear/remove/push/shrink behavior during live iteration for iter, rev_iter, and iter2.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
builtin/array.mbt Implements live array iterators and updates documentation to distinguish live iterator APIs from compiler-lowered for .. in behavior.
builtin/array_test.mbt Adds regression tests validating safety and determinism of the new live-iterator semantics under structural mutation.
Suppressed comments (2)

builtin/array.mbt:1821

  • rev_iter() is documented as live under mutation. Since the array can shrink/grow after iterator creation, the exact number of remaining elements is not statically known. Consider omitting size_hint here (or documenting that the hint may become inexact under mutation) to better match the Iter::new contract.
    },
    size_hint=self.length(),
  )

builtin/array.mbt:1860

  • With live semantics, iter2() can yield more pairs than were present at creation time if the array grows during iteration, so an initial size_hint=self.length() can become inaccurate while elements remain. Consider omitting size_hint for consistency with other mutation-tolerant iterators.
    },
    size_hint=self.length(),
  )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread builtin/array.mbt
Comment on lines +1749 to +1753
/// `rev_iter()` and `iter2()` have the same live semantics. Note that
/// `for x in xs` and helpers written with it (`each()`, `eachi()`, `fold()`)
/// currently use the compiler's specialized lowering, which still fixes the
/// traversal bounds up front; they do not get live semantics until the
/// compiler lowering is updated to match.
Comment thread builtin/array.mbt
Comment on lines +1768 to +1776
Iter::new(
fn() {
guard i < self.length() else { None }
let elem = self.unsafe_get(i)
i += 1
Some(elem)
},
size_hint=self.length(),
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants