fix(array): give Array::iter/rev_iter/iter2 live semantics - #3972
Open
bobzhang wants to merge 1 commit into
Open
fix(array): give Array::iter/rev_iter/iter2 live semantics#3972bobzhang wants to merge 1 commit into
bobzhang wants to merge 1 commit into
Conversation
Array::iter previously delegated to self[:].iter(), freezing (buf, len) in a view at iterator creation. Because shrink operations (remove, pop, truncate, clear, drain) null out vacated slots for GC, a later iterator step would read a nulled slot through unsafe_get and hand out an invalid value of type T: a null deref / trap on native and wasm-gc, undefined on JS (see #2195). The iterators now close over the array itself and re-read the live length and buffer at every step, with the element access adjacent to the bounds test, so unsafe_get stays sound under structural mutation: appended elements are visited, removals skip the shifted successor, and iteration stops as soon as the array shrinks below the cursor. rev_iter clamps its cursor to the live length before each step. ArrayView::iter and FixedArray::iter are intentionally unchanged: a view's contract is a frozen (buf, start, len) snapshot, and a FixedArray's buffer can never move or resize. xs[:].iter() remains the explicit opt-in for snapshot iteration with today's hoisted codegen. Note: `for x in xs` does not go through Array::iter - the compiler specializes that lowering with the same frozen-snapshot pattern, so it keeps the old (unsound) behavior until the compiler is updated to match. Docs on each/eachi/fold, which are written with `for .. in`, now say so explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes unsound behavior in Array::iter, Array::rev_iter, and Array::iter2 when an Array is structurally mutated during iteration (shrink operations that null vacated slots for GC and growth operations that may reallocate), addressing the backend inconsistencies described in #2195.
Changes:
- Reimplemented
Array::iter,Array::rev_iter, andArray::iter2to use live semantics by re-reading the array’s current length/buffer on each iterator step. - Updated docs for
Array::each/Array::foldto accurately describe thatfor .. inoverArrayis currently compiler-lowered with frozen traversal bounds. - Added tests covering clear/remove/push/shrink behavior during live iteration for
iter,rev_iter, anditer2.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| builtin/array.mbt | Implements live array iterators and updates documentation to distinguish live iterator APIs from compiler-lowered for .. in behavior. |
| builtin/array_test.mbt | Adds regression tests validating safety and determinism of the new live-iterator semantics under structural mutation. |
Suppressed comments (2)
builtin/array.mbt:1821
rev_iter()is documented as live under mutation. Since the array can shrink/grow after iterator creation, the exact number of remaining elements is not statically known. Consider omittingsize_hinthere (or documenting that the hint may become inexact under mutation) to better match theIter::newcontract.
},
size_hint=self.length(),
)
builtin/array.mbt:1860
- With live semantics,
iter2()can yield more pairs than were present at creation time if the array grows during iteration, so an initialsize_hint=self.length()can become inaccurate while elements remain. Consider omittingsize_hintfor consistency with other mutation-tolerant iterators.
},
size_hint=self.length(),
)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1749
to
+1753
| /// `rev_iter()` and `iter2()` have the same live semantics. Note that | ||
| /// `for x in xs` and helpers written with it (`each()`, `eachi()`, `fold()`) | ||
| /// currently use the compiler's specialized lowering, which still fixes the | ||
| /// traversal bounds up front; they do not get live semantics until the | ||
| /// compiler lowering is updated to match. |
Comment on lines
+1768
to
+1776
| Iter::new( | ||
| fn() { | ||
| guard i < self.length() else { None } | ||
| let elem = self.unsafe_get(i) | ||
| i += 1 | ||
| Some(elem) | ||
| }, | ||
| size_hint=self.length(), | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Array::iterdelegates toself[:].iter(), which freezes(buf, len)in a view at iterator creation. Shrink operations (remove,pop,truncate,clear,drain) go throughunsafe_truncate_to_length, which nulls the vacated slots for GC. A later iterator step then reads a nulled slot throughunsafe_getagainst the stale bounds and hands out an invalid value of typeT— a null deref/trap on native and wasm-gc,undefinedleaking in on JS. See #2195.Growth has the dual problem: a
pushthat reallocates detaches the iterator onto the old buffer.Fix
Array::iter,Array::rev_iter, andArray::iter2now close over the array itself and re-read the live length and buffer at every step, with the element access adjacent to the bounds test.unsafe_getremains sound under any structural mutation because the bounds evidence never outlives the step that produced it:rev_iterclamps its cursor to the live length.Cost: two dependent loads per step (
self.len,self.buf) instead of two registers. When the loop body provably cannot mutate the array, both loads are loop-invariant and hoisting recovers the old codegen exactly; when the body is opaque, the loads are precisely the price of soundness.Intentionally unchanged:
ArrayView::iter(a view's contract is a frozen(buf, start, len)snapshot —xs[:].iter()stays the explicit opt-in for snapshot iteration with the hoisted codegen) andFixedArray::iter(buffer can never move or resize, so the hoisted form is already sound).Caveat:
for x in xsis compiler-specializedfor x in xsover anArraydoes not go throughArray::iter— the compiler specializes the lowering with the same frozen-snapshot pattern, so it keeps the old behavior until the compiler is updated to match (tests in this PR drive the iterators explicitly for exactly this reason; internal issue compiler/ideas#2130 tracks the compiler side). Helpers written withfor .. in(each,eachi,fold) therefore also keep snapshot behavior for now; their docs now say so explicitly instead of claiming UB territory.No
.mbtichanges — the public interface is untouched. Tests pass on wasm-gc, native, and js targets.Follow-ups (out of scope here)
for .. inArray specialization to the same check-adjacent live loop, at which pointeach/eachi/foldinherit live semantics for free.let b = a[:]; a.pop() |> ignore; b[2]reads a nulled slot through the view's stale bounds.🤖 Generated with Claude Code