Repository navigation
test(studio): quarantine the seqplainkeys cases whose undo sometimes leaves the box off - #6990
Conversation
…eaves the box 240px off seqplainkeys-none-center-r0-root-z100 failed 2 of 3 runs in the merge group for two version-bump PRs (#6970, #6987). In each failing undo the bytes come back but the box stays 240.42px off, which is a Studio undo race, not noise. It stays measured and listed; #6989 tracks the fix, which removes this entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th the same undo race center-r30-root (undo box 313.6px off) and center-r0-nested (redo box 240.7px off) ejected 3 and 1 merge-queue entries since 10-30, the same race as center-r0-root. With all three quarantined, the gate passes on the shard results of all 8 failed merge-group runs since 10-30; main's gate fails each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jrusso1020
changed the title
![heygengenesis[bot]](https://avatars.githubusercontent.com/in/2913576?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 0 important · 1 nit
The quarantine is data-backed. I pulled the gate logs for all 8 cited merge-group runs, and each regressed case is one of these three ids (r0-root 5, r30-root 3, r0-nested 1). Run 41107324376 was the 28/30-shard shortfall. CI is green at this head.
Scope
Read ratchet.mjs gate/comment end-to-end. Quarantined ids leave regressed, overclaimed, unstable and both passing counts, so the gate still holds every other case. ratchet.test.mjs passes locally (21/21). The first CI run's red was Producer: integration tests (scene-unmount test), which this diff doesn't touch. The later CI run on the same head is green, edit-accuracy gate included. Non-blocking: issue #6989's body still names only center-r0-root and two runs. Adding r30-root and r0-nested would make it match the "per-run evidence" pointer.
— Genesis
| export const QUARANTINED = {}; | ||
| export const QUARANTINED = { | ||
| // Undo or redo of a keyframed plain element restores the bytes but sometimes leaves the box 240-314px off. | ||
| "seqplainkeys-none-center-r0-nested-z100": "#6989", |
There was a problem hiding this comment.
nit: comment() renders this value as (fixed by #6989), and that line is already in this PR's sticky comment. #6989 is the open issue, not a fix, so a reader skimming a later PR comment will think the race is fixed. Two ways to handle it: relabel the line tracked by in comment(), which means updating the ratchet.test.mjs:79-80 expectation too, or just accept the wording until the fixing PR swaps in its own number.
…ame undo race The race also hits the pct and px placements: 5 more cases left the box 230-400px off after undo or redo in the 9 failed merge-group runs since 10-30. Merge-group run 41485305789 bounced #6990 on px-r0-nested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jrusso1020
changed the title
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 1 important · 0 nits · cdb503e → e0c9844
- important: #6989 was closed as completed at 10:15 (#6998, then #7005 at 12:49), but merge groups based after both fixes still regress
center-r0-rootandpx-r0-nested(41481903765, 41485305789, 41489294150). With the tracker closed, no PR will remove these 8 entries, and the sticky comment says(fixed by #6989). Please reopen #6989 or point the entries at a new open issue.
What I checked
- The delta is the 8-id
QUARANTINEDlist only. Quarantined ids drop out ofregressed, both sides of the passing count, andoverclaimed, so they can no longer fail the gate. - There were 8 more gate failures today that aren't in the replayed set. 6 regressed only on
center-r0-rootand/orpx-r0-nested, and the passing-count drop equals the regressed count, so this list clears them. The other 2 (41448126362, 41471385877) were 29/30 shard shortfalls, which exit 2 either way. - Residual risk:
px-r0-root,pct-r0-nested, andcenter-r30-nestedare still gated and show up as Unstable withundo falsein 1 of 3 runs in several of those groups (e.g. 41439540336, 41481903765, 41489294150). A 2-of-3 failure there will bounce the queue again. - CI at e0c9844: CI (edit accuracy gate passes, 8 quarantined), Windows tests/render, CodeQL, regression, and captures are green. A duplicate Windows run from the 17:27 re-trigger was still queued when I posted.
What
Quarantine eight edit-accuracy cases in the
seqplainkeys-none-*family, so they are still measured and listed every run but no longer gated:center-r0-root,center-r30-root,center-r0-nestedpct-r0-root,pct-r30-rootpx-r0-nested,px-r30-nested,px-r30-root(full ids are
seqplainkeys-none-<placement>-<rotation>-<nesting>-z100)Why
This family caused 9 of the 10 "Studio: edit accuracy gate" merge-queue failures since 10-30. The other run (41107324376) listed no regressed case. Every failure was on a PR that does not touch Studio, including the v0.9.0 release PRs #6970 and #6987, and this PR's own first merge group (41485305789, bounced on
px-r0-nested). The shard artifacts show the same Studio race each time: undo or redo restores the file bytes but leaves the box a fixed distance off.Runs with the offset, across the 9 failed merge-group runs whose artifacts I read:
undo.box240.4pxundo.box313.6pxredoBox240.7pxundo.box328.7pxundo.box396.2pxredoBox240.8pxredoBox232.6pxundo.box380.6pxThe offset is the same every time for a given case, so it is a real bug, not timing noise, which is the case
QUARANTINEDexists for. The other four cases in the family (center-r30-nested,pct-r0-nested,pct-r30-nested,px-r0-root) never showed an offset in those runs, so they stay gated.Related work
Refs #6989, which has the per-run evidence. The PR that fixes the race removes these entries and re-banks the cases, per the
QUARANTINEDdoc comment.How
Eight entries in
QUARANTINED. Each value points at the tracking issue, not a fixing PR, because no fix exists yet. They still satisfy the#\d+check inratchet.test.mjs.Test plan
Replayed
ratchet.mjs gate --shards 30on every shardresults.json(first runs and reruns) from each of the 9 failed merge-group runs (41072670682, 41086527228, 41167835897, 41274079397, 41320729167, 41361409037, 41385906495, 41403756435, 41485305789), with main'sbaseline.jsonas base. main'sratchet.mjsexits 1 on all 9; this branch exits 0 on all 9, with no regressed or unstable case.ratchet.test.mjspasses (21 tests), including the#\d+check on everyQUARANTINEDvalue.lefthook pre-commit (oxfmt, oxlint, fallow) and commitlint pass.
Unit tests added/updated
Manual testing performed
Documentation updated (if applicable)
Comments follow CONTRIBUTING.md "Comments": they say why, not what, and a bug fix says what the code must do and how to reproduce the bug
— Rames
🤖 Generated with Claude Code