Skip to content

test: make workspace-switch and kit-registry concurrency tests deterministic - #870

Merged
nickolas-dimitrakas merged 1 commit into
mainfrom
test/stabilize-flaky-concurrency-tests
Aug 31, 2026
Merged

test: make workspace-switch and kit-registry concurrency tests deterministic#870
nickolas-dimitrakas merged 1 commit into
mainfrom
test/stabilize-flaky-concurrency-tests

Conversation

@nickolas-dimitrakas

Copy link
Copy Markdown
Contributor

Stacked on #867. Review that one first — this PR's base is ci/bound-simulator-and-network-hangs, so the diff here is only the test changes.

Why

These are the two tests that repeat most often in the ObjC crash failures blocking CI:

  • MParticleTests.testSwitchWorkspaceKitsWithStop
  • MPKitContainerTests.testActiveKitsRegistryThreadSafety

They turned out to have two completely different problems, and only one of them is a test problem.

testSwitchWorkspaceKitsWithStop — genuinely a bad test

It waited on two nested hard-coded 10-second dispatch_after blocks, so it always burned at least 20 seconds of wall clock and depended on fixed timing rather than on the SDK actually being ready. Under CI load, "10 seconds is surely enough" is exactly the assumption that produces flakes.

It now waits on the SDK's own readiness signals — MParticle.initialized after startWithOptions:, then the replacement shared instance after switchWorkspaceWithOptions: — by spinning the main run loop until the condition holds, so main-queue work the SDK schedules still runs.

The assertions are unchanged. Same behaviour checked, no weakening.

Before After
testSwitchWorkspaceKitsWithStop 20.2s 0.11s

Verified: 3/3 runs of the test alone, and 4/4 runs of the whole MParticleTests class (62 tests, 0 failures).

MParticle.initialized is exposed to the test via the existing @interface MParticle () block, alongside the privates already redeclared there. No product change.

testActiveKitsRegistryThreadSafety — the test is right, the SDK is wrong

This one is not flaky tooling. It is correctly catching a real data race, reproduced at 2 crashes in 10 runs of that test on its own, nothing else running.

flushSerializedKits (MPKitContainer.m:217) fast-enumerates kitsRegistry and mutates it through freeKit: on the main queue without taking kitsSemaphore — the lock every other accessor of that set uses. activeKitsRegistry reads it under the semaphore; configureKits: mutates it under the semaphore. And configureKits:nil reaches the unlocked flush via an early return before the lock is acquired (MPKitContainer.m:2027) — precisely what the test calls in a loop while three threads read the registry.

So the crash is mutation-during-enumeration on shared state, and the test is doing its job.

What this PR does and does not do

Fixed here (test-side defects, real but not the cause of the crash):

  • encounteredError was a plain __block BOOL written from four concurrent blocks — an unsynchronized flag is itself a data race. Now guarded by an NSLock, recording only the first exception.
  • XCTFail was called from those background queues, which is not safe with XCTest. Failures are now reported from the dispatch_group_notify block on the main queue.

Not fixed here: the race itself. This test will still flake at roughly its current rate until MPKitContainer locking is addressed. That is deliberate — a core-concurrency change in a public SDK does not belong in a CI-stabilization PR, and AGENTS.md asks for coordination on anything touching the kit interface.

Notes for whoever takes the race

Groundwork already checked:

  • All three flushSerializedKits callers (mParticle.m:611, mParticle.m:722, MPKitContainer.m:2029) are outside locked regions, so taking kitsSemaphore there would not deadlock.
  • There is no dispatch_sync to the main queue inside any locked region.
  • freeKit: must stay lock-free: configureKits: already calls it while holding the semaphore, and dispatch_semaphore is not recursive.
  • The naive fix (wrap the main-queue flush in the semaphore) would let a long background critical section stall the main thread — the locked regions at 1149–1624 and 1656–1939 are large. Snapshotting the registry under the lock and iterating the snapshot off-lock avoids that, but freeKit: would still read kitsRegistry unlocked, so it is only a partial fix.

Scope left alone

testSwitchWorkspaceOptions and the other tests in that family still use the 10-second WORKSPACE_SWITCHING_DELAY (three nested delays, ~30s). The macro is retained for them. The same treatment would cut roughly another 60s off the ObjC suite, but they were not among the tests flagged as flaky, so they are out of scope here.

No CHANGELOG.md entry: test-only, no consumer-facing SDK change.

🤖 Generated with Claude Code

@nickolas-dimitrakas
nickolas-dimitrakas requested a review from a team as a code owner August 27, 2026 13:35
@cursor

cursor Bot commented Aug 27, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Changes are limited to unit tests; assertions are preserved and no shipping SDK code paths are modified.

Overview
Test-only CI stabilization for two frequently failing ObjC tests, with no production SDK behavior changes.

testSwitchWorkspaceKitsWithStop drops nested fixed 10s dispatch_after delays in favor of MPWaitForCondition, which spins the main run loop until MParticle.initialized after startWithOptions: and until a new shared instance is initialized after switchWorkspaceWithOptions:. Same assertions (kit with stop stays in the registry); runtime drops from ~20s to sub-second under load.

testActiveKitsRegistryThreadSafety fixes test harness issues: the shared error flag is now NSLock-protected, and XCTFail runs on the main queue in dispatch_group_notify instead of from concurrent worker blocks. It does not fix the underlying MPKitContainer race (flushSerializedKits vs activeKitsRegistry / configureKits:); that test can still crash until kit locking is addressed separately.

The MParticle test category gains initialized (already public on the class) and trims unused private method declarations.

Reviewed by Cursor Bugbot for commit 0a33db8. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown

📦 SDK Size Impact Report

Measures how much the SDK adds to an app's size (with-SDK minus without-SDK).

Metric Target Branch This PR Change
App Bundle Impact 1.81 MB 1.81 MB +N/A
Executable Impact 848 bytes 848 bytes +N/A
XCFramework Size 6.51 MB 6.51 MB +N/A

➡️ SDK size impact change is minimal.

Raw measurements

Target branch (main):

{"baseline_app_size_kb":84,"baseline_executable_size_bytes":75464,"with_sdk_app_size_kb":1936,"with_sdk_executable_size_bytes":76312,"sdk_impact_kb":1852,"sdk_executable_impact_bytes":848,"xcframework_size_kb":6664}

This PR:

{"baseline_app_size_kb":84,"baseline_executable_size_bytes":75464,"with_sdk_app_size_kb":1936,"with_sdk_executable_size_bytes":76312,"sdk_impact_kb":1852,"sdk_executable_impact_bytes":848,"xcframework_size_kb":6664}

@nickolas-dimitrakas
nickolas-dimitrakas marked this pull request as draft August 27, 2026 13:37
@nickolas-dimitrakas
nickolas-dimitrakas force-pushed the test/stabilize-flaky-concurrency-tests branch from 192ce45 to 5006dff Compare August 27, 2026 14:25
@nickolas-dimitrakas nickolas-dimitrakas self-assigned this Aug 27, 2026
@nickolas-dimitrakas
nickolas-dimitrakas marked this pull request as ready for review August 27, 2026 20:19
@nickolas-dimitrakas
nickolas-dimitrakas force-pushed the test/stabilize-flaky-concurrency-tests branch from 5006dff to b0a010a Compare August 31, 2026 17:27
Base automatically changed from ci/bound-simulator-and-network-hangs to main August 31, 2026 17:55
…inistic

testSwitchWorkspaceKitsWithStop waited on two nested 10-second dispatch_after
blocks, so it always burned at least 20 seconds of wall clock and depended on
fixed timing rather than on the SDK actually being ready. Wait on the SDK's own
readiness signals instead - MParticle.initialized after startWithOptions:, and
the replacement shared instance after switchWorkspaceWithOptions: - by spinning
the main run loop until the condition holds. The assertions are unchanged, so
the test still checks the same behaviour; it now takes 0.11s instead of 20.2s.

testActiveKitsRegistryThreadSafety tracked failure in a plain __block BOOL
written from four concurrent blocks, which is itself an unsynchronized data
race, and called XCTFail from those background queues. Guard the shared state
with an NSLock, record only the first exception, and report it from the
dispatch_group_notify block on the main queue.

This commit does not change the flakiness of testActiveKitsRegistryThreadSafety
itself: that crash is a real race in MPKitContainer (flushSerializedKits
enumerates and mutates kitsRegistry without kitsSemaphore, reached via the
early return in configureKits: before the lock is taken). Reproduced at 2
crashes in 10 runs of that test alone. Fixing it changes core SDK locking, so
it is deliberately left for a separate, explicitly-reviewed change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@nickolas-dimitrakas
nickolas-dimitrakas force-pushed the test/stabilize-flaky-concurrency-tests branch from b0a010a to 0a33db8 Compare August 31, 2026 17:55

@BrandonStalnaker BrandonStalnaker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good but please make sure all the CI succeeds before merging

@nickolas-dimitrakas
nickolas-dimitrakas merged commit 5b92a61 into main Aug 31, 2026
78 checks passed
@nickolas-dimitrakas
nickolas-dimitrakas deleted the test/stabilize-flaky-concurrency-tests branch August 31, 2026 18:09
nickolas-dimitrakas added a commit that referenced this pull request Sep 2, 2026
Brings this branch's native-tests / build-kits CI up to date with main's #867
(stop simulator cloning that starved the daemons one Swift-scheme test blocks
on; bound the SPM resolve with a per-attempt watchdog instead of trusting
git's own low-speed abort, which does not reliably fire) and #870 (the
workspace-switch tests moved off nested 10-second dispatch_after chains onto
MPWaitForCondition, which polls the SDK's own readiness state instead of
guessing at a fixed delay).

Auto-merged cleanly everywhere except UnitTests/ObjCTests/MParticleTests.m,
where both branches had independently rewritten the same five workspace-switch
test bodies - main by removing dispatch_after for MPWaitForCondition, this
branch by adding migration-specific assertions (resetRegistry,
kitContainer_PRIVATE, registeredKits.anyObject) on top of the old dispatch_after
version. Git's line-based merge "succeeded" there without a conflict marker,
but silently interleaved hunks from both rewrites - 10 dispatch_after calls
survived alongside 3 MPWaitForCondition calls, split unevenly across the five
methods. Replaced that whole block with a hand-reconciled version carrying
both: MPWaitForCondition throughout, this branch's own assertions preserved.
The one real conflict marker, on the MParticle() class-extension additions,
was two property declarations that both belonged.

testActiveKitsRegistryThreadSafety merged without incident - this branch's
independent registry work (MPKitContainer.swift's registryLock/registry are
both static, unlike main's per-instance kitsSemaphore that #916 fixes) never
touched the same lines as main's #870 change to that test, which only reworked
how the test itself tracks failures across its four dispatch_group blocks.

Verified: no duplicate method names introduced, build-for-testing succeeds,
and MParticleTests + MPKitContainerTests run 147/147 clean, including all five
workspace-switch tests (0.1-0.4s each, versus 20-30s before) and the
thread-safety stress test.
nickolas-dimitrakas added a commit that referenced this pull request Sep 2, 2026
…st fixes (#915)

* feat: accept native GoDaddy R1 certificate chains (#879)

feat(network): accept native GoDaddy R1 certificate chains

Add GoDaddy TLS Root CA - R1 to the default certificate set while retaining all existing trust anchors.

Verify the embedded certificate against GoDaddy's published SHA-256 fingerprint to prevent accidental pin changes.

#agentic

* chore: Release v9.4.1 (#881)

chore: (release) 9.4.1

Updates version to 9.4.1 across the mParticle ecosystem.

* docs: trim AGENTS.md to the non-derivable core

* docs: correct the disabled-workflow and fork-PR notes

* docs: state the Xcode-pin split without pinning versions in prose

* docs: correct the disabled-workflow gotcha in AGENTS.md

release-ecosystem-from-main.yml is registered in Actions as
disabled_manually but its file is not on main at all - only on unmerged
branches - so describing it as a workflow file in the tree that never
runs was wrong. The real trap is that the workflow list and the tree
disagree in both directions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: record how to place a red or cancelled check

Observed on this branch: the build-kits Pod Lint jobs fail as a batch on
a CocoaPods CDN error, and a cancelled job is a timeout-minutes expiry or
a superseded push under pull-request.yml's cancel-in-progress concurrency
group - not a test result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: add CODEOWNERS to mirrored kit subtrees (#885)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* chore: bump actions/setup-java from 5 to 6 (#886)

Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5 to 6.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](actions/setup-java@v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* ci: stop simulator cloning and bound network/test hangs (#867)

* ci: stop simulator cloning and bound network/test hangs

Two unrelated hangs were holding up CI. Both were unbounded, so they
consumed a job's full budget instead of failing.

native-tests: the iOS x mParticle-Apple-SDK-Swift cell repeatedly hit its
15-minute timeout. A single test, MPDeviceTests.testDictionaryDescription,
took 156-459s across recent runs while every other test finished in under
1.6s; job duration tracked that test exactly. The same test takes 0.069s on
tvOS in the same run, so the cause is simulator state, not the assertions.
MPDevice reads UIDevice.current for name and identifierForVendor, which are
XPC calls into simulator daemons, and mParticle-Apple-SDK-Swift.xcscheme is
the only scheme in the repo with parallelizable="YES" - so xcodebuild clones
and boots extra iOS simulators on a runner that just erased one, starving the
daemons that test then blocks on.

Disable parallel testing for the run. The suite is ~2s of tests; locally
this is 2x faster end to end (33s vs 74s) because it no longer boots 8
simulator clones. Also wait for a real boot via simctl bootstatus - the
action's wait_for_boot only waits for state=Booted, not for first-boot work
to finish - and add a per-test timeout so a hung test fails in 2 minutes
with an explicit diagnostic instead of silently eating the job timeout. The
slowest genuine test on CI is ~30s, leaving 4x headroom.

build-kits: Build rokt-sdk-plus-ios stalled 30 minutes inside a git fetch of
stripe-ios (2.7 GB, reached transitively via rokt-payment-extension-ios),
leaving orphaned git and git-remote-http processes. The job has no
timeout-minutes, so only an unrelated concurrency cancel stopped it;
otherwise it would have held a macOS runner for the 6-hour default.

Make git abort a transfer stalled below 1 KB/s for 3 minutes, retry the
resolve three times (matching the existing pod-lint-kits pattern), and add
job timeouts so no hang here can run unbounded again. Healthy resolves take
2-5 minutes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* ci: bound each SPM resolve attempt with a kill-and-retry watchdog

The first version of this fix relied on git's own low-speed abort to break
a stalled fetch. It does not fire: on run 33010942238 the stripe-ios fetch
sat for 24.5 minutes with http.lowSpeedLimit/lowSpeedTime set, produced no
error, and never returned - so the retry loop never got a turn and the step
burned its whole 25-minute budget before failing.

Bound each attempt explicitly instead. A watchdog kills the resolve and its
surviving git children once an attempt passes ATTEMPT_TIMEOUT_SECONDS (480s,
against a healthy 2-5 minute resolve), so a stalled attempt is retried rather
than consuming the step. Drop to 2 attempts and a 20-minute step timeout,
since an attempt can no longer run unbounded.

Keep the git low-speed config: it is not sufficient on its own but still
aborts genuinely slow transfers where it does apply.

Verified against the extracted step body with a stubbed resolver: a healthy
resolve exits 0 in 1s with no kills; a permanent hang is killed twice and
fails bounded; a hang followed by a healthy retry is killed once and exits 0.
Defaulted the timeout in-place so an unset variable cannot make the watchdog
fire immediately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* ci: run the resolve watchdog cleanup in the parent so it actually executes

Bugbot caught that the watchdog's cleanup was dead code. The watchdog TERMed
the resolve subshell and then slept 5s before escalating, but TERMing the
resolve unblocks the parent's wait immediately, and the parent then kills the
watchdog mid-sleep - so the kill -KILL and the pkill of leftover xcodebuild
and git-remote-http children never ran. The retry could therefore start while
the previous stalled fetch was still alive and race it over SwiftPM's cache
locks, which is the exact failure the watchdog was added to prevent.

The watchdog now only records that it fired (via a marker file) and sends
TERM. Escalation and child cleanup run in the parent after wait returns, where
they cannot be pre-empted, followed by a short grace period before the retry.

Verified by extracting the step body, stubbing the resolver and replacing the
pkill calls with probes: on the old code the cleanup probes fired 0 times out
of 4 expected; they now fire 4/4 on a permanent hang and 2/2 when one attempt
hangs and the retry succeeds. Cleanup still does not run on a non-timeout
failure (0 probes), so a fast failure does not trigger a spurious pkill, and
the healthy path is unchanged at exit 0 with no kills.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* test: make workspace-switch and kit-registry concurrency tests deterministic (#870)

testSwitchWorkspaceKitsWithStop waited on two nested 10-second dispatch_after
blocks, so it always burned at least 20 seconds of wall clock and depended on
fixed timing rather than on the SDK actually being ready. Wait on the SDK's own
readiness signals instead - MParticle.initialized after startWithOptions:, and
the replacement shared instance after switchWorkspaceWithOptions: - by spinning
the main run loop until the condition holds. The assertions are unchanged, so
the test still checks the same behaviour; it now takes 0.11s instead of 20.2s.

testActiveKitsRegistryThreadSafety tracked failure in a plain __block BOOL
written from four concurrent blocks, which is itself an unsynchronized data
race, and called XCTFail from those background queues. Guard the shared state
with an NSLock, record only the first exception, and report it from the
dispatch_group_notify block on the main queue.

This commit does not change the flakiness of testActiveKitsRegistryThreadSafety
itself: that crash is a real race in MPKitContainer (flushSerializedKits
enumerates and mutates kitsRegistry without kitsSemaphore, reached via the
early return in configureKits: before the lock is taken). Reproduced at 2
crashes in 10 runs of that test alone. Fixing it changes core SDK locking, so
it is deliberately left for a separate, explicitly-reviewed change.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* chore: bump minimatch from 3.1.2 to 3.1.5 in /RNExample (#640)

Bumps [minimatch](https://github.com/isaacs/minimatch) from 3.1.2 to 3.1.5.
- [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
- [Commits](isaacs/minimatch@v3.1.2...v3.1.5)

---
updated-dependencies:
- dependency-name: minimatch
  dependency-version: 3.1.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: bump lodash from 4.17.21 to 4.18.1 in /RNExample (#697)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.18.1.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.21...4.18.1)

---
updated-dependencies:
- dependency-name: lodash
  dependency-version: 4.18.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: bump @babel/plugin-transform-modules-systemjs from 7.25.9 to 7.29.4 in /RNExample (#767)

chore: bump @babel/plugin-transform-modules-systemjs in /RNExample

Bumps [@babel/plugin-transform-modules-systemjs](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-transform-modules-systemjs) from 7.25.9 to 7.29.4.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.29.4/packages/babel-plugin-transform-modules-systemjs)

---
updated-dependencies:
- dependency-name: "@babel/plugin-transform-modules-systemjs"
  dependency-version: 7.29.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: bump shell-quote from 1.8.1 to 1.10.0 in /RNExample (#799)

Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.8.1 to 1.10.0.
- [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md)
- [Commits](ljharb/shell-quote@v1.8.1...v1.10.0)

---
updated-dependencies:
- dependency-name: shell-quote
  dependency-version: 1.10.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: stop the partial mock before releasing the object it wraps

MPRoktTests.m diverged too far from main's version for the preceding merge to
carry this over as part of it - main's tearDown reorders the same two lines,
but the files no longer share enough context for a line-based merge to find
that hunk. Applying it directly here instead: releasing self.rokt while
OCMPartialMock(self.rokt) is still installed is the wrong order regardless of
which branch's file structure it's in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Thomson Thomas <125323226+thomson-t@users.noreply.github.com>
Co-authored-by: mParticle Bot User <developers@mparticle.com>
Co-authored-by: Matt Bodle <matt.bodle@rokt.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: James Newman <james.newman@rokt.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bodle <22z33p@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants