Repository navigation
Conversation
jkmassel
added
Testing
Collaborator
Generated by 🚫 Danger |
Contributor
|
| App Name | Jetpack | |
| Configuration | Release-Alpha | |
| Build Number | 34854 | |
| Version | PR #26119 | |
| Bundle ID | com.jetpack.alpha | |
| Commit | b12ef60 | |
| Installation URL | 3fgg14jcsh5h8 |
Contributor
|
| App Name | WordPress | |
| Configuration | Release-Alpha | |
| Build Number | 34854 | |
| Version | PR #26119 | |
| Bundle ID | org.wordpress.alpha | |
| Commit | b12ef60 | |
| Installation URL | 77kfsghcf6f0g |
Contributor
🤖 Build Failure AnalysisThis build has failures. Claude has analyzed them - check the build annotations for details. |
2 of 4 tasks
jkmassel
force-pushed
the
jkmassel/ui-tests-targets-structure
branch
from
October 6, 2026 17:39
8c4b66c to
06ef515
Compare
jkmassel
changed the base branch from
jkmassel/fix-urlsession-lazy-race
to
jkmassel/fix-capabilities-callback-priority
October 6, 2026 17:39
Add `Scripts/sim-signin.sh`, a one-command wrapper around `xcrun simctl launch` that signs a Simulator into WordPress.com with a bearer token, and make the existing `-ui-test-wpcom-token` launch argument finish sign-in automatically instead of requiring a tap on "Continue with WordPress.com". `autoSignInUITestWPComAccountIfNeeded()` mirrors the existing `autoSignInUITestSite()` hook: it runs from `showSignInUI()`, no-ops unless the token argument is present and no account is signed in, then runs the real sign-in flow and swaps the window root with `windowManager.showUI()`.
Add a named `-t, --wpcom-token <token>` option alongside the positional form, and reject a missing option value instead of shifting past the end.
Introduce `WordPressDotComAuthenticator.launchArgumentToken`, which reads the new `-wpcom-token` launch argument and falls back to the legacy `-ui-test-wpcom-token`. Route both read sites — `attemptSignIn` and `autoSignInUITestWPComAccountIfNeeded()` — through it, and switch the helper script to pass `-wpcom-token`.
The launch argument is `-wpcom-token`, not UI-test-scoped, so `autoSignInUITestWPComAccountIfNeeded()` was misleading. Rename it to `autoSignInWPComAccountFromLaunchArgumentIfNeeded()`.
So the token can be set once instead of pasted every launch, sim-signin.sh falls back to the WPCOM_TOKEN environment variable, then a ~/.wpcom-token file, when neither --wpcom-token nor a positional token is given. Precedence: flag > env var > file.
Wraps `Scripts/sim-signin.sh` behind `make sim-login`, forwarding optional `DEVICE`, `APP`, and `RESET=1` variables plus an `ARGS` passthrough. The token still resolves from `--wpcom-token`, `WPCOM_TOKEN`, or `~/.wpcom-token`.
When --device is omitted, sim-signin.sh now resolves the target from the booted simulators: it uses the only one if a single device is booted, and otherwise lists them and prompts for a choice. Errors clearly when none are booted or no selection is made. An explicit --device still bypasses this.
If no token is found via --wpcom-token, WPCOM_TOKEN, or ~/.wpcom-token, sim-signin.sh now prompts for one instead of erroring out. Input is read hidden (it's a secret), whitespace-trimmed, and confirmed by length; cancelling with an empty entry still exits with the usage error.
After the interactive prompt accepts a token, ask whether to persist it to ~/.wpcom-token so later runs pick it up automatically. Defaults to no, and writes owner-only (umask 077 + chmod 600) since it's a secret.
Lead the simulator sign-in doc with `make sim-login` as the quickest path, and name the command inline in AGENTS.md so agents can sign a Simulator in without opening the doc first.
Signing out returns to the login screen, where `showSignInUI()` runs the auto sign-in again. With the `-wpcom-token` argument still present and the account now gone, both guards passed and it signed straight back in — a logout could never reach a logged-out state. Gate the auto sign-in on a per-process flag so it attempts at most once.
A token passed on the command line is saved to shell history, so drop the `--wpcom-token` flag and the positional form. The token now comes only from `WPCOM_TOKEN`, `~/.wpcom-token`, or a hidden prompt. While here, two more fixes from review: - `--reset` now uninstalls and reinstalls the app rather than running the in-app `-ui-test-reset-everything` wipe. Reinstalling clears the whole data container (caches and cookies too) and is synchronous, so it drops the `sleep 2` that could race the wipe. - The booted-simulator scan no longer aborts under `set -e` when a line has no UUID; `|| true` lets the skip logic run as intended.
The site menu rows, the dashboard cards, their headers and menu buttons, the Stats cards and date controls, domain rows, and the subscriber and user details. The chart's metric tabs also gain the selected trait, which they were missing for VoiceOver.
A URLProtocol, compiled into debug builds only, that answers every URLSession request from WireMock-style stub mappings and can log each request it sees. The app installs it when it's launched with -ui-test-http-fixtures.
The target replaces the UI tests removed in #25399. JetpackUITestCase launches the app signed in with the -wpcom-token launch argument, against either a real WordPress.com account or the fixtures.
The WireMock mappings the removed UI tests used, restored from before #25673 deleted them, with stats responses that follow today's date.
My Site, the dashboard cards, the site menu and its lists, the editor, and Stats. They check that screens open and controls change state, and skip rather than fail when the account's site has nothing to open.
The account, Me, Notifications, the Reader, Stats, liking a post, analytics events and request counts. The fixture account never changes, so these assert on what the screens show and on what the app sends.
A UI Tests step that depends on the Jetpack build for testing and runs the suites that declare the fixtures backend, which need no token. test_without_building infers the scheme from the xctestrun name again and takes an only_testing list.
The module that answers the app's requests from fixtures, and the code in the app that installs it, were compiled into every debug build. Now every file in HTTPFixtures and the hook in UITestConfigurator are wrapped in `#if UI_TEST_HTTP_FIXTURES`, so a build that doesn't set that condition has none of it: the module compiles to an object with no code in it. No configuration or scheme sets the condition. fastlane passes it to xcodebuild as a build setting, which is the only way to reach a package module as well as the app's targets: the `test` lane does for the Jetpack scheme, and `build_jetpack_for_testing` does when given `http_fixtures:true`, which is how CI now builds the app the UI tests run against. A fixture-backed test fails as soon as the app has launched, with a message naming the condition, when the app wasn't built with it. Without this the app would send the test's requests to WordPress.com. HTTPFixturesTests is compiled out with the module. CI's package tests set the condition; without it the target reports one skipped test that says how to run the rest. It leaves WordPressUnitTests.xctestplan, whose build doesn't set the condition. A SwiftLint rule fails a file in the module that doesn't start with the condition.
The 37 fixture-backed tests took 954 seconds on one Simulator; they now take 398. - Poll for elements instead of using XCTest's waits. `waitForExistence` and `wait(for:toEqual:timeout:)` check once a second and not before the first second is up, so each took a second even when the element was already there, and a test makes dozens of them. - Launch the app with `-ui-test-disable-animations`, which turns off UIKit's animations. XCTest waits for them before every tap and query. The argument is the one #25673 removed when nothing passed it. - Send analytics events once a second when the app runs against the fixtures, instead of every 15 seconds. The Stats suite slept 17 seconds per test to be sure of having them. - Stop the app asking to send push notifications, with a launch argument, instead of dismissing the question. The sheet appears a moment after the list does, so a faster test could get to the list first. The suites that run against a real account keep the second before a wait's first check. Their content arrives over the network after its screen does, and without that second five of the 79 tests failed on lists that were still loading. Three of those dependencies are fixed here, where the cause was clear: - Account Settings is empty until its request returns, so the assertions that something is shown now wait for it. - The Social list has its "Connect a New Account" button before it has the connections, so "the first row" opened that instead. - The Insights tab's first card loads after the tab appears, and scrolling before then scrolled past where it was about to be.
The suites move into a folder per area of the app (Me, MySite, Notifications, Reader and Stats), and the UI Tests step becomes a matrix of those folders, so the areas run at the same time on separate agents. Every job fails if it finds a fixture-backed suite in a folder the matrix doesn't list, so a new area can't be left out of CI unnoticed. `test_without_building` also takes `concurrent_workers`, to spread suites over clones of the Simulator. CI doesn't use it: clones start cold, and with four starting at once the first test on each timed out.
On CI, with `-ui-test-disable-animations`, three of the five UI Tests jobs
failed their first test: the app had signed in and fetched the account's
sites, but stayed on the sign-in screen behind its progress indicator.
In one of them the next test then found My Site without its create
button. The same jobs passed their other tests, and the run before the
argument was added passed all 37.
Remove the argument and the code behind it. XCTest waiting for
animations costs the fixture-backed tests about two minutes in nine on
one Simulator, which is less than a job that has to be retried.
Also report one commit status for the UI Tests step. GitHub receives the
context as it's written in the pipeline, so "UI Tests ({{matrix}})"
arrived with the braces in it.
With a job per area, five of ten jobs over two builds failed their first test and passed the rest: My Site hadn't appeared 60 seconds after the app launched. The app was still on the sign-in screen, behind its progress indicator, with the account and its sites already fetched. CI runs each job on a Simulator it has just erased and booted, and the first launch there is slow in a way no later one is: over 76 seconds to sign in, against under 10. One job a build could get by on that; five starting together can't. Wait up to three minutes for My Site after the first launch of a run, before the test starts its own 60-second wait. The previous commit blamed these failures on `-ui-test-disable-animations`. They happened again without it, so that was wrong, and the docs no longer say so.
XCTest waits for the app's animations to finish before every tap and query. Launching with `-ui-test-disable-animations`, which calls `UIView.setAnimationsEnabled(false)`, takes the 37 fixture-backed tests from 519 seconds to 402 on one Simulator. Each of them passed three times in a row with it, and a swipe scrolls a list just as far. This is the argument #25673 removed when nothing passed it. An earlier commit here added it back and the next took it out again, over first-test failures on CI that turned out to be the cold first launch, which has its own wait now.
A UI test job runs tests that an earlier job built, from an xctestrun file. Before it started them, scan resolved all 41 Swift packages and read the project's build settings: 150 seconds of a job that took 7 minutes, to find three things. Give it those instead: where the build is, the app's name, and the deployment target, read from config/Common.xcconfig. `disallow_xcodebuild_settings_lookup` makes scan fail if anything else asks for the build settings, so the cost can't come back unnoticed. scan also reads the build settings to choose a Simulator when it isn't given one, so this applies when the caller names a device. The UI test jobs do. The unit test job doesn't, and is unchanged: it spends the same 150 seconds, and would stop if it named its Simulator too.
Two changes to how the test jobs set up. The unit test job now names the Simulator it runs on: "iPhone 17 Pro (26.5)", which is the one fastlane had been choosing for it. With a device named, `test_without_building` doesn't resolve the Swift packages or read the build settings first, which was 147 seconds of that job. Both test jobs start booting their Simulator as their first step, so that it boots while the job downloads its build and installs its gems. A UI test job spent 53 seconds getting its Simulator ready once everything else was. The lane takes `reset_simulator:false` so that it doesn't erase the Simulator that's booting. A CI job runs in a VM made for it, so there's nothing on the Simulator to erase.
jkmassel
force-pushed
the
jkmassel/fix-capabilities-callback-priority
branch
from
October 7, 2026 02:26
3b2d9a8 to
f02c27a
Compare
jkmassel
force-pushed
the
jkmassel/ui-tests-targets-structure
branch
from
October 7, 2026 02:26
06ef515 to
b12ef60
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
JetpackUITeststarget with 116 UI tests for the Jetpack app, replacing the tests removed in #25399. 37 of them run against HTTP fixtures and now run in CI; the other 79 sign in to a real WordPress.com account and are for running locally.Stacked on #26116. The branch also carries the 13 commits of #25801, which adds the
-wpcom-tokenlaunch argument these tests sign in with and hasn't merged yet; they drop out of this diff when it does.What's in it
JetpackUITeststarget and test plan to the Jetpack scheme, inTests/JetpackUITests. The scheme's test action still pointed at the target Remove UI tests #25399 deleted. Tests are written against screen objects, andJetpackUITestCaselaunches the app already signed in.HTTPFixtures, a module with aURLProtocolthat answers everyURLSessionrequest in the app from WireMock-style stub mappings and can log each request it sees. The app installs it when it's launched with-ui-test-http-fixtures..isSelectedtrait, which they didn't expose.test_without_buildinginfers the scheme from the xctestrun name again, as it did before Clean up UI tests infrastructure #25673, and takes anonly_testinglist.docs/ui-tests.md, which covers running the tests, writing one, writing a fixture, and what the two kinds of suite do and don't check.Why two backends
Some screens change the account just by being shown. Opening the Notifications tab marks every notification as seen, opening one marks it as read, and opening a post in the Reader counts as a view of it. Those can't be tested against a real account without altering it, so Notifications, the Reader and Me run against the fixtures.
The fixture account never changes, which also lets those suites assert on content and on what the app sends.
ReaderActionTestslikes a post, then checks that the app sentPOST /rest/v1.1/sites/70135762/posts/125073/likes/newexactly once and that the post then appears in the Likes stream.The real-account suites check the opposite half: that the app works against live responses. They assert that screens open and controls change state, never on content, so they aren't tied to one account. When the account's site has nothing to open (no recent activity, no referrers this week), a test is skipped with a message naming what was missing rather than failed.
What a reviewer will want to know
The interceptor can't reroute requests in a release build.
FixtureURLProtocoland the call that installs it are inside#if DEBUG.The app-side changes are identifiers and one trait. Nothing else in the app's screens changes. The trait means the selected Stats metric tab now reports itself as selected to assistive technology; That was confirmed in the accessibility hierarchy, not with VoiceOver itself.
CI needs no token or secret. The script picks the suites that declare
override class var backend: Backend { .fixtures }, so a new fixture-backed suite is included without editing the pipeline, and a real-account suite never runs there.The fixture-backed suites depend on #26116. Without it the app intermittently crashes at launch under the fixtures, from the
URLSessionrace that PR describes.This PR's build is the first time the "UI Tests" step runs on an agent. Two things are unknown until it does: whether the Xcode 27.0 image has an "iPhone 18 Pro" simulator, and how long the step takes. Locally the 37 tests took about 21 minutes of lane time across four runs. The new commit status isn't a required check.
Not in this PR
Test plan
fastlane test_without_building name:JetpackUITests only_testing:…), on an iPhone 18 Pro simulator running iOS 27.0: 12, 9, 10 and 6 across four runs, with no app crash reports.HTTPFixturesTestspasses withswift test --filter HTTPFixturesTests: 52 tests in 5 suites, including one that loads every fixture file.ReaderActionTestsfails when it should: a copy that waited for a different endpoint failed and listed the requests the app had sent..buildkite/pipeline.ymlpasses Buildkite's schema validation.xcodebuild -workspace WordPress.xcworkspace -scheme Jetpack -testPlan JetpackUITests -destination 'platform=iOS Simulator,name=iPhone 18 Pro' -only-testing:JetpackUITests/ReaderActionTests testends withExecuted 1 test, with 0 failures.-only-testing:JetpackUITests/MySiteTests. It needs a WordPress.com token in~/.wpcom-tokenand signs the simulator in to that account. These suites only read; the ones that open the editor close it without saving. Without a token the suite reports 5 skipped tests.