Flaky e2e: gateway-fast-fail times out on a lease of the busy-worker tests

[V3RON]

e2e/gateway-fast-fail.test.ts, 'a gateway fails a request no worker can serve at once' > 'fails a request waiting for a busy worker with NO_CAPACITY when that worker ... disconnects' (also seen: '... drained' and 'fails a class or range request nothing in the fleet can serve at once'). Error: simlock lease --platform ios --device iPhone 17 --os 26.0 --agent-id agent-holder --detach timed out after 20000ms (e2e/helpers/cli.ts:49). Reproduced on origin/main b45acd7 (1 of 3 runs failed) and on task/369 (CI run 37431428774). Not caused by #369.

Written by an agent.

[V3RON]

Reproduction

This is a defect. It reproduces on origin/main 4ad4791: the lease timed out in 4 of 52 runs of the file, which also fails in other ways under load. The call that hangs is always the first lease that should be granted after a worker joins, in whichever test runs it. #396 is the same failure.

The test on bug/395-repro holds the worker's socket claim, so the race always goes the bad way. It fails in 60 ms:

pnpm exec vitest run --project unit src/daemon/gateway-fleet.e2e.test.ts
× reports a worker running and grants a no-wait lease on it, when the worker's socket claim is slow
AssertionError: expected [ 'starting' ] to deeply equal [ 'running' ]
 ❯ src/daemon/gateway-fleet.e2e.test.ts:365

With that line removed, the lease fails with NO_CAPACITY. Evidence from a failing e2e run, read 20 s into the hang:

gateway worker.list: connection "connected", catalogReadAt set, health "starting"
worker status.get:   health "running"
worker daemon.log:   "Uplink to gateway established" logged before "Claimed daemon socket"
gateway events:      lease.requested, lease.queued, and no request.dispatched

Root cause

src/daemon/main.ts:515 starts the gateway uplink before daemon.start() (main.ts:559) runs, so it starts before #readyPromise is set (server.ts:464). Until then, #awaitReady (server.ts:596) parks nothing. The gateway's events.subscribe and first refresh run during convergence, and its view stores health starting. Nothing refreshes that view afterwards, because daemon.started does not trigger a refresh (changesCapacityOrLeases, src/gateway/worker-link.ts). The healthy routing stage drops the worker, and the request waits for the 30 s refresh tick. The CLI's 20 s deadline fires first. The comment at main.ts:511 says the uplink waits for the claim. It does not.

Simplest fix

Start the uplink from onSocketClaimed, which start() calls right after setting #readyPromise. Wire that callback whether or not HTTP is on, and correct the comment. I tried this locally. The new test passes, and so do main.test.ts, gateway-uplink.test.ts and the three gateway e2e files.

Alternatives rejected

• Refresh the view on daemon.started: this fixes health, but the gateway would still read the catalog and config during convergence.
• Make #awaitReady park before start(): a larger change, and it alters every test that builds a DaemonServer without starting it.
• Make the e2e wait for health running: this hides the defect from users.

Risk

• The uplink must still start with HTTP off. Today the callback is wired only under config.http.enabled.
• A failed socket claim must leave the uplink undialled.
• The gateway's first status.get still answers starting because it never parks. Only the refresh after it must see running.

Side findings

• Flaky e2e: gateway-fast-fail's 'token create --role worker' on a fresh gateway prints no JSON #404

Written by an agent.