huntsman-e2e-tests: test_nn flakily fails with "resource group already exists" from a reused Compose DB volume

[20001020ycx]

Bug

huntsman-e2e-tests intermittently fails during test_nn at the very first job, with:

Error: neural-network job 0 in batch 0 with seed 0 failed

Caused by:
    resource group already exists
test test_nn ... FAILED

Observed on both linux/amd64 and linux/arm64 self-hosted runners in the same CI run (e.g. PR #498: amd64 job, arm64 job). The change under test in that PR was unrelated to resource-group creation, so the failure is spurious.

Why this is a real (latent) bug, not a one-off

The error is not a concurrency race inside the test:

• SpiderTestDriver is a process-wide singleton (OnceCell, tests/huntsman/e2e/src/test_driver.rs).
• SpiderTestDriver::resolve_resource_group holds a Mutex over its resource_groups cache for the entire create-and-insert, so SpiderClient::add_resource_group("e2e-nn") is invoked exactly once per process, even though test_nn submits 24 jobs (NUM_BATCHES = 3 × NUM_JOBS_PER_BATCH = 8) all under the same external id e2e-nn.

So add_resource_group receiving ResourceGroupAlreadyExists on its single call means e2e-nn was already present in the database before the test started — i.e. leaked state from a previous run.

Root cause

The test:huntsman-e2e-tests task (taskfiles/test.yaml) tears down the Compose stack and its volumes only via a deferred step:

• defer: :docker:compose:local:clean → docker compose --project-name spider-local ... down --volumes --remove-orphans (taskfiles/docker.yaml).

This deferred cleanup runs after the test. There is no volume wipe before :docker:compose:local:up. Consequently:

1. A prior run on the same self-hosted runner is hard-cancelled/killed (or the runner dies) before its deferred compose:local:clean executes.
2. The named volume spider-local_spider-database-data — containing the e2e-nn resource group row from that run — survives.
3. The next run's docker:compose:local:up reuses the existing named volume (Compose does not recreate named volumes that already exist).
4. test_nn's first add_resource_group("e2e-nn") hits the pre-existing row and fails, because registration is not idempotent: storage returns ResourceGroupAlreadyExists (components/spider-storage/src/grpc.rs, components/spider-storage/src/db/error.rs), surfaced to the client as resource group already exists (components/spider-client/src/error.rs).

This is environmental / infrastructure flakiness in the e2e harness, independent of the code under test. The e2e driver, nn.rs, and the e2e/compose taskfiles are identical across recent history, so any PR's e2e run can hit this whenever a preceding run leaves the volume behind.

Impact

• Spurious huntsman-e2e-tests failures block PR merges and require manual re-runs.
• Erodes trust in the e2e signal (a red e2e no longer reliably means "the PR broke something").

Spider version

main @ 737e85a423e552591fcf0acdf91c2b9400a1aa90 (the affected harness code predates this and is unchanged across recent history).

Environment

• GitHub Actions self-hosted runners: linux/amd64 and linux/arm64, ubuntu-noble, Docker.
• Docker Compose project spider-local; database image mariadb:10.11.16; named volume spider-local_spider-database-data.

Reproduction steps

1. Run task test:huntsman-e2e-tests locally (or trigger the CI job).
2. Hard-cancel/kill it after :docker:compose:local:up has started the stack but before the deferred :docker:compose:local:clean runs (this simulates a cancelled CI job). Confirm the volume survives: docker volume ls | grep spider-local_spider-database-data.
3. Run task test:huntsman-e2e-tests again. test_nn fails at job 0 with resource group already exists.

(The DB volume persisting between runs is the deterministic trigger; on shared self-hosted runners this happens naturally when a run is cancelled.)

Proposed fixes

Any one of these resolves the flake; combining a harness-hygiene fix with an idempotency fix gives defense-in-depth:

1. Wipe the Compose volume before up. Add an eager :docker:compose:local:clean (down --volumes --remove-orphans) step before :docker:compose:local:up in test:huntsman-e2e-tests, in addition to the existing deferred cleanup. Cheap and targeted.
2. Make the e2e driver tolerant of pre-existing groups. In resolve_resource_group, treat ResourceGroupAlreadyExists as success by resolving the existing id (get-or-create), so any leaked state is harmless. (Needs a client API to look up an existing external resource-group id.)
3. Use a unique external resource-group id per run (e.g. suffix e2e-nn with a run id/UUID) so a stale volume can never collide.
4. Runner hygiene: prune Docker volumes between jobs on the self-hosted runners.

Recommend (1) as the immediate fix, and (2) or (3) for robustness against any residual leaked state.

[20001020ycx]

Second occurrence — same root cause, different symptom (leaked stack → spider-storage unhealthy)

A rerun of the same CI run surfaced a different downstream symptom of the same leaked-state bug, this time on the linux/arm64 self-hosted runner (failing job). The linux/amd64 job passed on rerun, so the code was fine — the difference was purely the runner's leftover state.

The arm64 job never reached test_nn; it aborted during docker compose … up --wait:

Container spider-local-spider-storage-1 Started
Container spider-local-spider-storage-1 Waiting
Container spider-local-spider-storage-1 Error dependency spider-storage failed to start
dependency failed to start: container spider-local-spider-storage-1 is unhealthy
task: Failed to run task "docker:compose:local:up": exit status 1

The smoking gun is the start of up — a previous spider-local deployment was still running on the runner, and its database volume was reused:

Container spider-local-spider-database-1 Running        # reused from a prior job
Container spider-local-spider-scheduler-1 Stopping
Container spider-local-spider-worker-1  Stopping
...
Container spider-local-spider-database-1 Healthy        # reused DB, healthy in ~0.5s
Container spider-local-spider-storage-1 Started
Container spider-local-spider-storage-1 ... is unhealthy # failed ~0.8s after start (a crash, not the 300s wait-timeout)

A clean run would show Creating, not Stopping/Recreate against a pre-existing DB. So spider-storage started against the stale, reused database and went unhealthy almost immediately, aborting the up.

Why this reinforces the proposed fix

Both symptoms observed so far —

1. add_resource_group("e2e-nn") → resource group already exists, and
2. spider-storage unhealthy on startup against a reused DB —

stem from the same root cause: the test:huntsman-e2e-tests task only cleans the Compose stack/volumes in a deferred step, so a hard-cancelled prior run leaves both the containers and the spider-local_spider-database-data volume behind for the next run to reuse.

This makes proposed fix #1 (eager :docker:compose:local:clean — down --volumes --remove-orphans — before :docker:compose:local:up) the highest-value change: it addresses both symptoms at once by guaranteeing a clean slate regardless of how a prior run terminated.

[20001020ycx]

Flake confirmed: identical commit went red → green on reruns alone (no code changes)

To rule out the code under test, I reran the same CI run on PR #498 without pushing any changes. The head commit stayed e8fd07c the entire time, yet the huntsman-e2e-tests result flipped across attempts:

Attempt	Trigger	linux/amd64	linux/arm64
1 (original)	push	❌ resource group already exists	❌ resource group already exists
2	rerun failed jobs	✅ pass	❌ spider-storage unhealthy (reused stack/volume)
3	rerun failed jobs	— (already green)	✅ pass

Run: https://github.com/y-scope/spider/actions/runs/36057867398 · commit e8fd07c (unchanged across all three attempts).

Conclusion: the failures are nondeterministic and environmental — the identical commit reaches all-green purely by re-running, with no code change. This confirms the two symptoms above are the same leaked-state flake, not a regression in the PR. It also means the flake is currently masking real signal: contributors have to blindly hit "re-run" until the self-hosted runners happen to start from a clean slate.

This strengthens the case for the eager pre-up cleanup (proposed fix #1): a guaranteed clean slate before docker:compose:local:up would make a single run deterministic instead of requiring luck across reruns.

[LinZhihao-723]

Ok, this seems to be an issue beyond the described root cause: if spider-local_spider-database-data is used, does it mean that if the runner is shared by two e2e test GitHub workflows, there will be a conflict?
Let's take a step back:

• It looks like this is more of a resource leak problem: it is triggered by a forced termination of the runner container. But this should be solved by the GitHub runner manager, not on our side.
• The conflict I described sounds like a real issue that has existed for a long while, but we never actually triggered it. Shall we consider adding some UUID-like prefix/suffix to uniquely identify the resource used by each run?

[20001020ycx]

Update: the flake has more causes than a leaked volume

Following up on @LinZhihao-723's comment. I checked the self-hosted runner setup (y-scope/ci) and the logs of all 55 huntsman-e2e-tests jobs from 2026-09-24 12:00 to 2026-09-25 14:00 UTC. 10 of them failed. The concurrency concern is real and has already caused failures, and there is a third cause that accounts for half of them.

How the e2e job runs on our self-hosted runners

• Hosts and runners. Our CI machines ("hosts", e.g. baker1, msft-devkit-2) each run several GitHub runners side by side, each in its own container. A runner executes one CI job at a time, and GitHub hands any queued job to any idle runner. So e2e jobs from different PRs, or reruns, can land on the same host at the same time.
• One Docker engine per host. All runners on a host share the host's Docker engine. When our e2e job runs docker compose up, the Spider stack (containers, database volume, network) is created on the host, not inside the runner. It stays there after the job ends, unless the job's own cleanup step removes it.
• Runner containers are reused. After a job, the same runner container picks up the next job. Anything attached to it carries over.
• Our stack uses fixed names. Every e2e run uses the Compose project name spider-local, so the containers, volume, and network always have the same names, and the storage service always publishes host port 50051.

Two things follow: if a job is killed before its cleanup runs, the next job on that host inherits the leftovers; and two e2e jobs running on one host at the same time share a single stack.

What caused the 10 failures

Cause	What happens	Failures
A. Leftovers from a cancelled job	A job is cancelled (e.g. superseded by a new push) before cleanup. The next e2e job on that host reuses the old database.	2: resource group already exists (job), database unhealthy on a half-initialized volume (job, likely)
B. Two e2e jobs on one host at the same time	The second job's compose up takes over the first job's running stack. When the second job fails, its cleanup deletes the stack the first job is still testing against.	2, from one collision on msft-devkit-2: storage unhealthy (job), and the other job lost its storage mid-test with dns error (job)
C. Runner left attached to the old network	So the test can reach the storage service, our job attaches the runner container to the stack's network and detaches it at the end. If the job is cancelled in between, the runner stays attached. Since the runner container is reused, every later e2e job on it fails when attaching again. Cleanup can't fix this, because Docker won't delete a network something is still attached to.	5: endpoint with name ci-github-runner-1 already exists in network spider-local_default, on two runners: msft-dev-kit-1 (1, 2, 3) and msft-devkit-2 (1, 2)
Unrelated	Image build failed before Compose was involved.	1

Leftovers are common: 6 of the 7 e2e jobs that were cancelled after starting the stack left it behind.

Other possible failures (not seen yet)

• A job reuses leftovers and passes anyway, so the leak goes unnoticed (example).
• Leftover database from an older commit is reused by newer code (stale schema).
• A job is cancelled in the middle of cleanup (GitHub allows only ~10 s), leaving part of the stack behind.
• Leftover stacks keep running and use CPU (workers poll every 1 ms), slowing every job on that host. One leftover stack was still running 5.5 h later.
• With two jobs sharing one stack, one job can silently test the other PR's binaries.
• Each job starts one worker per CPU core, while several runners share a host, which can overload it.
• Leftover volumes, images, and networks pile up and eventually exhaust disk space or Docker's network address pool.

[20001020ycx]

Proposed fix

Why this only happens on baker. GitHub-hosted runners start every job on a fresh VM with its own Docker engine and delete the VM afterwards, so jobs can't see each other's containers or anything a killed job left behind. Our self-hosted runners (baker) are deliberately set up to share resources between jobs instead: jobs reuse the caches of previous jobs (installed actions and tools, Docker images), and all runners on a machine share one Docker engine. Giving every job an isolated, throwaway Docker engine on baker is possible, but it would give up that sharing for all y-scope repos. So we should make the e2e job safe on a shared engine instead.

Fix:

1. This repo: give each e2e run its own stack. Each run uses a random Compose project name (so its containers, volume, and network have unique names) and a free host port instead of 50051. Cleanup at the end of the job removes only that run's stack. Local task docker:compose:local:* keeps using spider-local.
   • Two concurrent jobs no longer share a stack (cause B).
   • Leftovers from a cancelled job can no longer affect later jobs, because no later job uses the same names (causes A and C).
2. Runner infra (y-scope/ci): periodically remove leftover e2e stacks on the baker hosts, including ones that are still running. After (1) leftovers no longer break tests, but they still use CPU, disk, and network addresses.
3. Now: manually clean up the leftover spider-local stacks and networks on the affected hosts (at least msft-dev-kit-1, msft-devkit-2, baker1, baker4, baker17). Until (1) lands, the two runners hit by cause C fail every e2e job they pick up.

The originally proposed fix, wiping the stack before compose up, isn't enough: it only helps with A, would delete another job's running stack in case B, and can't remove the network in case C. #504 will be reworked to implement (1) instead.

[LinZhihao-723]

Thanks @20001020ycx for the detailed investigation. The proposal looks good to me. I will review #504 once it's ready.