Issue report

Title: Harness: build_base_lxd_image races cloud-init in the builder and times out lxc publish on slow runners

• Affects: main
• Part of: Reduce Flaky Tests #773
• Related: ci: probe, retry and label network installs in the Robot harness #876 (changed the setup timing), Harness apt-get update runs once with a fixed timeout; one mirror stall failed 16 suites in a single run #842 (apt steps of the same function)
• Jobs: apt race 2 of 1,851 builder-based jobs since 2026-09-24 (Deferred-Ceph lifecycle and placement multi node, Test MicroCeph RBD Remote Replication features), publish timeout 2 of 4,356 (both Deferred-Ceph lifecycle and placement multi node); 12 job names, deduplicated, 2026-08-01 to 2026-10-01

What version of MicroCeph are you using ?

The apt failures ran on stack branches based on main at 745aad7 (2026-09-24 and 2026-09-29). The publish failures ran on a stack branch based on fd13d43 (2026-09-23) and on the core26-smb-feat branch at 0ff42a9 (2026-09-30). build_base_lxd_image is identical at the base commit of the apt jobs above and at 5df62e1, and the lxc publish line is the same in all of the above.

What are the steps to reproduce this issue ?

1. Run any multi-node Robot suite. build_base_lxd_image starts a builder container from the stock jammy image, waits 5 s, polls snap version, probes the network, runs apt-get update in it, installs the snap, then runs lxc publish with a 300 s timeout.
2. Intermittently, either the first apt-get update lands while the builder's first-boot cloud-init is still rewriting /etc/apt/sources.list, or the runner is so slow that the publish cannot finish in 300 s.

What happens (observed behaviour) ?

Two separate failures, both in suite setup, so every test in the suite fails.

Apt: Command failed (rc=100) with E: Type 'ubuntu' is not known on line 43 in source list /etc/apt/sources.list and E: The list of sources could not be read. The first apt-get update started 8.06 s and 10.15 s after lxc start. The harness only retries rc 124, so rc 100 fails at once. Apt reads the file with two read() calls; a rewrite between them yields the 2403-byte stock file plus the tail of the 2780-byte cloud-init rendering, whose line 43 is ubuntu jammy-security main restricted. The analysis rebuilt this byte for byte and reproduced the exact error with a concurrent rewriter in 11 of 400 local apt runs. The writer being cloud-init apt_configure is inferred; it was not observed in CI.

Publish: Command failed (rc=124) / Command timed out after 300s while Image pack was at 74% (5.14MB/s) and 33% (2.40MB/s). Passing jobs publish in 58 to 159 s (median 93 s, 54 jobs), and in both failing jobs lxc init took 80 to 90 s (passing median 16 s), so the host was about five times slower at that step and the publish was still progressing.

Counts before and after #876 (merged 2026-09-24T10:23Z), 2,505 jobs before and 1,851 after. Apt: 0 before, 2 after (one-sided Fisher p=0.18, not significant). The argument is timing: apt now starts a median 9.7 s after lxc start (n=28) against 6.5 s before (n=26), inside the cloud-init config window. Publish: 1 before (2026-09-23) and 1 after (2026-09-30), so no link to that change is shown. The RGW stack (#881 to #893) was ruled out: in the three stack jobs its tests and .github trees are identical to its base on main, and the fourth job ran on core26-smb-feat.

What were you expecting to happen ?

The builder's first apt call sees a settled sources.list, and a slow but progressing publish finishes.

Relevant logs, error output, etc.

• Apt, 2026-09-24: https://github.com/canonical/microceph/actions/runs/36048870586/job/107801877319 (log lines 863-876)
• Apt, 2026-09-29: https://github.com/canonical/microceph/actions/runs/36622075057/job/109592691239 (lines 532-552)
• Publish, 2026-09-23: https://github.com/canonical/microceph/actions/runs/35901666433/job/107322268696 (lines 928-930, 1092-1097)
• Publish, 2026-09-30: https://github.com/canonical/microceph/actions/runs/36720439303/job/109907650085 (lines 940, 1012-1017)
• Code:

  microceph/tests/robot/resources/microceph_harness.py

  Lines 1836 to 1890 in 5df62e1

  	def build_base_lxd_image(self, home):
  	"""Builds the ubuntu-22.04-microceph base LXD image with tools and MicroCeph pre-installed."""
  	logger.console("[setup] Building base LXD image with tools and MicroCeph...")
  	# Namespaced builder instance plus up-front best-effort cleanup (rc ignored).
  	builder = IMG_BUILDER_NAME
  	self.run_in_vm(f"lxc delete --force {builder}", 30)
  	self.run_in_vm(f"lxc image delete {MICROCEPH_IMAGE_ALIAS}", 30)
  	self.run_in_vm_and_check(f"lxc init local:{BASE_IMAGE_ALIAS} {builder}", 120)
  	self.run_in_vm_and_check(f"lxc config set {builder} security.privileged true", 10)
  	self.run_in_vm_and_check(f"lxc config set {builder} security.nesting true", 10)
  	# run_in_vm already wraps the command in `bash -eo pipefail -c`, so the
  	# original's redundant outer `bash -c "..."` wrapper is dropped and the
  	# printf|lxc pipeline is passed directly. RAW_LXC_DEVICE_ALLOW carries the
  	# literal backslash-n the remote printf needs.
  	self.run_in_vm_and_check(
  	f"printf '{RAW_LXC_DEVICE_ALLOW}' | lxc config set {IMG_BUILDER_NAME} raw.lxc -",
  	10,
  	)
  	self.run_in_vm_and_check(f"lxc config device add {builder} homedir disk source={home} path=/mnt", 10)
  	self.run_in_vm_and_check(f"lxc start {builder}", 60)
  	time.sleep(5)
  	# snap-version readiness loop: break as soon as `snap version` succeeds.
  	# raise_on_timeout=False mirrors the original break-and-fall-through: if snap never
  	# reports ready we proceed anyway and the apt/snap calls below surface any real problem.
  	self._poll_until(
  	lambda: self.exec_in_container(builder, "snap", "version", timeout=10).rc == 0,
  	attempts=20,
  	interval=3,
  	fail_msg="",
  	raise_on_timeout=False,
  	)
  	self.probe_instance_network(builder)
  	# Two calls rather than one chained string, so a stalled install is retried
  	# without repeating the update, and each gets its own attempt budget.
  	self.apt_update(builder)
  	self.apt_install(VM_APT_TOOLS, builder)
  	self.run_in_container_with_snap_retry(
  	builder, f"snap install --dangerous {MNT_SNAP_GLOB}", 600
  	)
  	connects = " && ".join(f"snap connect microceph:{iface}" for iface in SNAP_INTERFACES_MINIMAL)
  	self.run_in_container_and_check(builder, connects, 120)
  	self.exec_in_container(builder, "snap", "alias", "microceph.ceph", "ceph", timeout=30, check=True)
  	# The remote sed command must contain a SINGLE literal backslash before each '*'
  	# (matching the original .resource cell '\\*\\*', which Robot unescapes to '\*\*').
  	# In this Python string each '\\' is one literal backslash, so '\\*\\*' -> '\*\*'.
  	self.run_in_container_and_check(
  	builder,
  	f"sed -e 's|/sys/devices/\\*\\*/ r,|/sys/devices/** r,|' -i.bak {SNAP_APPARMOR_PROFILE}",
  	30,
  	)
  	self.run_in_vm_and_check(f"lxc stop {builder}", 60)
  	self.run_in_vm_and_check(f"lxc publish {builder} --alias {MICROCEPH_IMAGE_ALIAS}", 300)
  	self.run_in_vm_and_check(f"lxc delete {builder}", 10)
  	logger.console("[setup] Base image ubuntu-22.04-microceph ready.")

  (lxc start at 1855, apt_update at 1870, publish at 1887); the existing cloud-init wait for the outer VM is at

  microceph/tests/robot/resources/microceph_harness.py

  Lines 1562 to 1565 in 5df62e1

  	logger.console(f"[setup] Waiting for cloud-init in {vm_name}...")
  	res = self._exec(["lxc", "exec", "-n", vm_name, "--", "cloud-init", "status", "--wait"], 300)
  	if res.rc != 0:
  	raise AssertionError(f"cloud-init failed in {vm_name}: {res.stderr}")

Additional comments.

Proposed fix, one small PR. After lxc start and before the network probe, run lxc exec BUILDER -- cloud-init status --wait with a 300 s timeout, as launch_outer_test_vm does for the VM; accept rc 0 and 2, since the VM check rejects any non-zero rc. Optionally let _is_transient_apt_stall also retry once on rc 100 with The list of sources could not be read, with a pytest in tests/robot/resources/test_harness_helpers.py. For the publish, raise the timeout from 300 to about 900 s and use lxc publish BUILDER --alias ALIAS --compression none, since the image never leaves the VM and the gzip pack dominates the time.