Issue report

Title: WAL/DB cleanup: partx -d fails with "error deleting partition 1" while the kernel entry still exists

• Affects: main and squid (also seen on two other branches)
• Part of: Reduce Flaky Tests #773
• Supersedes: WAL/DB partition cleanup fails when partx -d runs after sfdisk --delete already dropped the kernel entry #835 (closed; reopening it with this evidence is the alternative to filing a new issue)
• Related: WAL/DB partition cleanup fails when partx -d runs after sfdisk --delete already dropped the kernel entry #835 (closed 2026-09-17, after the squid backport backport: treat already-removed partition entry as cleaned on WAL/DB cleanup (squid) #857 merged), Flaky: WAL/DB partition cleanup fails with 'partx -d ... error deleting partition' on shared-device OSD removal #749 (closed by ceph: treat already-removed partition entry as cleaned on WAL/DB cleanup #750)
• Jobs: DSL WAL-DB cleanup tests, case rm2, 4 failures on non-stack branches out of 155 runs (2 on main out of 83); DSL functional tests on squid, 1 failure

What version of MicroCeph are you using ?

CI builds of main; the earliest failures examined here are at b8f8772 (2026-08-03) and again at 496197b (2026-08-17). The code is the same at 5df62e1, and on squid at f9e5a87, which already has the backport #857.

What are the steps to reproduce this issue ?

1. Add two OSDs whose generated WAL and DB partitions sit on the same carrier disks (microceph disk add ... --wal-match ... --wal-size 1GiB --db-match ... --db-size 2GiB).
2. Run microceph disk remove <first-osd-id> --bypass-safety-checks. This is case rm2 of the DSL cleanup test; it is intermittent, and in every failing job rm1 and rmr in the same run pass.

What happens (observed behaviour) ?

The removal fails with failed to clean generated WAL/DB partitions for osd.1: failed to remove kernel partition entry 1 on ...: Failed to run: partx -d --nr 1:1 ...: exit status 1 (partx: ...: error deleting partition 1). That message is returned only when partx -d failed and the partition still resolves afterwards, so the kernel entry was still there. The "already removed" handling of #750 and #857 does not apply. It also means this is not the race described in #835 (sfdisk had already dropped the entry): util-linux partx treats ENXIO as success, so the failure was another errno. EBUSY is the likely one, with an opener such as udev probing the carrier after the zap and sfdisk writes. This is inferred: partx prints no strerror and the logs of the failing jobs contain no daemon journal. Rm2 is the only case hit, probably because the sibling partition is held open by the second, running OSD, so that sfdisk's re-read cannot refresh the kernel table and partx -d is the only remover, with no retry (inferred).

Rate: about 2.6% of cleanup runs (9 of 343). Both main failures ran after #750 merged on main, and one squid run failed after the backport, so the existing handling does not prevent it. Five jobs on the RGW stack (#881 to #893) hit it at 5/188 (2.7%), the same rate as non-stack branches (4/155, 2.6%), and the stack changes none of the code involved.

What were you expecting to happen ?

Removing one of two OSDs that share a carrier succeeds even if something briefly holds the partition open.

Relevant logs, error output, etc.

• main 2026-08-03: https://github.com/canonical/microceph/actions/runs/30780942450/job/91585433912 (line 1544, rm2 FAIL line 1875; rm1 passes at 893, rmr at 1458)
• main 2026-08-17: https://github.com/canonical/microceph/actions/runs/32045067391/job/95432753810 (line 1541)
• squid 2026-09-22: https://github.com/canonical/microceph/actions/runs/35678317751/job/106589704288 (line 1323, DSL functional tests, with the backport applied)
• stack 2026-09-23: https://github.com/canonical/microceph/actions/runs/35817330753/job/107043196746 (line 1565, DB carrier); 2026-09-29: https://github.com/canonical/microceph/actions/runs/36622063361/job/109593410755 (line 1563, WAL carrier)
• other branches: https://github.com/canonical/microceph/actions/runs/36720433634/job/109907395713 (core26-smb-feat, 2026-09-30, line 1526)
• Code at main:

  microceph/microceph/ceph/osd_waldb.go

  Lines 565 to 594 in 5df62e1

  	func (m *OSDManager) deletePartition(parentPath string, partition uint64) error {
  	_, err := m.resolvePartitionStablePath(parentPath, partition)
  	if err != nil {
  	logger.Infof("Partition %d on %s is already absent, skipping delete", partition, parentPath)
  	return nil
  	}
  	_, err = m.runner.RunCommand("sfdisk", "--delete", parentPath, strconv.FormatUint(partition, 10))
  	if err != nil {
  	_, resolveErr := m.resolvePartitionStablePath(parentPath, partition)
  	if resolveErr != nil {
  	logger.Infof("Partition %d on %s disappeared despite delete error, treating as cleaned", partition, parentPath)
  	return nil
  	}
  	return fmt.Errorf("failed to delete partition %d on %s: %w", partition, parentPath, err)
  	}
  	partitionRange := fmt.Sprintf("%d:%d", partition, partition)
  	_, err = m.runner.RunCommand("partx", "-d", "--nr", partitionRange, parentPath)
  	if err != nil {
  	_, resolveErr := m.resolvePartitionStablePath(parentPath, partition)
  	if resolveErr != nil {
  	logger.Infof("Kernel partition entry %d on %s already removed, treating as cleaned", partition, parentPath)
  	return nil
  	}
  	return fmt.Errorf("failed to remove kernel partition entry %d on %s: %w", partition, parentPath, err)
  	}
  	return nil
  	}

  (sfdisk --delete without --no-reread at line 572, partx -d at 583, swallowed only when the partition no longer resolves at 585-588, returned at 590); createPartition already uses --no-reread at

  microceph/microceph/ceph/osd_waldb.go

  Line 358 in 5df62e1

  cmd := fmt.Sprintf("printf ',+%dMiB\n' | sfdisk --append --force --no-reread %q", sizeMiB, parentPath)

Additional comments.

#857 left --no-reread and skipping partx as a separate follow-up on both branches, and #835 was closed without it.

Proposed fix in deletePartition, on main and squid:

1. Pass --no-reread to sfdisk --delete, so partx is the only actor on the kernel table.
2. Retry partx -d with a short backoff (for example 5 x 200 ms) while the partition still resolves; poll rather than rely on udevadm settle, whose availability under strict confinement was not checked.
3. After the retries, fall back to a re-read (partx -u or blockdev --rereadpt) and re-check that the partition is gone before returning an error.
4. Only run partx when the kernel entry exists, and make a re-run of microceph disk remove skip sfdisk if the GPT entry is already gone (inferred that a re-run fails at the sfdisk step today; not exercised in CI).
5. Add a unit test next to TestDeletePartitionTreatsAlreadyRemovedKernelEntryAsCleaned where partx fails twice with EBUSY and then succeeds.

On the test side, collect snap logs microceph.daemon and the partx and sfdisk output when "first OSD remove should succeed" fails, so the errno becomes observable.