test(robot): add the three-VM RGW placement suite

[johnramsden]

Description

Some behaviour only shows with real, separate machines, so this suite runs three LXD VMs as independent cluster members: a policy sent to one member enabling a gateway on another, a third member reading S3 from it, a port change requested from yet another member, and failures that only exist across a real network. Its 9 cases cover a migration during which an in-guest sampler reads one object without pause and every read must succeed on at least one gateway; two applies started together on different members ending as one 200 and one 409; a member that is switched off still reporting its last applied frontend, failing the apply with 500, not starving the other addition, and holding back the paired removal; a keep-one refusal reported together with an unreachable member; and scale to zero, twice.

resources/rgw_placement.py gains helpers to start detached PUTs inside a guest and to run a read sampler inside a guest rather than on the host — a migration finishes in under a second, and a host-side sampler is too slow to catch a gap that short. resources/microceph_harness.py gains multi-VM lifecycle helpers and cluster-readiness waits. wait_for_microceph_ready_in_vm checks the daemon actually answers again after a guest is stopped and started, since the control socket file itself survives that and is not proof the daemon behind it does.

resources/placement_status.py's cluster_member_names now matches exact member names rather than prefixes, and the new migration_samples detects an outage or an unfinished sampler in the collected read log, both covered by pytest.

Adds rgw-placement-multivm-tests to the robot-tests matrix. The suite keeps three nested guests running at once, so it sets runner: self-hosted-linux-amd64-jammy-xlarge, the runner deferred-ceph-multinode-tests already uses.

Stack: PR 11 of 13. Merge in order. Each PR is one logical change on top of the one before it.

1. feat(db): add a table for each member's last applied RGW frontend #881 feat(db): add a table for each member's last applied RGW frontend
2. fix(services): keep error classes through the service enable path #882 fix(services): keep error classes through the service enable path
3. feat(api): describe an RGW placement as an object with explicit TLS intent #883 feat(api): describe an RGW placement as an object with explicit TLS intent
4. feat(rgw): make the member-side RGW apply safe to repeat #884 feat(rgw): make the member-side RGW apply safe to repeat
5. feat(rgw): wait until the gateway really serves the requested frontend #885 feat(rgw): wait until the gateway really serves the requested frontend
6. feat(rgw): reuse the member's own pair when TLS is asked for without one #886 feat(rgw): reuse the member's own pair when TLS is asked for without one
7. fix(rgw): make RGW disable converge when repeated or interrupted #887 fix(rgw): make RGW disable converge when repeated or interrupted
8. feat(placement): accept the RGW object in the policy and keep its secrets out #888 feat(placement): accept the RGW object in the policy and keep its secrets out
9. feat(placement): reconcile RGW across the cluster #889 feat(placement): reconcile RGW across the cluster
10. test(robot): add the single-VM RGW placement suite #890 test(robot): add the single-VM RGW placement suite
11. test(robot): add the three-VM RGW placement suite #891 test(robot): add the three-VM RGW placement suite (this PR)
12. docs: keep the RGW placement contract in HACKING.md for now #892 docs: keep the RGW placement contract in HACKING.md for now
13. fix(rgw): do not trust a pending-apply marker the operator completed by hand #893 fix(rgw): do not trust a pending-apply marker the operator completed by hand

Reviewer note: microceph_api_delete_in_vm (in microceph_harness.py) is defined but has zero callers anywhere in the Robot tree (verified with a repo-wide grep) — dead code as of this suite.

Type of change

• Clean code (code refactor, test updates; does not introduce functional changes)

How has this been tested?

• Unit tests pass at this stack's cumulative tip
• Robot tests pass

Contributor checklist

Please check that you have:

• self-reviewed the code in this PR
• added code comments, particularly in less straightforward areas
• checked and added or updated relevant documentation
• checked and added or updated relevant release notes
• added tests to verify effectiveness of this change

[sabaini]

some testing nits

[johnramsden]

I restructured the tests that need to run concurrently to be more resilient and easier to understand.