feat(placement): reconcile RGW across the cluster

[johnramsden]

Description

This is where the feature turns on end to end, from PUT /1.0/placement through to a member dispatch. reconcilePlacement now calls the new applyRgwPlacement once the control services (mon, mgr, mds) are settled, so control quorum is established before any gateway moves.

applyRgwPlacement splits the policy's members into those with rgw.enabled=true and those with enabled=false, sorted for a deterministic dispatch order, and calls enableRgwServiceFunc for every enable, collecting failures instead of stopping at the first. If any enable failed it returns without disabling anything, so a migration never stops the old gateway before the new one serves. Only when every enable succeeds does it move to the disable loop, which runs unconditionally: RGW has no keep-one rule, so a pure scale-to-zero is allowed, unlike mon/mgr/mds.

Error classes cross the network boundary here: rgwDispatchFailure treats a forwarded HTTP 400 from the member's own serviceErrorResponse as client-classified, and everything else — an unreachable member, a failed start, a failed teardown — as operational. rgwAggregateError then returns ErrRgwFrontendInvalid only when every collected failure was a forwarded 400, and ErrPlacementOperationFailed the instant any failure was not, so a genuinely mixed batch answers 500, never 400. retainRefusals appends an earlier keep-one refusal from the control pass to a later failure, so the reported reason names both instead of one silently replacing the other.

prod_wiring.go wires the production hooks, each a fail-closed default: with no implementation assigned, the outer function returns an explicit error naming the member rather than silently reporting success. prodEnableRgwServiceFunc first calls client.CheckRGWPlacementSupport, which asks the target member's own GET /1.0/cluster/capabilities for placement-rgw and fails closed with 400 if it is missing — an older member would otherwise ignore ssl and could turn a TLS retry into plaintext. api/capabilities.go advertises placement-rgw from this commit on, since this is the commit that starts acting on the field.

Stack: PR 9 of 13. Merge in order. Each PR is one logical change on top of the one before it.

1. feat(db): add a table for each member's last applied RGW frontend #881 feat(db): add a table for each member's last applied RGW frontend
2. fix(services): keep error classes through the service enable path #882 fix(services): keep error classes through the service enable path
3. feat(api): describe an RGW placement as an object with explicit TLS intent #883 feat(api): describe an RGW placement as an object with explicit TLS intent
4. feat(rgw): make the member-side RGW apply safe to repeat #884 feat(rgw): make the member-side RGW apply safe to repeat
5. feat(rgw): wait until the gateway really serves the requested frontend #885 feat(rgw): wait until the gateway really serves the requested frontend
6. feat(rgw): reuse the member's own pair when TLS is asked for without one #886 feat(rgw): reuse the member's own pair when TLS is asked for without one
7. fix(rgw): make RGW disable converge when repeated or interrupted #887 fix(rgw): make RGW disable converge when repeated or interrupted
8. feat(placement): accept the RGW object in the policy and keep its secrets out #888 feat(placement): accept the RGW object in the policy and keep its secrets out
9. feat(placement): reconcile RGW across the cluster #889 feat(placement): reconcile RGW across the cluster (this PR)
10. test(robot): add the single-VM RGW placement suite #890 test(robot): add the single-VM RGW placement suite
11. test(robot): add the three-VM RGW placement suite #891 test(robot): add the three-VM RGW placement suite
12. docs: keep the RGW placement contract in HACKING.md for now #892 docs: keep the RGW placement contract in HACKING.md for now
13. fix(rgw): do not trust a pending-apply marker the operator completed by hand #893 fix(rgw): do not trust a pending-apply marker the operator completed by hand

Reviewer note: two later PRs in this stack only add proof and documentation on top of this, nothing functional — the single-VM and three-VM Robot suites exercise this reconciler against a real snap and a real gateway, and the HACKING.md write-up lands after the behaviour it documents already exists.

Type of change

• New feature (non-breaking change which adds functionality)

How has this been tested?

• Unit tests pass at this stack's cumulative tip

Contributor checklist

Please check that you have:

• self-reviewed the code in this PR
• added code comments, particularly in less straightforward areas
• checked and added or updated relevant documentation
• checked and added or updated relevant release notes
• added tests to verify effectiveness of this change

[sabaini]

lgtm