feat(rgw): make the member-side RGW apply safe to repeat

[johnramsden]

Description

EnableRGW in ceph/rgw.go was a one-shot. It wrote radosgw.conf, overwrote server.crt and server.key in place, returned as soon as the service start command succeeded, and RgwServicePlacement.HospitalityCheck refused a gateway that was already running. A caller that repeats a request until the cluster converges needs the opposite, and this is the commit the rest of the series exists to enable: EnableRGW is replaced by applyRGWFrontend, and both microceph enable rgw and microceph certificate set rgw go through it from here on.

applyRGWFrontend renders the wanted radosgw.conf in memory and compares only its rgw frontends line against the file on disk: matching line, matching pair on disk and a running service means nothing to do. Comparing only that one line, not the whole file, is deliberate: it lets an out-of-band mon host refresh made by the existing per-minute updateRadosGWMonHost loop survive an apply untouched instead of the two fighting over the file.

The certificate and key move into rgw-tls/<hash of both>/, one directory per pair, because a pair rewritten in place cannot promise the certificate and key change together: a crash between two renames could leave a certificate beside the wrong key, which the old flat writeSSLFiles could do. The config only ever names a generation once both files are staged, so switching pairs is one line in the config, and rolling back is pointing that line at the previous, still-intact directory. A journal, radosgw.conf.pending, is written before anything changes: a marker still present at the start of the next apply means an earlier attempt did not finish, so the next apply forces a restart rather than trusting whatever the process happens to be serving, and takes the marker's recorded state, not whatever is half-published on disk, as the rollback target. radosgwConfMu is released before the restart and the readiness wait, which can take minutes, so a slow apply never blocks the unrelated per-minute mon-host refresh, which needs the same lock for one line.

In ceph/services_placement_rgw.go, the five-phase pipeline finally exercises two things this series built earlier and left unused: PopulateParams validates with types.RgwPlacement.Normalized and returns ErrRgwFrontendInvalid on a bad request; DbUpdate upserts the rgw_frontends row in the same transaction as the services row, so presence and frontend can never disagree. HospitalityCheck becomes a deliberate no-op: an already-running gateway must be reconciled, not rejected, which is this branch's whole point.

Stack: PR 4 of 13. Merge in order. Each PR is one logical change on top of the one before it.

1. feat(db): add a table for each member's last applied RGW frontend #881 feat(db): add a table for each member's last applied RGW frontend
2. fix(services): keep error classes through the service enable path #882 fix(services): keep error classes through the service enable path
3. feat(api): describe an RGW placement as an object with explicit TLS intent #883 feat(api): describe an RGW placement as an object with explicit TLS intent
4. feat(rgw): make the member-side RGW apply safe to repeat #884 feat(rgw): make the member-side RGW apply safe to repeat (this PR)
5. feat(rgw): wait until the gateway really serves the requested frontend #885 feat(rgw): wait until the gateway really serves the requested frontend
6. feat(rgw): reuse the member's own pair when TLS is asked for without one #886 feat(rgw): reuse the member's own pair when TLS is asked for without one
7. fix(rgw): make RGW disable converge when repeated or interrupted #887 fix(rgw): make RGW disable converge when repeated or interrupted
8. feat(placement): accept the RGW object in the policy and keep its secrets out #888 feat(placement): accept the RGW object in the policy and keep its secrets out
9. feat(placement): reconcile RGW across the cluster #889 feat(placement): reconcile RGW across the cluster
10. test(robot): add the single-VM RGW placement suite #890 test(robot): add the single-VM RGW placement suite
11. test(robot): add the three-VM RGW placement suite #891 test(robot): add the three-VM RGW placement suite
12. docs: keep the RGW placement contract in HACKING.md for now #892 docs: keep the RGW placement contract in HACKING.md for now
13. fix(rgw): do not trust a pending-apply marker the operator completed by hand #893 fix(rgw): do not trust a pending-apply marker the operator completed by hand

Reviewer note — three things here are deliberately simple, each replaced by name in one of the next three PRs in this stack: checkRGWReady only checks that the service is active, not that it actually serves (replaced in 05-readiness); ServiceInit refuses ssl=true without a supplied certificate and key instead of reusing the member's own pair (replaced in 06-tls-reuse); DisableRGW only removes the new files and keeps its old mock-runner test rather than converging on a repeat or interrupted call (replaced in 07-disable). No cluster pass dispatches RGW yet — a gateway is still placed only by a direct enable or certificate request, until 09-reconcile wires this pipeline into the reconciler.

Type of change

• New feature (non-breaking change which adds functionality)

How has this been tested?

• Unit tests pass at this stack's cumulative tip

Contributor checklist

Please check that you have:

• self-reviewed the code in this PR
• added code comments, particularly in less straightforward areas
• checked and added or updated relevant documentation
• checked and added or updated relevant release notes
• added tests to verify effectiveness of this change

[sabaini]

lgtm