Repository navigation
Commit 71e3fa0
feat(rgw): make the member-side RGW apply safe to repeat
EnableRGW in ceph/rgw.go was a one-shot. It wrote radosgw.conf, overwrote server.crt and server.key in place, returned as soon as the service start command succeeded, and RgwServicePlacement.HospitalityCheck refused a gateway that was already running. A caller that repeats a request until the cluster converges needs the opposite, and this is the commit the rest of the series exists to enable: EnableRGW is replaced by applyRGWFrontend, and both "microceph enable rgw" and "microceph certificate set rgw" go through it from here on, not later.
applyRGWFrontend renders the wanted radosgw.conf in memory and compares only its "rgw frontends" line against the file on disk: matching line, matching pair on disk and a running service means nothing to do. Comparing only that one line, not the whole file, is deliberate: it is what lets an out-of-band "mon host" refresh made by the existing per-minute updateRadosGWMonHost loop survive an apply untouched instead of the two fighting over the file.
The certificate and key move into rgw-tls/<hash of both>/, one directory per pair, because a pair rewritten in place cannot promise the certificate and key change together: a crash between two renames could leave a certificate beside the wrong key, which is exactly what the old flat writeSSLFiles could do. The config only ever names a generation once both files are staged, so switching pairs is one line in the config, and rolling back is pointing that line at the previous, still-intact directory. A journal, radosgw.conf.pending, is written before anything changes: a marker still present at the start of the next apply means an earlier attempt did not finish, so the next apply forces a restart rather than trusting whatever the process happens to be serving, and takes the marker's recorded state, not whatever is half-published on disk, as the rollback target. radosgwConfMu is released before the restart and the readiness wait, which can take minutes, so a slow apply never blocks the unrelated per-minute mon-host refresh, which needs the same lock for one line.
In ceph/services_placement_rgw.go, the five-phase pipeline is what finally exercises two things this series built earlier and left unused: PopulateParams validates the request with types.RgwPlacement.Normalized, added inert in "feat(api): describe an RGW placement as an object with explicit TLS intent", and returns ErrRgwFrontendInvalid on a bad one, the sentinel added inert in "fix(services): keep error classes through the service enable path". DbUpdate upserts the rgw_frontends row, added inert in "feat(db): add a table for each member's last applied RGW frontend", in the same transaction as the services row, so presence and frontend can never disagree. HospitalityCheck becomes a deliberate no-op: an already-running gateway must be reconciled, not rejected, which is this branch's whole point. PostPlacementCheck and DbUpdate both call restore if the gateway stops or the database write fails. UpdateRGWCertificates now takes the restart flag itself, and cmdCertificatesRGWPut in api/certificates.go maps its errors with serviceErrorResponse.
Three things here are deliberately simple, and each is replaced by name in one of the next three commits: checkRGWReady only checks that the service is active, not that it actually serves, replaced in "feat(rgw): wait until the gateway really serves the requested frontend"; ServiceInit refuses ssl=true without a supplied certificate and key instead of reusing the member's own pair, replaced in "feat(rgw): reuse the member's own pair when TLS is asked for without one"; and DisableRGW only removes the new files and keeps its old mock-runner test rather than converging on a repeat or interrupted call, replaced in "fix(rgw): make RGW disable converge when repeated or interrupted". No cluster pass dispatches RGW yet; a gateway is still placed only by a direct enable or certificate request, until "feat(placement): reconcile RGW across the cluster" wires this pipeline into the reconciler.
Co-authored-by: Peter Sabaini <peter.sabaini@canonical.com>
Assisted-by: claude-code:claude-fable-5-1
Assisted-by: pi:gpt-6-astra
Signed-off-by: John Ramsden <john.ramsden@canonical.com>1 parent 610f8b4 commit 71e3fa0
11 files changed
Lines changed: 1729 additions & 434 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
28 | 33 | | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
| 34 | + | |
37 | 35 | | |
38 | 36 | | |
39 | 37 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
| 5 | + | |
4 | 6 | | |
5 | 7 | | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
| 12 | + | |
| 13 | + | |
10 | 14 | | |
11 | 15 | | |
12 | 16 | | |
| 17 | + | |
13 | 18 | | |
14 | 19 | | |
15 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
16 | 60 | | |
17 | 61 | | |
18 | 62 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
404 | 404 | | |
405 | 405 | | |
406 | 406 | | |
407 | | - | |
408 | | - | |
409 | | - | |
410 | | - | |
411 | | - | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
412 | 413 | | |
413 | 414 | | |
414 | 415 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
35 | 54 | | |
36 | 55 | | |
37 | 56 | | |
| |||
0 commit comments