fix(rgw): make RGW disable converge when repeated or interrupted

[johnramsden]

Description

DisableRGW returned at the first file that was already missing, and removeServiceDatabase returned an error when the services row was already gone. Disabling RGW is not one atomic step: stopping the snap service, dropping two database rows, and deleting several files can each fail independently, and a disable interrupted partway, or simply repeated, could therefore never finish.

DisableRGW now takes serviceStartMu for the whole teardown, matching every other path that starts, stops or reconfigures a service on the member, and follows a fixed order: stop the service, drop the record, only then delete files. The record is dropped before any file so a failure there leaves files and record consistent — a well-defined point for a retry to resume from — and every removal past that point tolerates a file already being gone instead of failing at the first one. Remaining failures are collected rather than returned on the first hit, then joined into one error and wrapped in ErrPlacementOperationFailed.

removeServiceDatabase treats a not-found from database.DeleteService as success, but only for the rgw service: mon, mgr and mds keep the strict behaviour, since an absent row for those means a real inconsistency, not a convergent retry.

Stack: PR 7 of 13. Merge in order. Each PR is one logical change on top of the one before it.

1. feat(db): add a table for each member's last applied RGW frontend #881 feat(db): add a table for each member's last applied RGW frontend
2. fix(services): keep error classes through the service enable path #882 fix(services): keep error classes through the service enable path
3. feat(api): describe an RGW placement as an object with explicit TLS intent #883 feat(api): describe an RGW placement as an object with explicit TLS intent
4. feat(rgw): make the member-side RGW apply safe to repeat #884 feat(rgw): make the member-side RGW apply safe to repeat
5. feat(rgw): wait until the gateway really serves the requested frontend #885 feat(rgw): wait until the gateway really serves the requested frontend
6. feat(rgw): reuse the member's own pair when TLS is asked for without one #886 feat(rgw): reuse the member's own pair when TLS is asked for without one
7. fix(rgw): make RGW disable converge when repeated or interrupted #887 fix(rgw): make RGW disable converge when repeated or interrupted (this PR)
8. feat(placement): accept the RGW object in the policy and keep its secrets out #888 feat(placement): accept the RGW object in the policy and keep its secrets out
9. feat(placement): reconcile RGW across the cluster #889 feat(placement): reconcile RGW across the cluster
10. test(robot): add the single-VM RGW placement suite #890 test(robot): add the single-VM RGW placement suite
11. test(robot): add the three-VM RGW placement suite #891 test(robot): add the three-VM RGW placement suite
12. docs: keep the RGW placement contract in HACKING.md for now #892 docs: keep the RGW placement contract in HACKING.md for now
13. fix(rgw): do not trust a pending-apply marker the operator completed by hand #893 fix(rgw): do not trust a pending-apply marker the operator completed by hand

Reviewer note: this makes DisableRGW convergent at the unit level only; nothing yet calls it from a reconciliation loop, and nothing exercises it against a real snap. 09-reconcile dispatches removal through this same path, and the Robot suite in 10-robot-single exercises the partial-cleanup and repeat cases live.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How has this been tested?

• Unit tests pass at this stack's cumulative tip

Contributor checklist

Please check that you have:

• self-reviewed the code in this PR
• added code comments, particularly in less straightforward areas
• checked and added or updated relevant documentation
• checked and added or updated relevant release notes
• added tests to verify effectiveness of this change

[sabaini]

tiny test nit

[sabaini]

ty!