feat(placement): accept the RGW object in the policy and keep its secrets out

[johnramsden]

Description

MemberPlacement.Rgw changes from *bool to *RgwPlacement, the object added earlier in this series. A bare rgw boolean no longer decodes: encoding/json cannot unmarshal true into a struct field, so a pre-branch client's request now fails at decode time with a 400 instead of silently doing the wrong thing. cmdPlacementPut also now rejects any JSON after the policy object, closing a framing gap where a client confused about its own payload had the extra bytes silently discarded.

ApplyPlacementPolicy calls the new normalizePlacement first, before validation, storage or reconciliation. It runs RgwPlacement.Normalized on every member's rgw entry and returns ErrRgwFrontendInvalid, now mapped to 400, so a malformed entry is rejected before it can replace the stored policy or reach a member. This is live from this commit: a PUT with a bad rgw entry is rejected today, even though nothing yet dispatches a good one.

The certificate and key must never reach the database or a GET, and that guard is live too. placementPolicyJSON is now the only encoder storePlacementPolicy uses, and it always encodes policyForStorage(policy), a copy with the certificate and key blanked; the apply itself keeps the original, material-bearing policy for reconciliation to use once it exists. As defense in depth, GetPlacementStatus also calls redactStoredPolicy on whatever it reads back, in case a future code path ever stores the raw policy by accident.

GetPlacementStatus also reads rgw_frontends through its first production reader, populateRGWFrontends. It attaches an RgwObservedFrontend (port, ssl_port, ssl) to a member only when that member is separately observed running RGW; a member with no row keeps no rgw_frontend field at all, so "unknown" is never reported as a plaintext default.

Stack: PR 8 of 13. Merge in order. Each PR is one logical change on top of the one before it.

1. feat(db): add a table for each member's last applied RGW frontend #881 feat(db): add a table for each member's last applied RGW frontend
2. fix(services): keep error classes through the service enable path #882 fix(services): keep error classes through the service enable path
3. feat(api): describe an RGW placement as an object with explicit TLS intent #883 feat(api): describe an RGW placement as an object with explicit TLS intent
4. feat(rgw): make the member-side RGW apply safe to repeat #884 feat(rgw): make the member-side RGW apply safe to repeat
5. feat(rgw): wait until the gateway really serves the requested frontend #885 feat(rgw): wait until the gateway really serves the requested frontend
6. feat(rgw): reuse the member's own pair when TLS is asked for without one #886 feat(rgw): reuse the member's own pair when TLS is asked for without one
7. fix(rgw): make RGW disable converge when repeated or interrupted #887 fix(rgw): make RGW disable converge when repeated or interrupted
8. feat(placement): accept the RGW object in the policy and keep its secrets out #888 feat(placement): accept the RGW object in the policy and keep its secrets out (this PR)
9. feat(placement): reconcile RGW across the cluster #889 feat(placement): reconcile RGW across the cluster
10. test(robot): add the single-VM RGW placement suite #890 test(robot): add the single-VM RGW placement suite
11. test(robot): add the three-VM RGW placement suite #891 test(robot): add the three-VM RGW placement suite
12. docs: keep the RGW placement contract in HACKING.md for now #892 docs: keep the RGW placement contract in HACKING.md for now
13. fix(rgw): do not trust a pending-apply marker the operator completed by hand #893 fix(rgw): do not trust a pending-apply marker the operator completed by hand

Reviewer note: nothing yet acts on the object's contents — reconcilePlacement's RGW handling does not exist, so a PUT is accepted, validated and stored, but no gateway is started, stopped or reconfigured from it. 09-reconcile is the first commit to read the field's actual contents.

Known and accepted: a policy stored while rgw was still a boolean no longer decodes, so GET /1.0/placement returns 500 until the next PUT or DELETE replaces it.

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected): the rgw policy field changes from a boolean to an object

How has this been tested?

• Unit tests pass at this stack's cumulative tip

Contributor checklist

Please check that you have:

• self-reviewed the code in this PR
• added code comments, particularly in less straightforward areas
• checked and added or updated relevant documentation
• checked and added or updated relevant release notes
• added tests to verify effectiveness of this change

[sabaini]

lgtm

[hillpd]

Merging #887 and then #888 conflicts in rgw_test.go and services_placement_rgw_test.go (git merge-tree --write-tree pr887 pr888). Tracing it back, this PR is based on an stale version of #885 and carries its own copies of #886 and #887 (2b8c12e, 3c82483). Since then #885 gained e935718 and 38c72c3 and #887 gained 0d9089a, none of which are present here. Could you rebase onto the current #887 head and drop the two copied commits? That resolves the conflict and brings the PEM block-skip fix into #889 to #893, which currently inherit the bug.