johnramsden · Sep 18, 2026 · Oct 1, 2026
diff --git a/HACKING.md b/HACKING.md
@@ -10,6 +10,7 @@ Wait! No-one's "hacking" anything. The aim of this document is to enable new dev
 4. [🏭 Build Guide](#🏭-build-guide)
 5. [👍 Unit-Testing](#👍-unit-testing)
 6. [DB Schema Updates](#db-schema-updates)
+7. [RGW placement (in development)](#rgw-placement-in-development)
 
 ## ⚡️ Introduction to snaps
 
@@ -162,3 +163,77 @@ go generate ./database/host_tag.go
 ```
 
 This reads the `//go:generate mapper` directives in the source file and produces the corresponding `.mapper.go` file with SQL statements and Go functions.
+
+## RGW placement (in development)
+
+MicroCeph can place the RADOS Gateway (RGW) on cluster members from the placement policy sent to `PUT /1.0/placement`. Each member entry in that policy can carry an `rgw` object. This is still being built, so it has no page in the published documentation yet. Until it does, this section is the reference. Keep it current when you change the behaviour.
+
+### The `rgw` object
+
+`mode` is required by the placement API and is always `reconcile`.
+
+```json
+{
+  "mode": "reconcile",
+  "members": {
+    "node-a": { "rgw": { "enabled": true, "ssl": false, "port": 8080 } },
+    "node-b": { "rgw": { "enabled": true, "ssl": true, "ssl_certificate": "<base64 PEM>", "ssl_private_key": "<base64 PEM>" } },
+    "node-c": { "rgw": { "enabled": false } }
+  }
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `enabled` | Required. `true` runs a gateway on the member, `false` removes it. |
+| `ssl` | Required when `enabled` is true. TLS is never guessed from the certificate fields. |
+| `port` | Plaintext port. Defaults to 80 without TLS. With TLS it is optional: set it to serve plaintext as well. |
+| `ssl_port` | TLS port. Defaults to 443. |
+| `ssl_certificate`, `ssl_private_key` | Base64 PEM. Write-only: sent to the member, never stored in the database, never returned by `GET`. |
+
+### Rules that hold today
+
+- **A member with no `rgw` object is not touched.** Nothing reconciles between `PUT`s. An empty policy and `DELETE /1.0/placement` stop no gateway.
+- **The shape is strict.** A bare boolean `rgw`, a missing `enabled` or `ssl`, a certificate without its key, and the same port for both listeners are all rejected before anything changes.
+- **`ssl: true` without a certificate and key means reuse.** The member keeps the pair it already has. If it has no usable pair the request fails. It never falls back to plaintext.
+- **Additions come before removals.** Within one request every gateway addition is tried, then the removals. If any addition fails, all removals wait, so a migration never stops the old gateway before the new one serves. Gateways can scale to zero.
+- **`rgw_frontend` in `GET /1.0/placement` is the last applied setting,** read from the `rgw_frontends` table. It is not a health check. If it is absent, the setting is unknown, not plaintext.
+- **One owner per member.** Do not use `microceph enable rgw` or `microceph disable rgw` on a member whose policy entry has an `rgw` object. The next `PUT` applies the policy again without warning. `microceph certificate set rgw` is safe and never changes the policy.
+- **Applying is safe to repeat.** The same settings cause no restart. A failed change puts the previous config and gateway back. A first gateway start can use the whole 2 minute readiness wait, and the call to each member times out after 5 minutes.
+- **Certificate updates have the same 5 minute request budget.** `microceph certificate set rgw --restart` can wait up to 2 minutes for readiness and then roll back on failure; the client timeout leaves time for both.
+
+### What a `PUT` returns for RGW
+
+| Status | Meaning |
+|---|---|
+| 200 | Stored, and every gateway change succeeded. |
+| 400 | Invalid `rgw` object or certificate, unknown member, or a target member that does not list `placement-rgw` in `GET /1.0/cluster/capabilities`. |
+| 409 | Another apply is in progress. |
+| 500 | A gateway could not be enabled or disabled. The message names each member and keeps any other refusal from the same request. |
+| 503 | The cluster is partly refreshed. |
+
+### On the member
+
+| File | Role |
+|---|---|
+| `/var/snap/microceph/current/conf/radosgw.conf` | Once the file exists, only the `rgw frontends` line is rewritten, so a monitor list refresh made at the same time survives. The first enable writes the whole file. |
+| `/var/snap/microceph/current/conf/radosgw.conf.pending` | Holds the previous config while a change is in flight. If it is present at the next apply, the gateway is restarted rather than trusted. |
+| `/var/snap/microceph/common/rgw-tls/<hash>/server.crt`, `server.key` | One directory per certificate and key pair, named by a hash of both. The config switches to a directory only once both files are complete. Old directories are removed once the apply commits: after the database row on a `PUT`, or right after `certificate set rgw --restart`. |
+
+### Upgrades
+
+- A policy stored while `rgw` was still a boolean cannot be decoded. `GET /1.0/placement` returns 500 until the next `PUT` or `DELETE` replaces it.
+- A TLS gateway set up by an earlier revision keeps its pair in `server.crt` and `server.key`. The first apply that touches it moves the pair into a hashed `rgw-tls/` directory and restarts the gateway once.
+- A gateway that was running before this revision has no `rgw_frontends` row until its first apply.
+
+### Where the code lives
+
+All paths are under `microceph/` unless noted.
+
+| Path | Role |
+|---|---|
+| `api/types/placement.go` | The `rgw` object and its validation. |
+| `ceph/placement.go` | The cluster pass: order, held removals, error classes. |
+| `ceph/services_placement_rgw.go`, `ceph/rgw.go` | Applying on one member: TLS pair, readiness, rollback, disable. |
+| `database/rgw_frontend.go` | The `rgw_frontends` table. |
+| `tests/robot/rgw-placement-tests`, `tests/robot/rgw-placement-multivm-tests` (repository root) | The end-to-end suites. |