docs: clarify the ALTER CLUSTER graceful-resize default and system clusters

[claude]

Requested by Tyler Hartwig · Slack thread

Motivation

User feedback relayed internally reported that the ALTER CLUSTER /
WAIT UNTIL READY docs are confusing on three points:

1. It is unclear whether system/builtin clusters (mz_catalog_server,
   mz_system, mz_probe, mz_support, mz_analytics) behave differently
   from user clusters.
2. Old and new ALTER CLUSTER behavior are hard to tell apart, as is what is
   GA versus private preview.
3. It is not clear that the new default behavior is a graceful, zero-downtime
   resize.

All three trace to specific text, and each one checks out against the code:

• The SIZE syntax element in doc/user/data/examples/alter_cluster.yml
  carried a warning that "changing the size of a cluster may incur
  downtime". That is the first thing a reader sees when looking up how to
  resize, and it directly contradicted the Resizing section on the same page,
  which says a resize incurs no downtime.
• A bare, unlabeled {{< private-preview />}} sat on the legacy section
  heading, where it reads as marking zero-downtime resizing as a whole rather
  than the option it was meant to scope.
• Nothing in doc/user/ said anything about how ALTER CLUSTER treats system
  clusters, while the self-managed troubleshooting guide asserted graceful
  resizing of mz_catalog_server unconditionally. That assertion does not hold
  on any currently released version.

Description

Before / After (what a reader of the docs page sees):

• SIZE option, ALTER CLUSTER syntax table. Before: a warning that
  changing a cluster's size "may incur downtime". After: a note that resizing
  is graceful and incurs no downtime by default, that no WITH option is
  needed to get that, and a pointer to the system-cluster caveat.

• Resizing, current behavior. Before: an H4 titled "Downtime
  considerations for v26.35 or after", whose body opened with unformatted
  ALTER CLUSTER <name> SET (SIZE = ...), and which interleaved the default
  behavior with the timeout options. After: "Resizing is graceful by default",
  with the version stated in the body as v26.35 on Cloud and v26.34.1 on
  Self-Managed rather than in the heading, stating plainly that this is the
  default and that
  WAIT UNTIL READY is not required for a zero-downtime resize, that the
  statement returns immediately, and that the default deadline is 24 hours with
  a rollback if the new replicas have not hydrated by then.

• The distinction at the heart of the confusion. Before: the page implied
  that customizing the timeout is what gets you the graceful path. After: the
  page says the WAIT options do not enable graceful resizing and are not
  required for it, and that their only effect is to set the deadline and choose
  what happens when it passes. The timeout options move under their own
  "Customizing the timeout" heading, so the default and its customization are
  no longer interleaved. It also records that the WAIT options still return
  immediately and do not hold the session open, which is the part that changed
  from the blocking behavior of v26.33 and earlier.

• Preview badges. Before: two badges, one labeled and scoped to the timeout
  options, one bare and unlabeled on the legacy heading. After: only the
  labeled one, still scoped to WAIT UNTIL READY / WAIT FOR. The legacy
  heading is retitled "Resizing in v26.33 and earlier", which is what it is
  actually about.

• System clusters. Before: nothing. After: a "System clusters" section
  saying ALTER CLUSTER ... SET (SIZE = ...) applies to system clusters too,
  and a warning that a bare resize of one did not take the graceful path before
  v26.38, while an explicit WAIT UNTIL READY was honored and blocked the
  session, with the option shown as the workaround on earlier versions.

• Self-managed troubleshooting. Before: the mz_catalog_server resize
  walkthrough stated unconditionally that the resize is a graceful
  reconfiguration. After: the same guidance, qualified by version, with the
  WAIT UNTIL READY spelling given for v26.37 and earlier. The walkthrough is
  kept rather than removed, since resizing mz_catalog_server is still the
  right fix for an unresponsive Console.

How. Three files, docs only. doc/user/data/examples/alter_cluster.yml
holds the rendered syntax tables, so the SIZE warning lives there rather than
in the page. The Resizing process, Monitoring a resize and Cancel a resize sub-headings are left untouched, because #resizing-process and
#monitoring-a-resize are linked from other pages. The retitled legacy heading
carries an explicit {#resizing-in-v2633-and-earlier} anchor, since the
version number's period makes the generated ID non-obvious.

Two drive-by fixes in text already being edited: a REPLICATON FACTOR typo,
and a SET (SIZE '100cc') example missing its equals sign.

Verification

• Confirmed against the code on main that a bare
  ALTER CLUSTER c SET (SIZE = ...) is graceful and ungated. The planner
  produces AlterClusterPlanStrategy::None
  (src/sql/src/plan/statement/ddl.rs), a shape change is still reshaped into
  a durable reconfiguration record, and the record's defaults for a statement
  with no WITH clause are DEFAULT_CLUSTER_RECONFIGURATION_TIMEOUT (24h) and
  OnTimeoutAction::Rollback
  (src/adapter/src/coord/sequencer/inner/cluster.rs). The feature-flag check
  for enable_zero_downtime_cluster_reconfiguration runs only when a strategy
  was given, so the default path is not gated but the WAIT options are, which
  is why the labeled badge stays.
• Confirmed that the WAIT options no longer block. The immediate return is
  gated on the enable_background_alter_cluster dyncfg (default on) and not on
  the strategy, and the only producer of NeedsFinalization::Yes is
  unreachable while the controller owns every managed cluster. An earlier draft
  of this PR claimed the options still block; that was wrong and is corrected.
• Confirmed the system-cluster behavior change and its direction from adapter: let the cluster controller own system clusters #38102,
  which dropped the three is_user() conjuncts and records the behavior change
  as a bare ALTER CLUSTER mz_system SET (SIZE ...) going "from a synchronous
  whole-set recreation to a background graceful reconfiguration", noting that
  "the graceful path did not exist for system clusters before".
• Verified doc/user/data/examples/alter_cluster.yml still parses. The SIZE
  description is a plain (unquoted) multiline YAML scalar, so a ": " in the
  new prose broke the document mid-edit; the wording avoids the colon rather
  than restructuring the scalar, which keeps the existing folding behavior.
• Checked every in-page anchor on alter-cluster.md, including the links
  rendered onto it from the YAML data file. All resolve, including the two new
  headings and the explicit legacy anchor.
• Pinned the graceful-default version per deployment type against the tags and
  the release notes. enable_cluster_controller and
  enable_background_alter_cluster are both false in v26.34.0-rc.1 and both
  true in v26.34.0-rc.2 and released v26.34.0, so the compiled default
  flipped in v26.34.0. v26.34.0 nonetheless shipped without the SQL-530 fix
  (storage/adapter: fix graceful cluster reconfiguration for clusters with single-replica sources #37740), which merged 2026-07-20T16:12:12Z, 14 minutes after the v26.34.0
  tag at 2026-07-20T15:58:00Z. The fix is present from v26.34.1 onward,
  verified by Instance::get_active_replicas_for_object in
  src/storage-controller/src/instance.rs still carrying the old
  ingestion-export-only branch in v26.34.0 and the rewritten
  active_replica_ids match in v26.34.1.
• Confirmed the release notes draw the same split. v26.34.1 announces
  asynchronous cluster reconfiguration under an explicit
  "Materialize Self-Managed only" marker and lists the single-replica-source
  deadlock among its bug fixes, v26.34.0 never mentions the feature at all,
  and v26.35.0 announces it with a Cloud release date. _index.md gives
  v26.34.1 a Self-Managed line and no Cloud line.
• Did not run the Hugo build: no Hugo in this environment. The changes are
  markdown and YAML only and add no new shortcodes, only existing note,
  warning and private-preview invocations.

Open questions

1. RESOLVED: the release that carries adapter: let the cluster controller own system clusters #38102 is v26.38, now confirmed against
   a real tag. v26.38.0-rc.1 was tagged 2026-08-14T02:59:14Z, after this
   PR's earlier checks found no v26.38.* tag at all, and it does carry
   adapter: let the cluster controller own system clusters #38102. The routing gate is the evidence:
   src/adapter/src/coord/cluster_controller.rs filters with
   .filter(|c| c.is_managed() && c.id.is_user()) in v26.37.0 and with
   .filter(|c| c.is_managed()) in v26.38.0-rc.1, whose module comment reads
   "The controller owns the replica set of every managed cluster, user and system
   alike." enable_cluster_controller is also gone from
   src/adapter-types/src/dyncfgs.rs in that tag, so the controller is
   unconditional there rather than flag-gated. The v26.38 number in these
   edits is therefore confirmed rather than predicted. Note that v26.38.0 final
   is not out yet: doc/user/content/releases/v26.38.md still carries
   released: false with date 2026-08-19.

2. STILL OPEN: whether graceful resizing should carry a private-preview badge
   at all. This is a PM decision and is deliberately not resolved here beyond
   removing the unlabeled badge. What the code says is unchanged: the graceful
   default is ungated, while WAIT UNTIL READY and WAIT FOR still require
   enable_zero_downtime_cluster_reconfiguration, whose compiled-in default is
   false. So the badge is arguably stale for the behavior and accurate for the
   options, which is the split this PR encodes.

   One factual correction to an earlier note on this item: that flag does have
   a LaunchDarkly flag. It sits in INTENTIONAL_LD_OVERRIDES
   (test/launchdarkly-flag-consistency/mzcompose.py:505), the list of
   parameters whose "production LaunchDarkly value is deliberately different
   from the compiled-in default", not in KNOWN_MISSING_FROM_LD. That answers
   the sub-question of how it comes to be enabled where WAIT UNTIL READY
   demonstrably works: LaunchDarkly serves it on in cloud, which is also why a
   self-managed deployment has to enable it by hand. The badge decision itself
   remains for a PM.

3. RESOLVED: the system-cluster divergence was accidental. The author of
   adapter: let the cluster controller own system clusters #38102 confirmed it directly: the divergence "was an accident of how we
   incrementally developed and then rolled it out." It was not a policy that
   system clusters should behave differently. That matches adapter: let the cluster controller own system clusters #38102's mechanical
   reasoning recorded earlier in this PR: the exclusion was load-bearing only
   because the boot-time builtin replica migration and the controller would
   otherwise have been two conflicting writers of one replica set, a conflict
   adapter: make a builtin cluster's config own its replica set #37929 removed. Both pages were re-read against this: they state only what
   each version does, and nothing frames the difference as intended or as a
   property system clusters are supposed to have.

4. Not addressed here, suggested as follow-ups. reference/system-clusters.md
   lists per-cluster characteristics but says nothing about resizing, and omits
   mz_analytics entirely. adapter: let the cluster controller own system clusters #38102 is in v26.38.0-rc.1, so the v26.38 release
   notes need an entry covering the system-cluster change. Separately,
   alter-cluster.md carries a pre-existing em-dash in the legacy section
   ("cause a rollback — no size change will take effect"), which is already on
   main and is left alone here to keep this diff scoped.

5. RESOLVED: v26.35 on Cloud and v26.34.1 on Self-Managed, and the page now
   says both. A maintainer supplied the missing history from two directions:
   a LaunchDarkly flag was disabled 2026-07-24 over a bug and re-enabled
   2026-07-31 after the fix, and the bug "was about hydration checks for
   single replica sources."

   That second detail identifies the bug exactly. It is SQL-530, fixed by
   storage/adapter: fix graceful cluster reconfiguration for clusters with single-replica sources #37740, "storage/adapter: fix graceful cluster reconfiguration for
   clusters with single-replica sources": the readiness check gated cut-over on
   every ingestion reporting hydrated on the pending replicas, but a
   single-replica source stays on the replica it already runs on and is never
   placed on a pending replica until cut-over, so it could never report hydrated
   there and the reconfiguration deadlocked until its deadline, then rolled back
   without resizing. The fix skips off-target ingestions in
   collections_hydrated_on_replicas and additionally requires pending replicas
   to be online before cut-over, on both the legacy foreground path and the
   controller-owned path.

   The dates, all UTC:

   Event	Date	Note
   v26.33.0 tag	2026-07-15 15:26	Cloud 07-16, Self-Managed 07-17
   v26.33.1 tag	2026-07-16 22:46	not listed in _index.md
   v26.34.0-rc.1 tag	2026-07-17 03:08	both flags false
   v26.34.0-rc.2 tag	2026-07-18 01:25	both flags true
   v26.33.2 tag	2026-07-19 18:20	not listed in _index.md
   v26.34.0 tag	2026-07-20 15:58	Cloud 07-21, Self-Managed 07-21
   storage/adapter: fix graceful cluster reconfiguration for clusters with single-replica sources #37740 merged	2026-07-20 16:12	14 min after the v26.34.0 cut
   v26.34.1 tag	2026-07-23 20:51	Self-Managed 07-24, no Cloud line
   LD flag disabled	2026-07-24	per the maintainer
   v26.35.0-rc.1 tag	2026-07-24 03:09	carries the fix
   v26.35.0 tag	2026-07-29 14:56	Cloud 07-29, Self-Managed 07-30
   LD flag re-enabled	2026-07-31	per the maintainer
   v26.36.0 tag	2026-08-05 15:10	Cloud 08-07, Self-Managed 08-07
   v26.37.0 tag	2026-08-12 09:13	not listed in _index.md
   v26.38.0-rc.1 tag	2026-08-14 02:59	v26.38.0 unreleased

   The arithmetic lines up and explains the divergence. v26.34.0 was cut 14
   minutes before its own fix merged, so it shipped the graceful default with
   SQL-530 intact and went to Cloud on 07-21. The bug bit, and the flag was
   turned off on 07-24 while Cloud was still on v26.34.0. v26.34.1 carried
   the fix from 07-23 but went to Self-Managed only, so Cloud never ran a
   fixed v26.34 at all and only got the fix when v26.35.0 rolled out on 07-29.
   The 07-31 re-enable sits two days after that rollout, which is what the
   maintainer's "you know when we re-enabled it" points at: the first Cloud
   release carrying the fix is v26.35.0, not v26.34.1.

   The release notes already draw exactly this split, which is the strongest
   evidence available and does not depend on inferring anything. v26.34.0's
   notes never mention the feature. v26.34.1's notes announce "Asynchronous
   Cluster Reconfiguration" under an explicit "Materialize Self-Managed only"
   marker and list the single-replica-source deadlock among its bug fixes.
   v26.35.0's notes announce the same feature with a Cloud release date.

   So no single number is accurate for both audiences, and the page now says
   both rather than picking one that is wrong for half the readers:
   v26.35 on Cloud, v26.34.1 on Self-Managed. The version moves out of
   the heading, which can no longer carry one number, into a sentence in the
   body. Nothing linked the old heading's generated anchor, so nothing dangles.

   Two honest caveats for a reviewer with LaunchDarkly access. First, the
   maintainer did not name the flag, and this repo cannot identify it: the two
   flags that produce the graceful default, enable_cluster_controller and
   enable_background_alter_cluster, are both listed in KNOWN_MISSING_FROM_LD
   (lines 191-438), the parameters that "exist in Materialize but have no
   LaunchDarkly flag", and enable_cluster_controller appears there twice in
   v26.37.0, which suggests hand-editing around it. That list is self-described
   as drifting and pruned lazily, so its absence is evidence rather than proof.
   Either way the conclusion holds, because it rests on the tags and the release
   notes rather than on the flag. Second, v26.34.0 is still a version a
   Self-Managed operator can be running, and on it a bare resize of a cluster
   hosting a single-replica source will stall and roll back at its 24-hour
   deadline. The page does not currently warn about that. Worth deciding whether
   it should, though a defect in a superseded patch release is arguably not a
   docs-worthy version boundary.

Update 2026-09-04: merged with main, and the claims re-validated

main moved 315 commits while this PR sat. origin/main is merged in. One
conflict, in doc/user/content/sql/alter-cluster.md, over the legacy section
heading: upstream deleted the bare {{< private-preview />}} badge under it,
and this branch deleted that same badge and retitled the heading to "Resizing
in v26.33 and earlier". Both sides removed the badge, so the resolution keeps
this branch's version and nothing is duplicated.

Two upstream changes overtake parts of the original diff. Both are taken as-is
rather than re-asserted:

• sql: accept WITH (WAIT ...) on ALTER CLUSTER unconditionally #38627 removed the enable_zero_downtime_cluster_reconfiguration feature
  flag, so WITH (WAIT ...) is now accepted unconditionally, and the same
  commit removed every private-preview label from alter-cluster.md and from
  the WITH options table in alter_cluster.yml. The paragraph this PR added
  saying the WAIT options are "additionally gated behind" that flag on
  self-managed deployments no longer matches the code, so it is dropped. Open
  question 2 above is moot as a result: there is no badge left to decide about,
  and the flag it discussed is gone. The verification note above about "the
  labeled badge stays" is superseded for the same reason.
• adapter: make WAIT FOR roll back on timeout #38621 made WAIT FOR roll back on timeout rather than committing
  regardless of hydration status. Upstream rewrote that bullet on the page and
  in the YAML options table, and the merge takes upstream's wording rather than
  this branch's older "commits when it expires ... prefer WAIT UNTIL READY"
  text.

The self-managed troubleshooting note keeps its feature-flag sentence, because
that warning is scoped to v26.37 and earlier, where the gate did apply. It now
reads "On those versions the option is also gated behind ...", so the scope
cannot be misread as current.

v26.38 has shipped, tagged v26.38.0 on 2026-08-19, and its release notes
carry "Self-Managed: Graceful resizing of system clusters". The caveat in Open
question 1 that v26.38.0 final was not out yet no longer applies. The version
numbers in the diff are unchanged and still correct against the tags. The one
wording change is "System clusters resize gracefully starting in v26.38",
which read as a promise about an unreleased version and is now "in v26.38 and
later".

The three code claims were re-checked against current main and all still
hold:

1. The immediate return in src/adapter/src/coord/sequencer/inner/cluster.rs
   is gated only on ENABLE_BACKGROUND_ALTER_CLUSTER, with no reference to the
   wait strategy, and that dyncfg still defaults true
   (src/adapter-types/src/dyncfgs.rs).
2. controller_owns is still hardcoded true in the same file, so the
   foreground wait shim and the direct create/drop branches remain unreachable.
3. The builtin-cluster filter in src/adapter/src/coord/cluster_controller.rs
   is still .filter(|c| c.is_managed()), with no is_user() conjunct.

DEFAULT_CLUSTER_RECONFIGURATION_TIMEOUT is still 24 hours, so the deadline
figure on the page is unchanged.

The SIZE downtime warning in alter_cluster.yml was not fixed upstream,
so that hunk is still needed and is retained.

Re-verified after the merge: the YAML still parses, every in-page anchor still
resolves including the links rendered from the YAML data file, and both
#resizing-in-v2633-and-earlier and #system-clusters exist. The headings
this PR renamed have no inbound links from elsewhere in doc/user/. The only
inbound links to the page target #monitoring-a-resize,
#speed-up-hydration-by-autoscaling-to-a-larger-size and #system-clusters,
all of which resolve.

---

Generated by Claude Code