CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits - #5128
Open
pmcfadin wants to merge 10 commits into
Open
CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits#5128pmcfadin wants to merge 10 commits into
pmcfadin wants to merge 10 commits into
Conversation
Keyspaces.diff and Tables.diff built created and dropped with filter(), costing one BTreeMap removal per table in the cluster on every diff. Collect the differences directly, and skip comparing tables carried over by reference. patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Assisted-by: Claude Opus 5 <noreply@anthropic.com>
…ollection Tables rebuilt three ImmutableMaps on every mutation, copying all N tables per CREATE TABLE. Back them with BTreeMap and maintain index tables incrementally alongside their base table. Requires a clean build: Tables.indexTables() narrows from ImmutableMap to Map, which ant's incremental javac will not pick up in unchanged callers. patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Measures Keyspaces.diff, the three Tables mutations, and
DistributedSchema.withLastModified at 400 and 3200 tables in one keyspace.
The metric of interest is gc.alloc.rate.norm (bytes/op) under -prof gc: the
failure mode these tickets address is a GC wall, not a gradual slowdown.
ant microbench -Dbenchmark.name=SchemaChangeBench \
-Djmh.args="-prof gc -f 1 -wi 3 -i 5 -r 1 -w 1"
patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…uction DistributedSchema built a flat TableId map over every table in every keyspace, twice per DDL, duplicating the one Keyspaces already maintains. Delegate to it instead. patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Assisted-by: Claude Opus 5 <noreply@anthropic.com>
…very schema change SchemaKeyspace.applyChanges previously called a blocking, synchronous flush of every system_schema table on every schema change (SchemaKeyspace.flush(), called from the log follower for every schema-changing epoch). Under a burst of DDL this serialises all schema changes behind disk I/O: measured 92.4ms of SchemaKeyspace.flush -> waitOnFuture per CREATE TABLE, a 3.65x slowdown end-to-end versus the flush disabled. system_schema has durable_writes=true, so mutations are already in the commitlog (and, under TCM, the TCM log) before any flush runs; the flush is belt-and-braces, not what makes the change durable. Add cassandra.schema_flush_coalesce_ms (CassandraRelevantProperties SCHEMA_FLUSH_COALESCE_MS, default 1000): -1 legacy synchronous flush on every schema change 0 asynchronous, flush scheduled immediately >0 asynchronous, at most one flush per that many milliseconds (coalesced) SchemaKeyspace.flush() is renamed to flushBlocking() (public, unchanged body) and applyChanges now calls scheduleFlush(), which either flushes synchronously (coalesce < 0) or schedules a coalesced flush on ScheduledExecutors.nonPeriodicTasks, guarded by an AtomicBoolean so a burst of DDL statements pays for one flush instead of one per statement. The scheduled task resets the flag before flushing (so schema changes arriving mid-flush schedule the next flush rather than being silently folded into it), is wrapped in try/catch logging at WARN, and skips when DatabaseDescriptor.isUnsafeSystem(). StorageService.drain() already flushes every local-strategy keyspace (Keyspace.system(), which includes system_schema per SchemaConstants.LOCAL_SYSTEM_KEYSPACE_NAMES) synchronously before shutting down the commitlog; a comment was added there rather than a second flush call, since coalescing/deferring the schema flush does not change drain's existing guarantee. Tests: SchemaFlushCoalesceTest exercises scheduleFlush() directly (package- private, @VisibleForTesting) for coalesce=-1/0/N, independent of the cassandra.test.flush_local_schema_changes gate that unit tests run under. SchemaFlushRestartTest (in-JVM dtest) creates 30 tables under a 60s coalesce window, kills the node without a drain, restarts, and confirms all 30 tables and their data survive via commitlog/TCM-log replay alone. patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ting tables Three items: 1. Keyspaces.withAddedOrUpdated computed a delta by without(name) then with(keyspace), which walked every table and view of the keyspace to remove it from the by-TableId BTreeMap and then re-added every one (withoutKsTablesViews / withTablesViews), ~2N BTreeMap mutations per call. It now computes the delta directly: existing tables/views whose instance is unchanged (reference identity, same argument KeyspacesDiff.diff relies on) are left alone; only added, removed or replaced entries touch the map. Measured at N=400 vs N=3200 tables in one keyspace (KeyspacesDiffScalingTest): before, 307,088 B vs 3,808,408 B (12.4x, fails); after, 1,288 B vs 1,656 B (1.3x). 2. ThreadLocalMeter.allocateRateGroupOffset grew the static `rates` array by exactly RATES_COUNT (3) slots per new meter, copying the whole array on every growth once the initial 16-meter capacity was exhausted - quadratic total copying in meter count. Growth is now geometric (max(needed, length + length/2)), mirroring ThreadLocalMetrics.calculateNewCapacity. Measured via a new @VisibleForTesting reallocation counter: growing from a warm array to accommodate 16,000 more meters took 12,335 reallocations under fixed growth vs 4 under geometric growth. 3. DistributedSchema.validate() walks every table/view/type/function of every keyspace on every construction. Added an overload that takes the previous DistributedSchema and skips keyspaces whose KeyspaceMetadata instance is unchanged (already validated when `previous` was built), without weakening any check for keyspaces that are new or did change. Wired into the two per-DDL construction sites that already have the previous schema at hand and previously always fully re-validated: AlterSchema.execute's construction of snapshotAfter, and ClusterMetadata's deduplicateReplicationParams (reached from every ClusterMetadata deserialize). Tests: extended KeyspacesDiffScalingTest with a withAddedOrUpdated scaling test plus correctness guards (added table findable by id/name, removed table gone, replaced table resolves to the new instance, views by their own id, other keyspaces untouched); added a ThreadLocalMeter reallocation-count test at M=2,000 and M=16,000. Both scaling assertions were verified to fail against the prior implementation before being restored. patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
MetadataSnapshots.SystemKeyspaceMetadataSnapshots.storeSnapshot writes the entire serialised ClusterMetadata as a single mutation via SystemKeyspace.storeSnapshot. Past max_mutation_size (default 16 MiB) that write throws MutationExceededMaxSizeException, which MetadataSnapshotListener.notify only logged at WARN with no metric -- so production clusters past the ceiling silently stopped snapshotting with nothing but a stack trace in the logs. - storeSnapshot now compares the serialised size against DatabaseDescriptor.getMaxMutationSize() and, once it reaches 75% of the limit (SNAPSHOT_SIZE_WARNING_THRESHOLD), logs a rate-limited (5 min, NoSpamLogger) WARN with the serialised size, the limit, and the current table count from metadata.schema, before the write can start failing. - MetadataSnapshotListener.notify now logs the catch at ERROR (still swallowing the exception so the log-processing thread survives) with the transformation kind and epoch, and increments the new TCMMetrics.snapshotStoreFailures counter. TCMMetrics also gained a lastSnapshotSize gauge, updated on every attempt regardless of outcome. - Measured serialised size (test/unit/org/apache/cassandra/tcm/ MetadataSnapshotSizeWarningTest.java): a (k int PRIMARY KEY, v text) table serialises at ~527 bytes as part of a ClusterMetadata, so 5,000 such tables (~2.6 MB) reliably exceeds the 2,621,440 byte max_mutation_size that test/conf/cassandra.yaml's 5MiB commitlog_segment_size produces for this test JVM. patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit added cassandra.schema_flush_coalesce_ms (a system
property) with three modes: -1 synchronous, 0 async-immediate, >0
coalesced. Promote it to a proper cassandra.yaml setting and drop the
async-immediate mode: it measured slower than coalescing (45s vs 33s
at 1,000 tables in the in-JVM scaling test), so there is no reason to
keep it as a distinct option.
Add Config.schema_flush_coalescing_window (DurationSpec.IntMillisecondsBound,
default 1000ms) alongside a DatabaseDescriptor getter/setter, following
the pattern used for the neighbouring DurationSpec settings (e.g.
hints_flush_period). Semantics: 0ms flushes synchronously on every
schema change (the pre-patch behaviour); any positive value flushes
asynchronously, coalescing concurrent schema changes into at most one
flush per window. SchemaKeyspace.scheduleFlush() now reads
DatabaseDescriptor instead of the removed CassandraRelevantProperties
entry; flushBlocking() and the drain comment in StorageService are
unchanged.
Document the setting in cassandra.yaml next to the tables guardrail
thresholds, since it is about the same schema-DDL cost, including the
durability rationale (system_schema is durable through the commitlog
and rebuilt from the cluster metadata log on startup, so delaying the
flush does not risk schema loss) and the measured effect.
SchemaFlushCoalesceTest now drives DatabaseDescriptor's setter directly
instead of the property, and drops the async-immediate test case in
favor of a synchronous (0ms) test case. SchemaFlushRestartTest sets the
window through the instance's yaml config
(c.set("schema_flush_coalescing_window", "60s")) instead of a system
property override.
patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
pmcfadin
force-pushed
the
pmcfadin/schema-at-scale
branch
3 times, most recently
from
September 8, 2026 03:41
c4f05a4 to
d85584a
Compare
…mits Set tables_warn_threshold's default to 1000 (Config.java), so operators get a warning as their table count grows instead of silently paying an increasing per-table heap and per-schema-change cost. Leave tables_fail_threshold at its -1 (disabled) default: a positive fail default could break clusters that are already above whatever value was chosen, on upgrade. Update the commented default in cassandra.yaml and cassandra_latest.yaml, and the nodetool GuardrailsConfigCommandsTest and GuardrailsOptionsTest expectations that assumed the old -1 default; GuardrailsOptionsTest's testZeroThresholdsAreAccepted also now sets tables_warn_threshold explicitly, since the new non-disabled default otherwise trips the warn-must-be-<=-fail validation added for zero thresholds. Add a "Schema Size" section to the hardware/capacity-planning doc page (doc/modules/cassandra/pages/managing/operating/hardware.adoc): the tested envelope (~10,000 tables), the per-table heap cost (~290KiB, mostly metrics), the O(epoch) cost of schema changes, the new warn guardrail default, and the schema_flush_coalescing_window setting. NEWS.txt updated for the new schema_flush_coalescing_window setting and this commit's guardrail default change. patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21664 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
pmcfadin
force-pushed
the
pmcfadin/schema-at-scale
branch
from
September 8, 2026 05:03
d85584a to
51d5c63
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CASSANDRA-21664
Schema changes cost time in proportion to the number of tables that already exist, and each one blocks on a disk flush. Creating 10,000 tables takes 28 minutes.
What
Per
CREATE TABLEat small N, about 121 ms: 92 ms is a synchronous flush of everysystem_schematable inSchemaKeyspace.applyChanges→flush(), on the log follower, and 22 ms is the blocking TCM log flush. Beyond that fixed cost, every schema change diffs the whole schema and rebuilds whole-keyspace maps, so the per-statement cost climbs from 112 ms to 234 ms across the first 10,000 tables.Separately, once the serialized cluster metadata exceeds
max_mutation_size(about 31,000 minimal tables at the 16 MiB default),SystemKeyspace.storeSnapshotthrowsMutationExceededMaxSizeException,MetadataSnapshotListenerlogs a WARN, and snapshots stop being written with nothing else to tell an operator.Change
Keyspaces.diff,Tables(persistent map), theDistributedSchematable map,Keyspaces.withAddedOrUpdated,DistributedSchema.validate, andThreadLocalMeterarray growth.system_schemaflush is coalesced behind a new setting,schema_flush_coalescing_window(default1000ms;0msrestores the synchronous flush).system_schemahasdurable_writesand is rebuilt from the cluster metadata log on startup, so delaying the flush cannot lose schema. Drain still flushes synchronously.TCM.SnapshotStoreFailuresmetric, and a WARN fires once serialized metadata passes 75% ofmax_mutation_size.tables_warn_thresholddefaults to 1000. The tested envelope and per-table heap cost are documented.No serialized-format or API changes.
Result
ManyTablesScalingTest, single node in-JVM, same machine, 16 GiB heap. Before is the synchronous flush; after is this branch.Per statement at 10,000 tables: 234 ms before, 95 ms after. The fixed floor is gone; about 6 ms per 1,000 existing tables remains, from
TablesDiffscanning both sides in full. That is a follow-up.Scope
The flush default is the coalesced mode. Whether that ships as default or opt-in is with dev@ ("[DISCUSS] Making the synchronous system_schema flush on every DDL async"); either is a one-line change to the yaml default.
Not included: chunking the snapshot across rows, a bulk (many statements per epoch) schema transformation, and the per-table memory footprint.
Tests
Scaling assertions at N vs 8N on every touched path (
KeyspacesDiffScalingTest,TablesScalingTest,DistributedSchemaScalingTest,ThreadLocalMeterTest), each verified red before the change.SchemaFlushCoalesceTest;SchemaFlushRestartTest(30 tables, stop without drain, restart, all present);MetadataSnapshotSizeWarningTest. Regression: all 30org.apache.cassandra.schematest classes, thetcm, guardrail andDatabaseDescriptorsuites, 0 failures.ant checkstyleandant checkstyle-testclean.Parts of this patch were produced with AI assistance and reviewed by the author.