[iceberg] Add opt-out of REST catalog drop+recreate recovery - #9349
Draft
vbabenkoru wants to merge 4 commits into
Draft
[iceberg] Add opt-out of REST catalog drop+recreate recovery#9349vbabenkoru wants to merge 4 commits into
vbabenkoru wants to merge 4 commits into
Conversation
When Iceberg metadata is created from scratch (compatibility enabled on a table that already has snapshots, or the previous metadata is unusable), only the latest Paimon snapshot was exposed to Iceberg, losing time travel and tags (apache#6107). With the new opt-in option the whole retained Paimon history is replayed instead: metadata is created afresh for the earliest retained snapshot, then every following snapshot is applied on top of its predecessor, so schemas, tags and the v3 row-id space accumulate exactly like live commits. Each step persists its metadata file, making an interrupted rebuild resumable; a resume base is first validated to cover the retained history prefix, so single-snapshot metadata from a plain rebuild never truncates the replayed history. Intermediate steps skip the version hint, the external catalog commit and cleanup, which only the final step performs. The rollback and self-heal floors (inherited table uuid, last-column-id and next-row-id) are threaded through every replay step so a rebuild never reuses ids handed out by abandoned metadata. Replayed snapshots stay subject to the snapshot retention policy exactly like live commits.
…scratch Creating metadata from scratch dropped every DataSplit that is not rawConvertible. For primary key tables this discards whole buckets whenever they contain level-0 files or overlapping key ranges, silently losing rows that the incremental commit path would have published (files above level 0 with their deletion vectors, or max-level files), and a full-history replay bakes the loss into every replayed snapshot. Collect files per file instead of per split: raw-convertible splits keep their exact export, and non-raw-convertible splits now contribute every file that shouldAddFileToIceberg accepts, together with its deletion vector. Files that genuinely cannot be read without merging (unmerged level-0 data) are counted and reported in a warning with exact file and row counts, pointing at full compaction as the immediate remedy.
vbabenkoru
marked this pull request as draft
August 21, 2026 20:00
A from-scratch full-history rebuild deleted the old build's manifest lists and manifests up front, so an external catalog still pointing at the old metadata could resolve files that no longer existed while the replay was running, or forever if it failed midway. The rebuild also could not recover from a corrupted metadata file: the resume scan tolerated it, but the cleanup walk parsed the same file unguarded and failed the commit. The up-front deletion is replaced by a tolerant collection of the old build's file names; each replay step removes only its own target file just before writing the replacement (a leftover from a regenerated build can match the step's commit identity while carrying other content), and the collected files are deleted only after the final step has published, skipping anything the replayed chain references. The expiration walk and the with-base read of a base metadata file now tolerate unreadable files as well, so a corrupted base self-heals like a structurally invalid one instead of failing every commit. Reported by JingsongLi in review.
…8875) The REST metadata committer treated any catalog state that was not exactly one snapshot behind as an invalid base and recovered by dropping and recreating the catalog table. On AWS Glue this requires glue:DeleteTable, replaces the table's identity for downstream consumers (Glue, Snowflake), and turns a transient publication failure into a destructive operation: the local metadata file is written before the catalog commit, so an ambiguous failure (CommitStateUnknown) leaves the catalog behind while retries skipped publication entirely because the file already existed. With metadata.iceberg.rest-auto-recreate=false (default true keeps the legacy behavior) the committer now reconciles the catalog in place: * A commit whose metadata file already exists re-drives the REST publication; the committer no-ops when the catalog already holds the intended snapshot (in every mode - previously this recreated the table just to reproduce identical content). * CommitStateUnknownException is resolved by reloading the catalog: if the commit landed it is treated as success, otherwise it is retried once against the unchanged base before surfacing. * A catalog that is behind is brought up to date by replaying, in order, every retained snapshot it is missing - each Iceberg snapshot is self-contained through its manifest list, so this reproduces exactly what consecutive successful commits would have written. Snapshots the catalog already holds are re-activated by a ref move instead of an add (Iceberg rejects adding an existing snapshot id). * Regenerated local history (same ids, different manifest lists, e.g. after a full-history rebuild) is detected by comparing manifest lists, not just snapshot ids; stale catalog entries are removed, and entries whose sequence numbers were already consumed are skipped with a warning since Iceberg sequence numbers are single-use. * States that cannot be reconciled non-destructively - catalog ahead, or a different snapshot under the id being published - fail with IcebergRestCatalogOutOfSyncException naming both states and the manual remediation. The catalog table is never dropped. Initial table creation, empty-table recovery (apache#6906/apache#8335), field-id mapping, schema deduplication and next-row-id handling are unchanged.
vbabenkoru
force-pushed
the
iceberg-rest-reconcile
branch
from
August 24, 2026 15:09
cb42c7e to
58a1bff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR is stacked on #9348 (the first two commits). Review only the top commit.
The REST metadata committer expects the catalog to be exactly one snapshot behind. It treats any other catalog state as an invalid base. To recover, it drops and recreates the catalog table. Since #9245, it may drop and then re-register the table instead. On AWS Glue, this requires
glue:DeleteTable. It also gives the catalog table a new identity for downstream users such as Glue and Snowflake. This turns a temporary publishing failure into a destructive operation. The local metadata file is written before the catalog commit. An unclear failure (CommitStateUnknownException) can therefore leave the catalog behind. Today, recovery requires dropping the table (#8875).This PR adds
metadata.iceberg.rest-auto-recreate. Its default value istrue, which keeps the current behavior. When this option isfalse, the committer fixes the catalog in place and never drops the catalog table:CommitStateUnknownException), the committer reloads the catalog. If the commit succeeded, it reports success. Otherwise, it retries the same updates once using the unchanged base. If the result is still unclear, it reports the ambiguity.IcebergRestCatalogOutOfSyncException. These states include a catalog that is ahead, a different snapshot stored under the id being published, or a v3 row-id space that the catalog cannot accept. The error names both states and explains the manual fix.Initial table creation, empty-table recovery, the v3 nonzero-watermark registration path from #9245, field-id mapping, schema deduplication, and next-row-id handling are unchanged. The v3 nonzero-watermark registration path remains available only in the default mode.
Tests
Five new tests were added to
IcebergRestMetadataCommitterTest: a behind catalog catches up by replaying multiple snapshots; an already-published snapshot is a no-op in both modes; an ahead catalog fails with the out-of-sync error without dropping the table; regenerated history is found by comparing manifest lists and stale entries are removed; and a retry publishes existing metadata again. The fullpaimon-icebergsuite passes with both configurations: JDK 11 / Iceberg 1.8.1 and JDK 17 /-Piceberg-ga(Iceberg 1.11).API and Format
New optional table option:
metadata.iceberg.rest-auto-recreate(defaulttrue). The default behavior is unchanged, except that the table is no longer recreated when the catalog already contains the commit's snapshot.Documentation
The option is documented in its description, which is used to generate the configuration docs.
AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.