[iceberg] Add metadata.iceberg.sync-full-history to rebuild full history

[vbabenkoru]

Purpose

When Iceberg metadata is created from scratch, only the latest Paimon snapshot was available in Iceberg. This happens when Iceberg compatibility is enabled on a table that already has snapshots, or when the old metadata can no longer be used. Time travel and tags were lost (#6107).

This PR has two commits.

1. metadata.iceberg.sync-full-history (opt-in, default false): This option replays the full retained Paimon history. It creates new metadata for the earliest retained snapshot. It then applies each later snapshot on top of the previous one. This makes schemas, tags, and the row-id space for format version 3 build up in the same way as live commits.

Design points:

• Resumable. Each replay step saves its metadata file. If a rebuild stops, the next commit continues from the newest metadata already written. The resume candidate is checked first. It must include the start of the retained history. Otherwise, metadata left by a normal single-snapshot rebuild could silently cut off the replayed history.
• Single external transition. Intermediate steps do not update the version hint, commit to the external catalog, or clean up. Only the final step publishes the result. This means an external catalog sees one change and never points to files deleted by an intermediate step.
• Correct time travel. Replayed snapshots keep the original Paimon commit timestamps. Tags that point to any replayed snapshot become Iceberg refs.
• Row-id safety. The rollback and self-heal floors from [iceberg] Emit row-lineage metadata fields for Iceberg format version 3 #9244/[iceberg] Assign manifest-level row lineage for Iceberg format version 3 #9245 (inheritUuid, lastColumnIdFloor, nextRowIdFloor) are passed through every replay step. This prevents a rebuild from reusing row ids or column ids that abandoned metadata already assigned.
• Snapshot retention (snapshot.num-retained.*, snapshot.time-retained) works on replayed history in the same way as on live commits.

2. Keep live-parity files when creating metadata from scratch. Creating metadata from scratch used to drop every DataSplit that is not raw-convertible. For primary key tables, this removed entire buckets when they contained level-0 files or overlapping key ranges. This silently lost rows that the incremental commit path would have published. A full-history replay then included that loss in every replayed snapshot. Files are now collected one file at a time instead of one split at a time. Raw-convertible splits keep their exact export. Non-raw-convertible splits add every file that the incremental path would accept, along with its deletion vector. Only unmerged level-0 data is excluded. These files are counted, and a warning recommends full compaction as the fix.

Tests

• IcebergSyncFullHistoryTest (paimon-core): checks that the default still exposes only the latest snapshot; checks a full replay of retained snapshots with schema changes and a tag in the middle of the history that an Iceberg client can read; checks that an interrupted replay resumes from the newest metadata; checks that a resume candidate without the retained history prefix is rejected; checks that changing the format version rebuilds history while keeping v3 row lineage correct.
• IcebergBootstrapNonRawSplitsTest (paimon-core): checks that creation from scratch exports compacted files from non-raw-convertible splits; checks full-history replay with non-raw splits.
• IcebergFullHistoryCompatibilityTest (paimon-iceberg): checks that enabling the option on an existing v3 DV table rebuilds history correctly with the Iceberg 1.8/1.11 readers; checks that an uncompacted DV bucket exports its compacted files.
• The full paimon-iceberg test suite passes on both configurations: JDK 11 / Iceberg 1.8.1 and JDK 17 / -Piceberg-ga (Iceberg 1.11).

API and Format

Adds the optional table option metadata.iceberg.sync-full-history with a default of false. The default behavior stays the same, except for the live-parity fix in commit 2. That fix only adds files that the incremental commit path would already publish.

Documentation

The option is documented through its description in the generated configuration docs.

---

AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.

[JingsongLi]

• rebuildFullHistory calls expireAllBefore at IcebergCommitCallback:544 before the new metadata is fully constructed and published. This method deletes the manifest/list files referenced by the old metadata (lines 1853-1880), even though the Hive/REST catalog might still point to that old metadata. During the reconstruction process—or if it fails midway—external readers could obtain metadata with deleted dependencies, thereby violating the "single external transition" guarantee claimed in the PR.
• The "self-healing for corrupted metadata" path is effectively non-functional: while a candidate read operation catches the exception at lines 519-533 and initiates a rebuild from scratch, the subsequent call to expireAllBefore at line 1861 attempts to parse the same corrupted file without safeguards, causing the rebuild to fail immediately.