Skip to content

[default values] Refuse a column default that cannot be applied, even with the policy bypass - #917

Open
cbb330 wants to merge 2 commits into
mainfrom
chbush/bypass-unusable-defaults-ahv5f
Open

cbb330 wants to merge 2 commits into
mainfrom
chbush/bypass-unusable-defaults-ahv5f

Conversation

@cbb330

@cbb330 cbb330 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

With the column-default policy bypass, a commit could store a column default that cannot be applied. Every later load of the table then failed.

The bypass header, X-OpenHouse-Dangerously-Bypass-Column-Default-Policy: true, skipped every column-default check. That included asking the deployment's ColumnDefaultsSource for the defaults the commit declares. A source throws ColumnDefaultException when it cannot produce them, for example from a truncated Avro schema. Without the bypass, the server refuses that commit with a 400. With the bypass, the server stored it. Every load computes the table's defaults, so every later load failed with 500, with or without the bypass. No client could read, write or repair the table.

SETUP.md already says default resolution still applies under the bypass. Now the code matches it. On a bypass commit, ColumnDefaultCommitPolicy now calls ReadBridgeStripProtection.declaredColumnDefaults, the check enforce runs on every other commit. It refuses the commit with the same 400 as without the bypass. Nothing is stored, so the table keeps loading. The bypass still skips the opt-in lock, Type 1, Type 2 and the lookup of the table's current defaults.

A bypass commit to a table whose stored default already cannot be applied must now fix the default or remove the opt-in. Before, the bypass accepted any other commit too, such as a rename. Now the server refuses it, because the table would still declare the default. A table stored before this fix still fails every load, and needs an operator.

#911 adds a check that each default fits every schema version of its column. The bypass skips that check too. Whichever of the two PRs merges second runs it on bypass commits too.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.

  • Added new tests for the changes made.

  • Updated existing tests to reflect the changes made.

  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.

  • Some other form of testing like staging or soak time in production. Please explain.

  • ReadBridgeColumnDefaultE2ETest.put_failsAsBadRequestWhenRequestDeclaresUnusableDefault: with the bypass, a PUT whose table declares a default the stub source cannot apply now fails with 400 COLUMN_DEFAULT_UNUSABLE. The table still loads with its default US. On main, the same PUT returned 200.

  • ReadBridgeColumnDefaultE2ETest.rename_isCheckedLikeEveryCommit: with the bypass, a rename of a table whose stored default cannot be applied now fails with 400. On main, it returned 204.

  • With JDK 17: ReadBridgeColumnDefaultE2ETest (14), RepositoryTest (71), ColumnDefaultCommitPolicyTest (21), ReadBridgeStripProtectionTest (22), ReadBridgeConfigResolverTest (8) and ClientCapabilityGateTest (5) pass, and the pre-commit Spotless check passes.

  • A throwaway Spark 3.1 itest, not committed. Its ColumnDefaultsSource threw ColumnDefaultException for a table with a marker property. On an opted-in table with the row [1, CA], a catalog that sends the bypass header added the marker with ALTER TABLE ... SET TBLPROPERTIES.

    • On this PR, the ALTER fails with 400 COLUMN_DEFAULT_UNUSABLE. The table does not have the marker, reads [1, CA] with and without the bypass, and drops.
    • On main, the ALTER fails with 500 COLUMN_DEFAULT_UNUSABLE, but the commit is stored: a later read through the bypass catalog fails with the same 500.

Additional Information

  • Breaking Changes: with the bypass, a commit that declares a column default that cannot be applied was accepted, and is now refused with a 400.
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

… with the policy bypass

With X-OpenHouse-Dangerously-Bypass-Column-Default-Policy, ColumnDefaultCommitPolicy
skipped every column-default check, including the check that the
defaults a commit declares can be applied. So an operator could store a
table whose defaults cannot be applied, such as one with a truncated
avro.schema.literal. Every load computes the table's column defaults, so
every later load failed with 500, with or without the bypass, and no
client could read, write, repair or drop the table.

The bypass still skips the opt-in lock, Type 1 and Type 2, but
ReadBridgeStripProtection.rejectUnusableDefaults refuses a commit that
declares a default that cannot be applied, with the same 400 as without
the bypass. Nothing is stored, so the table keeps loading.

On a table whose stored default already cannot be applied, a bypass
commit must now fix that default or remove the opt-in. Any other commit,
such as a rename, leaves a table that still declares the default, and is
refused.
enforce already turns a declared default that cannot be applied into a 400. The bypass branch now calls that method instead of a second one that repeated its opt-in test and table build.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant