Skip to content

Two fixes to column_fractional_z_index - #692

Open
ewquon wants to merge 7 commits into
mainfrom
eq/lcc-plvd-fractional-indices
Open

ewquon wants to merge 7 commits into
mainfrom
eq/lcc-plvd-fractional-indices

Conversation

@ewquon

@ewquon ewquon commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

Two fixes under column_fractional_z_index: which column it bisects, and the dispatch that reaches it. The second closes #508 and is what a conformal-conic parent needs in order to get its own column's vertical; the first is a pre-existing bug found while writing the second.

1. Pick the column by nearest node, not by truncation

column_fractional_z_index chose the column to bisect with Base.unsafe_trunc, which floors. Its ii/jj are 1-based node indices, so flooring takes the column west and/or south of the query whenever the fractional part is at least a half.

The sampled height is then off by however much that level's height surface tilts across one parent cell. Measured on an ERA5 0.25° parent over the Southern Great Plains (45×39×37, terrain 32–1518 m, 2016-09-10 12Z), queried from a 1/24° child at 12 heights above ground — 592,860 points, each interpolating the parent's own height field through the full stencil, so (sampled − requested) is the error this function contributes:

floor nearest
mean height error, all queries −1.424 m +0.153 m
rms height error, all queries 3.242 m 2.338 m
mean / rms, ≥ 500 m AGL −1.915 / 3.016 m −0.010 / 1.515 m
mean / rms, ≤ 250 m AGL −0.442 / 3.651 m +0.479 / 3.437 m

74.5 % of queries change column. What the fix buys is removing the bias: flooring always picks west and south, so its error carries a sign, and over a nested run that is a systematic displacement of the parent's height field rather than noise that averages out.

Across all 592,860 queries the largest temperature change is 0.151 K and the largest wind change 0.402 m/s, and nothing below 250 m AGL moves more than 7 m in height.

The residual rms does not go to zero and cannot. Its reference is the height solved on the bilinearly blended column, so what remains is the cost of picking one column at all: _interpolate blends four columns for the field value while column_fractional_z_index bisects a single one, which makes the height surface zeroth-order where the field on it is first-order. Going first-order would mean bisecting a blended profile, and that sits on the hot path for every boundary getbc, relaxation target and child initial condition — worth its own issue rather than this one.

A second path reaches the same defect, and is what the lat-lon regression test exercises: where the grid spacing is not a binary fraction, a query placed exactly on a node round-trips a few ulp below the integer and floors a whole column down.

2. Compose the conformal-conic horizontal with the per-column vertical

Closes #508.

PressureLevelVerticalDiscretization specializes _fractional_indices on the vertical; LambertConformalConicGrid specializes it on the horizontal, because its projected axes are not separable and cannot route through fractional_x_index/fractional_y_index. On a grid that is both, the Lambert method is more specific and wins outright — no ambiguity, no error — and the vertical falls back to the generic fractional_z_index, which reads the column-mean height profile.

Forcing the two apart on an 8×6×5 grid whose column i spans 100i to 100i + 1600 m, so that the domain-mean profile (450–2050 m) matches no column. k is the fractional level index _fractional_indices returns, on 5 levels spaced 400 m apart, so k = 1.5 sits halfway between a column's first and second level rather than a fraction of its depth:

column that column spans query height k before k after exact k
i = 1 100–1700 m 300 m 0.625 1.500 1.500
i = 3 300–1900 m 1000 m 2.375 2.750 2.750
i = 8 800–2400 m 1000 m 2.375 1.500 1.500

The two 1000 m rows are the tell: before, columns 3 and 8 return the same index although their heights differ by 500 m, because both read the domain mean rather than their own column. Row 1 shows the second hazard — k = 0.625 is below level 1, underground, and nothing clamps it; a query above a column's top runs past k > Nz the same way. column_fractional_z_index clamps both ends.

This is not specific to conformal conic grids: any grid that specializes _fractional_indices jointly on the horizontal defeats a vertical specialization the same way. A horizontal_fractional_indices hook upstream would let the two compose rather than compete, and the joint method here is the local fix. It imports lcc_fractional_indices, which Oceananigans does not export; reimplementing it from the exported lcc_forward would duplicate the Center/Face node offsets.

Verification

On Oceananigans 0.114.0 and Julia 1.12.7:

  • test/test_pressure_level_grid.jl passes 1466/1466 on CPU under --check-bounds=yes, including two new testsets. The lat-lon one fails without fix 1; the Lambert one fails without fix 2.
  • No new method ambiguities. Pairwise Base.isambiguous over all 25 _fractional_indices methods finds 8, every one of them pre-existing and none involving the new method. The Union{Center, Face} restriction is load-bearing: dropping it adds eight more.
  • Type-stable and allocation-free — @inferred concrete, @allocated 0 — and it runs inside a KernelAbstractions kernel on the CPU backend.
  • Unaffected paths dispatch where they did before: plain LatitudeLongitudeGrid and plain LambertConformalConicGrid reach Oceananigans' own methods, and LatitudeLongitudeGrid + PressureLevelVerticalDiscretization reaches the existing method in this file.

Compatibility

No API change: no signatures or exports move, and the new _fractional_indices method is additive.

Results do change for every PressureLevelVerticalDiscretization parent, by the amounts in §1 — at most 0.151 K and 0.402 m/s on the grid measured there. Nested runs will not reproduce bit-for-bit across this commit. Flagging rather than deciding: whether that warrants a minor bump or ships as a fix is a maintainer call.

ewquon and others added 2 commits September 17, 2026 10:22
`column_fractional_z_index` chose which column to bisect with `Base.unsafe_trunc`,
which floors. Its `ii`/`jj` are 1-based node indices, so flooring takes the column
west and south of the query whenever the fractional part exceeds a half — half of
all queries per axis, and 74.5% for at least one axis, measured on a real parent.

That is a half-cell offset, so the sampled height is displaced by roughly half the
column-to-column height difference. Measured against ERA5 0.25 degree over the
Southern Great Plains (45x38x37, terrain 32-1518 m, 906612 queries at 12 heights),
binned by the adjacent-column height difference:

    |dZ| (m/cell)   median |dh|    p95     ratio
    0-1             0.31 m         1.19    0.53
    2-4             1.68           4.35    0.60
    8-16            4.75          13.48    0.44

Aloft the effect is small but biased: floor gives a mean height error of -1.53 m
where nearest gives +0.02 m, and the rms against an exactly inverted reference
halves, 2.63 -> 1.41 m. Temperature and wind move by 0.02 K and 0.05 m/s at p95,
which is not forecast-relevant.

Below 250 m AGL it is not small. There the index is clamped to the column's first
above-ground level, and in ~9% of columns — concentrated over steeper terrain —
the two choices clamp one ERA5 level apart. The sampled height then jumps a whole
layer, 200-240 m, carrying temperature up to 2-3 K and meridional wind up to
7.9 m/s. Domain-wide near-surface temperature rms difference is 0.38 K at 10 m AGL.
A nested child's initial and boundary state over terrain is sensitive to this line,
so runs predating it will not reproduce bit-for-bit near the surface.

Flooring is also inconsistent with what follows: `interpolator` turns ii = 2.7 into
the stencil (2, 3, 0.7), i.e. mostly column 3, while the vertical came from column
2. `column_fractional_z_index` picks a representative column rather than a stencil
corner, so nearest is the right choice.

A second, rarer path reaches the same defect. Where the grid spacing is not a
binary fraction, a query placed exactly on a node round-trips a few ulp below the
integer and floors a whole column down — 0/45 nodes on ERA5's 0.25 degree grid,
but 44/110 on a 0.1 degree grid. That is what the regression test exercises.

Not addressed here, and larger than either version's error: near the surface both
choices sample ~90 m above the query, because the first above-ground ERA5 level
sits ~100 m above terrain on average. No column choice removes that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RpzYqdp9fuGrbRvfXNvJ52
Closes #508.

`PressureLevelVerticalDiscretization` specializes `_fractional_indices` on the
vertical; `LambertConformalConicGrid` specializes it on the horizontal, because
its projected axes are not separable and so cannot route through
`fractional_x_index`/`fractional_y_index` the way a `LatitudeLongitudeGrid` does.
On a grid that is both, the Lambert method is the more specific and wins outright
— no ambiguity, no error — and the vertical falls back to the generic
`fractional_z_index`, which reads the column-*mean* height profile.

Forcing the two paths apart with `invoke` on an 8x6x5 grid whose column `i` spans
100i to 100i + 1600 m:

    i  z        before   after   exact
    1  300.0    0.625    1.5     1.5
    3  1000.0   2.375    2.75    2.75
    8  1000.0   2.375    1.5     1.5

Before, every column returns the same answer, because every column is reading the
domain mean. The index is also unclamped in both directions — 0.625 is below the
surface and a query above the column top gives k > Nz — which is what
`column_fractional_z_index` clamps against, and what the comment there records as
having NaN'd a terrain-following initial state on the GPU.

This is not specific to Lambert conformal. Any grid that specializes
`_fractional_indices` jointly on the horizontal defeats a vertical specialization
the same way. A `horizontal_fractional_indices` hook upstream would let the two
compose rather than compete; the joint method here is the local fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RpzYqdp9fuGrbRvfXNvJ52
@codecov

codecov Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...rc/Grids/pressure_level_vertical_discretization.jl 87.50% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ewquon ewquon changed the title Pick pressure-level columns by nearest node, and compose the Lambert-conformal horizontal with the per-column vertical Two fixes to column_fractional_z_index Sep 17, 2026
ewquon and others added 2 commits October 21, 2026 12:19
`column_fractional_z_index` was rewritten on main by #580, which dropped
`clip_subsurface!`, `first_above_surface_level` and the hand-rolled
`index_binary_search` blend in favor of Oceananigans' `fractional_index`:
sub-surface pressure levels now interpolate across the surface rather than
clamping above the first above-ground level. Take main's body whole. This
branch's only edit to that function is the two `nearest_column_index` calls,
which choose the column, and are orthogonal to how the column is bisected.

Verified against Oceananigans 0.114.0: test/test_pressure_level_grid.jl passes
1466/1466 on CPU. Reverting the two calls to `Base.unsafe_trunc` fails 6 of
them; removing the joint `_fractional_indices` method fails 6. Both halves of
the branch still bite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rule 7 of the restraint rules: a comment describes the code that is there. The
import TODO was a design memo about Oceananigans' export list; the method
comment restated Oceananigans' `_fractional_indices` and argued for this one;
the `Face`-location note described `column_fractional_z_index`. All three move
to the pull request description. What is left names the dispatch position and
the rounding idiom, which the code does not say on its own.

Also fold the test file's second `Oceananigans.Fields` import into the first
and separate the final testset from the one above it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ewquon
ewquon requested a review from glwagner October 21, 2026 20:02
Comment on lines +293 to +294
const LCCPressureLevelGrid =
LambertConformalConicGrid{<:Any, <:Any, <:Any, <:Any, <:PressureLevelVerticalDiscretization}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const LCCPressureLevelGrid =
LambertConformalConicGrid{<:Any, <:Any, <:Any, <:Any, <:PressureLevelVerticalDiscretization}
const LCCPressureLevelGrid = LambertConformalConicGrid{<:Any, <:Any, <:Any, <:Any, <:PressureLevelVerticalDiscretization}

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Oceananigans' LambertConformalConicGrid defeats PressureLevelGrid's vertical fractional index

3 participants