Skip to content

Add a parabolic vertical reconstruction to the FiniteVolume pressure gradient - #636

Draft
xylar wants to merge 12 commits into
E3SM-Project:developfrom
xylar:omega/add-finite-volume-horiz-press-grad-phase2
Draft

xylar wants to merge 12 commits into
E3SM-Project:developfrom
xylar:omega/add-finite-volume-horiz-press-grad-phase2

Conversation

@xylar

@xylar xylar commented Oct 25, 2026 •

Copy link
Copy Markdown

Add a parabolic vertical reconstruction to the FiniteVolume pressure gradient, selected with VerticalReconstruction: ppm. It makes the scheme exact for resting profiles quadratic in pressure, not only linear ones, and it is more accurate than the linear reconstruction on most of the Polaris two-column tests. The linear reconstruction stays the default until a realistic configuration shows the gain is worth the cost. Builds on #635 and the design in #448; only the last six commits are new.


The design is revised here rather than in #448, which records it as it stood when the second-order scheme merged. The first three new commits update PGradHighOrder.md (rendered).

On sigma layers over the seamount, ppm makes the existing instability grow faster, which is one more reason linear stays the default.

The Polaris reference and tests are in E3SM-Project/polaris#902.

Checklist

  • Documentation:

  • Linting

  • Building

    • CMake build does not produce any new warnings from changes in this PR
  • Testing

    aurora, intel, mpich

    • CTests Pass
    • Polaris omega_pr Pass
    • Coupled CIME tests Pass

    aurora, intelgpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass
    • Coupled CIME tests Pass

    chrysalis, intel, openmpi

    • CTests Pass
    • Polaris omega_pr Pass
    • Coupled CIME tests Pass

    pm-cpu, gnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass
    • Coupled CIME tests Pass

    pm-gpu, gnugpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass
    • Coupled CIME tests Pass
  • Provide relevant details in a comment to the PR titled Testing

  • New tests:

    • CTest unit tests for new features have been added per the approved design.
    • Polaris tests for new features have been added per the approved design (and included in a test suite)

Posted by Claude Code on @xylar's behalf. The wording above is AI-authored; please check it accordingly.

🤖 Generated with Claude Code

@xylar

xylar commented Oct 25, 2026

Copy link
Copy Markdown
Author

Testing

Chrysalis, Intel, OpenMPI: CTests 62/62 at 88631ec, including the new parabolic tests in double and single precision, with no new build warnings. With E3SM-Project/polaris#902, run on an earlier build of the same model source, all 10 horiz_press_grad tasks pass with ppm as a third scheme, Omega matching the Polaris parabolic reference to 1.6e-13 m/s², and all 224 centered and linear forward steps are bit-for-bit with #635. omega_pr has not been run.

/lcrc/group/e3sm/ac.xylar/polaris_1.2/chrysalis/test_20261025/fv-ppm
/lcrc/group/e3sm/ac.xylar/polaris_1.2/chrysalis/test_20261025/fv-ppm-seamount

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 25, 2026

Copy link
Copy Markdown
Author

What the parabolic reconstruction does

two-column refinement

Against the quasi-analytic reference, ppm is 2.8 to 3.0 times more accurate than linear where the coordinate tilts the layers, and 1.5 times where surface pressure does, at every resolution. With level layers (temperature_gradient and salinity_gradient, not shown) the two are the same.

resting states, tilted layers

At rest, any tendency is error. With T and S quadratic in pressure, ppm brings it down to 1 to 4% of Centered's, close to the floor both reconstructions reach when T and S are linear in pressure, which comes from the sea-floor anchor rather than the reconstruction. At the steepest tilts linear is sometimes 1.6 times better. On the curved profile ppm is 1.4 to 2.6 times more accurate.

resting states, stepped floor

Over a stepped sea floor with T and S quadratic in pressure, ppm is 74 to 1.5×10⁷ times more accurate than linear. On the curved profile it is up to 960 times more accurate on 128 m layers, but up to 3 times less accurate on 256 m layers, where the profile's curvature jumps inside the bottom layers' window.

seamount

On the z-star seamount, kinetic energy after 6 days is 3.7 times lower with ppm. On sigma layers ppm makes the existing instability grow faster: between 6 hours and 1 day, mean kinetic energy grows at an e-folding rate of 11.6 per day with the exponential profile, against 2.3 with linear, and 7.9 against 2.8 with T and S linear in pressure. These runs use the seamount stratification variants on Polaris's fv-pgf-integration branch, with the finite-volume runs switched to ppm.

Scripts: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/xasaydavis/fv_hpg_ppm_20261025/scripts/


Posted by Claude Code on @xylar's behalf. The figures, analysis and wording above are AI-authored; please check them accordingly.

@vanroekel vanroekel added this to the Feature Freeze milestone Oct 26, 2026
xylar and others added 10 commits October 30, 2026 18:09
Where the two columns' floors are more than a layer apart in pressure, as
with terrain-following layers over a steep slope, compare the shallower
floor with the deeper column's own interface just below it, then carry
the anchor down to the edge's deepest interface with the matched-pressure
integrand. Shifting both floors to their mean pressure with one
first-order expansion left an error that grows with the cube of the
offset and is the same at every depth.

Where the floors are within one layer of each other, as on a flat floor,
the anchor is unchanged. Partial bottom cells usually keep a sloping
floor within one layer too, but not where a thin bottom cell lies
entirely above the deeper column's layer of the same index, so realistic
meshes change slightly at some edges along steep slopes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resting, linear-in-pressure state on terrain-following layers whose
floors are 6.4 layers apart. The host reference mirrors the new anchor
and must reproduce the kernel, the tendency must stay well below the
centered scheme's, and the mean-pressure anchor must fail that gate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The figure's runs predate the anchor treatment for sea floors more than
a layer apart, and the residual the old text referred to is no longer
described under "Over a sloping sea floor".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Choose the unlimited piecewise parabolic method of Colella and
  Woodward: interface values from the polynomial through the
  cumulative integral at the five nearest interfaces, built from
  divided differences of the layer means, and a mean-preserving
  parabola through them.  Short columns take the polynomial through
  all their interfaces.
- At the ends of a column take the coefficients from that
  polynomial's own derivatives at the layer midpoint, so a thin
  bottom cell does not amplify round-off when its reconstruction is
  extrapolated below the floor.
- Record the measurements behind the choice over a parabola fitted to
  three layer means, and what the default two-point rule costs the
  parabolic reconstruction.
- Say what the reconstruction adds to the tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The parabolic reconstruction ships first, on its own with the
  two-cell stencil, then the wide stencil, then the second-order
  equation-of-state expansion; in each, Polaris gains the change
  first, written from the design.
- The second-order expansion is kept unless measurement with the
  parabolic reconstruction and the wide stencil shows it does not
  help.
- The defaults stay linear with the two-cell stencil until a realistic
  configuration shows the gain is worth the cost.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Confirm the parabolic reconstruction in Omega: the unit test, the
  ten horiz_press_grad variants against the Polaris kernel, and the
  seamount for six days.
- On the sigma seamount it makes the instability grow three to five
  times faster; record that where the instability is described.
- Qualify the third-order statement of section 6.7, whose refinement
  holds the displacement at a fixed number of layers, against the
  rates the unit test measures at fixed tilt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
VerticalReconstruction: ppm now selects the unlimited piecewise
parabolic method that the design specifies. Each layer's parabola keeps
the layer mean and takes, at its interfaces, the derivative of the
polynomial through the cumulative integral at the five nearest
interfaces, built from divided differences of the layer means. Short
columns use the polynomial through all their interfaces, and at the ends
of a column the coefficients come from that polynomial's own derivatives,
so a thin bottom cell does not amplify round-off when it is extrapolated
below the floor.

The column scan's three evaluation sites -- the integrand of each edge
layer, the anchor's short integrals and its carry-down -- all go through
one function, reconEval, so they cannot describe different water. The
linear reconstruction is evaluated exactly as before and remains the
default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The estimator on a single column: a quadratic reproduced everywhere,
  beyond the column included; a cubic met at every interface; the
  layer integral of the deviation; columns of one to three layers; and a
  thin bottom cell extrapolated below the floor, against the interface
  form, which this test shows amplifying round-off.
- The matched-pressure integrand zero at every quadrature point on a
  quadratic, with a cubic as the control.
- On the two-column state: the exactness gate on linear and quadratic
  profiles in both precisions, the linear reconstruction as the control,
  the host assembly's fidelity to the kernel, and a cubic converging
  faster than with the linear reconstruction.

The host assembly now evaluates the reconstruction through reconEval,
as the kernel does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The user guide gives VerticalReconstruction's two values and what 'ppm'
makes exact. The developer guide describes how the coefficients are
formed, why the end layers take theirs from the polynomial's own
derivatives, and that every evaluation goes through reconEval, and it
lists the curvature arrays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar
xylar force-pushed the omega/add-finite-volume-horiz-press-grad-phase2 branch from 88631ec to ae76f38 Compare October 31, 2026 20:37
@xylar

xylar commented Oct 31, 2026

Copy link
Copy Markdown
Author

Testing on chrysalis

Tested the merge of #636 (ae76f3830d) into develop (11e7cbc817), with #669, #670, #671 merged in, as a4f1650e91, against Polaris' Omega submodule (11e7cbc817). The baseline ran with Polaris d8d9ed7ca3, without the changes in Polaris 37bc1c47b0 that the PR needs.

Row CTests omega_pr Coupled New build warnings
chrysalis, intel, openmpi 62 of 62 passed 33 of 36 passed, 3 with baseline differences 2 of 2 passed 1
chrysalis, intel, openmpi

Polaris omega_pr suite

  • Baseline workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-d8d9ed7_omega-11e7cbc/omega_pr
  • Baseline build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-d8d9ed7_omega-11e7cbc/build
  • PR build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/build
  • PR workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/omega_pr
  • Baseline Polaris: d8d9ed7ca3edcd1ac222015772c24e3f4761bb68 (1.1.0-609-gd8d9ed7ca-dirty)
  • PR Polaris: 37bc1c47b0dd4a621746795de774144fce36483b (1.1.0-615-g37bc1c47b)
  • Baseline Omega: 11e7cbc8176ba35bb2c660df931955e504124d17 (Omega-v0.1.0-alpha.1-6373-g11e7cbc817)
  • PR Omega: a4f1650e91ad1623aa3dd4ce2a83dcca0d7613a6 (Omega-v0.1.0-alpha.1-6391-ga4f1650e91)
  • Machine: chrysalis
  • Partition: debug
  • Compiler: intel
  • Build type: Release
  • Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/omega_pr/polaris_omega_pr.o1319234
  • Result:
    • Diffs (3 of 36):
    - ocean/column/horiz_press_grad/salinity_gradient
      - no norms (missing file or variable)
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/split_explicit
      - temperature:    l1=5.964e-03  l2=1.165e-04  linf=5.793e-05
      - salinity:       l1=8.831e-04  l2=1.954e-05  linf=1.483e-05
      - normalVelocity: l1=5.787e-03  l2=6.569e-05  linf=1.387e-05
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4
      - normalVelocity: l1=1.883e-01  l2=1.500e-02  linf=7.976e-03
      - temperature:    l1=2.191e-02  l2=2.376e-03  linf=1.486e-03
      - salinity:       l1=2.375e-03  l2=1.800e-04  linf=1.087e-04
Recent commits

Baseline Polaris:

d8d9ed7ca Merge pull request #936 from xylar/pr-test-resume-after-suite
8fcdbe2bb Merge pull request #934 from xylar/document-commit-message-style
84e7190dc Merge pull request #933 from xylar/pr-test-update-main
d561288c0 Merge pull request #932 from xylar/update-omega
87fa5f06c Merge pull request #847 from xylar/make-fv-pgf-default

PR Polaris:

37bc1c47b Run the finite-volume scheme with its parabolic reconstruction too
09a397b0f Run the finite-volume kernel with the parabolic reconstruction
4330c1e2e Add resting-state variants quadratic in pressure
9af03457a Add the parabolic reconstruction to the finite-volume reference
c9282e2ab Record the resting-state ratios the anchor rule moved

Baseline Omega:

11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping
8a1cfdcf93 Merge pull request #624 from andrewdnolan/omega/remove-pyyaml-workarounds
e1c6689d4d Merge pull request #645 from andrewdnolan/omega/fix-division-by-zero
27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1
fd66ecb7d1 Merge pull request #448 from xylar/omega/design/add-higher-order-hpg-design

PR Omega:

a4f1650e91 Test merge of E3SM-Project/Omega#671 (48fecaab7b) for testing
c7bcf4ef11 Test merge of E3SM-Project/Omega#670 (5e7c80513c) for testing
242b8aef1f Test merge of E3SM-Project/Omega#669 (c329de826a) for testing
dcca042f25 Test merge of E3SM-Project/Omega#636 (ae76f3830d) into develop (11e7cbc817)
11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping

CTest unit tests:

  • Machine: chrysalis
  • Compiler: intel
  • Build type: Release
  • Build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/build
  • Omega: a4f1650e91ad1623aa3dd4ce2a83dcca0d7613a6 (Omega-v0.1.0-alpha.1-6391-ga4f1650e91)
  • Result: All tests passed
  • Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/build/ctests.log

Coupled CIME tests

  • Cases: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/coupled
  • CIME environment: /lcrc/soft/climate/e3sm-unified/load_latest_cime_env.sh
  • Result: All tests passed

ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel.pr636-ae76f38

PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel CREATE_NEWCASE
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel XML
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SETUP
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SHAREDLIB_BUILD time=77
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel MODEL_BUILD time=180
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SUBMIT
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel RUN time=1526
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel COMPARE_base_rest
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel MEMLEAK
PASS ERS_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SHORT_TERM_ARCHIVER

PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel.pr636-ae76f38

PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel CREATE_NEWCASE
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel XML
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SETUP
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SHAREDLIB_BUILD time=77
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel MODEL_BUILD time=180
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SUBMIT
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel RUN time=1238
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel COMPARE_base_modpes
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel MEMLEAK
PASS PEM_D.T62_oQU240.COMEGA-IAF.chrysalis_intel SHORT_TERM_ARCHIVER

Build warnings

1 new warnings, 0 in files this PR changes (37 in the PR build, 37 in the baseline build). Those in files the PR changes are marked *.

New warnings
  81: [-Wformat] format specifies type 'unsigned int' but the argument has type 'MPI_Comm' (aka 'ompi_communicator_t *')
  • Baseline build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-d8d9ed7_omega-11e7cbc/build/build_omega.log
  • PR build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/build/build_omega.log

Notes

The two QU240 realistic_global differences have the same norms as in #635's Chrysalis testing (#635 (comment)), so they come from #635's sea-floor anchor fix, which this PR builds on. #635's sigma seamount difference does not appear because omega_pr on Polaris main now runs those tasks with Centered only.

horiz_press_grad/salinity_gradient has no norms only for its 7 new finite_volume_ppm forward steps, which the baseline cannot run (Omega 11e7cbc817 rejects VerticalReconstruction: ppm). Its centered, finite-volume and init steps match the baseline exactly, and the task passes.

The new build warning is a SCORPIO pio_rearr_utils.cpp warning whose file name was split off by interleaved build output; both builds have the same 37 warnings.

Cost of the parabolic reconstruction

QU240 cached_forward runs from the PR build, with each scheme run twice in one job; the two runs agree within 0.3%. Times are means from rank 0's omega.timing.0.

Task Scheme doStep (s) pressureGradTerm (s) doStep vs Centered doStep vs FV linear
rk4 Centered 52.7 0.67
rk4 FV linear 63.6 4.69 +20.9%
rk4 FV ppm 77.0 12.50 +46.1% +20.9%
split_explicit Centered 6.82 0.05
split_explicit FV linear 8.02 0.37 +17.6%
split_explicit FV ppm 9.18 1.02 +34.6% +14.5%

With ppm, pressureGradTerm takes 2.7× as long as with linear. In rk4 it accounts for 7.8 s of the 13.3 s increase in doStep; the rest falls outside Omega's timers, as for FV linear against Centered in #658. Runs: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr636-ae76f38/chrysalis_intel/ppm_perf


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 31, 2026 •

Copy link
Copy Markdown
Author

Testing on aurora

The intel row tested the merge of #636 (ae76f3830d) into develop (11e7cbc817), with #669, #670, #671 merged in, as a4f1650e91, against Polaris' Omega submodule (11e7cbc817). The baseline ran with Polaris d8d9ed7ca3, without the changes in Polaris 37bc1c47b0 that the PR needs.

The intelgpu row was rerun after this PR's test-only fixes, testing the merge of #636 (2e45303eca), with the same PRs merged in, as f2e71b9b4c, against the same baseline.

Row CTests omega_pr Coupled New build warnings
aurora, intel, mpich 62 of 62 passed 33 of 36 passed, 3 with baseline differences 2 of 2 passed 0
aurora, intelgpu, mpich 62 of 62 passed 33 of 36 passed, 3 with baseline differences 2 of 2 passed 0
aurora, intel, mpich

Polaris omega_pr suite

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-d8d9ed7_omega-11e7cbc/omega_pr
  • Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-d8d9ed7_omega-11e7cbc/build
  • PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_intel/build
  • PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_intel/omega_pr
  • Baseline Polaris: d8d9ed7ca3edcd1ac222015772c24e3f4761bb68 (1.1.0-609-gd8d9ed7ca)
  • PR Polaris: 37bc1c47b0dd4a621746795de774144fce36483b (1.1.0-615-g37bc1c47b)
  • Baseline Omega: 11e7cbc8176ba35bb2c660df931955e504124d17 (Omega-v0.1.0-alpha.1-6373-g11e7cbc817)
  • PR Omega: a4f1650e91ad1623aa3dd4ce2a83dcca0d7613a6 (Omega-v0.1.0-alpha.1-6391-ga4f1650e91)
  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_intel/omega_pr/polaris_omega_pr.o9011975
  • Result:
    • Diffs (3 of 36):
    - ocean/column/horiz_press_grad/salinity_gradient
      - no norms (missing file or variable)
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/split_explicit
      - temperature:    l1=5.965e-03  l2=1.165e-04  linf=5.793e-05
      - salinity:       l1=8.833e-04  l2=1.954e-05  linf=1.483e-05
      - normalVelocity: l1=5.787e-03  l2=6.569e-05  linf=1.387e-05
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4
      - normalVelocity: l1=1.883e-01  l2=1.500e-02  linf=7.976e-03
      - temperature:    l1=2.191e-02  l2=2.376e-03  linf=1.486e-03
      - salinity:       l1=2.375e-03  l2=1.800e-04  linf=1.087e-04
Recent commits

Baseline Polaris:

d8d9ed7ca Merge pull request #936 from xylar/pr-test-resume-after-suite
8fcdbe2bb Merge pull request #934 from xylar/document-commit-message-style
84e7190dc Merge pull request #933 from xylar/pr-test-update-main
d561288c0 Merge pull request #932 from xylar/update-omega
87fa5f06c Merge pull request #847 from xylar/make-fv-pgf-default

PR Polaris:

37bc1c47b Run the finite-volume scheme with its parabolic reconstruction too
09a397b0f Run the finite-volume kernel with the parabolic reconstruction
4330c1e2e Add resting-state variants quadratic in pressure
9af03457a Add the parabolic reconstruction to the finite-volume reference
c9282e2ab Record the resting-state ratios the anchor rule moved

Baseline Omega:

11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping
8a1cfdcf93 Merge pull request #624 from andrewdnolan/omega/remove-pyyaml-workarounds
e1c6689d4d Merge pull request #645 from andrewdnolan/omega/fix-division-by-zero
27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1
fd66ecb7d1 Merge pull request #448 from xylar/omega/design/add-higher-order-hpg-design

PR Omega:

a4f1650e91 Test merge of E3SM-Project/Omega#671 (48fecaab7b) for testing
c7bcf4ef11 Test merge of E3SM-Project/Omega#670 (5e7c80513c) for testing
242b8aef1f Test merge of E3SM-Project/Omega#669 (c329de826a) for testing
dcca042f25 Test merge of E3SM-Project/Omega#636 (ae76f3830d) into develop (11e7cbc817)
11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping

CTest unit tests:

  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_intel/build
  • Omega: a4f1650e91ad1623aa3dd4ce2a83dcca0d7613a6 (Omega-v0.1.0-alpha.1-6391-ga4f1650e91)
  • Result: All tests passed
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_intel/build/ctests.log

Coupled CIME tests

  • Cases: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_intel/coupled
  • CIME environment: /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_cime_env.sh
  • Result: All tests passed

ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel.pr636-ae76f38

PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel CREATE_NEWCASE
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel XML
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SETUP
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SHAREDLIB_BUILD time=135
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel MODEL_BUILD time=125
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SUBMIT
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel RUN time=942
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel COMPARE_base_rest
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel MEMLEAK
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SHORT_TERM_ARCHIVER

PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel.pr636-ae76f38

PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel CREATE_NEWCASE
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel XML
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SETUP
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SHAREDLIB_BUILD time=144
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel MODEL_BUILD time=116
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SUBMIT
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel RUN time=777
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel COMPARE_base_modpes
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel MEMLEAK
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SHORT_TERM_ARCHIVER

Build warnings

No new warnings (14 in the PR build, 14 in the baseline build).

  • Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-d8d9ed7_omega-11e7cbc/build/build_omega.log
  • PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_intel/build/build_omega.log
aurora, intelgpu, mpich

Polaris omega_pr suite

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-d8d9ed7_omega-11e7cbc/omega_pr
  • Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-d8d9ed7_omega-11e7cbc/build
  • PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-2e45303/aurora_intelgpu/build
  • PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-2e45303/aurora_intelgpu/omega_pr
  • Baseline Polaris: d8d9ed7ca3edcd1ac222015772c24e3f4761bb68 (1.1.0-609-gd8d9ed7ca)
  • PR Polaris: 37bc1c47b0dd4a621746795de774144fce36483b (1.1.0-615-g37bc1c47b)
  • Baseline Omega: 11e7cbc8176ba35bb2c660df931955e504124d17 (Omega-v0.1.0-alpha.1-6373-g11e7cbc817)
  • PR Omega: f2e71b9b4ca71b0704d7e8adc76c78e882e95686 (Omega-v0.1.0-alpha.1-6393-gf2e71b9b4c)
  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-2e45303/aurora_intelgpu/omega_pr/polaris_omega_pr.o9021451
  • Result:
    • Diffs (3 of 36):
    - ocean/column/horiz_press_grad/salinity_gradient
      - no norms (missing file or variable)
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/split_explicit
      - temperature:    l1=5.964e-03  l2=1.165e-04  linf=5.793e-05
      - salinity:       l1=8.830e-04  l2=1.954e-05  linf=1.483e-05
      - normalVelocity: l1=5.786e-03  l2=6.569e-05  linf=1.387e-05
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4
      - normalVelocity: l1=1.883e-01  l2=1.500e-02  linf=7.976e-03
      - temperature:    l1=2.191e-02  l2=2.376e-03  linf=1.486e-03
      - salinity:       l1=2.375e-03  l2=1.800e-04  linf=1.087e-04
Recent commits

Baseline Polaris:

d8d9ed7ca Merge pull request #936 from xylar/pr-test-resume-after-suite
8fcdbe2bb Merge pull request #934 from xylar/document-commit-message-style
84e7190dc Merge pull request #933 from xylar/pr-test-update-main
d561288c0 Merge pull request #932 from xylar/update-omega
87fa5f06c Merge pull request #847 from xylar/make-fv-pgf-default

PR Polaris:

37bc1c47b Run the finite-volume scheme with its parabolic reconstruction too
09a397b0f Run the finite-volume kernel with the parabolic reconstruction
4330c1e2e Add resting-state variants quadratic in pressure
9af03457a Add the parabolic reconstruction to the finite-volume reference
c9282e2ab Record the resting-state ratios the anchor rule moved

Baseline Omega:

11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping
8a1cfdcf93 Merge pull request #624 from andrewdnolan/omega/remove-pyyaml-workarounds
e1c6689d4d Merge pull request #645 from andrewdnolan/omega/fix-division-by-zero
27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1
fd66ecb7d1 Merge pull request #448 from xylar/omega/design/add-higher-order-hpg-design

PR Omega:

f2e71b9b4c Test merge of E3SM-Project/Omega#671 (48fecaab7b) for testing
d684a6bd91 Test merge of E3SM-Project/Omega#670 (5e7c80513c) for testing
2d7a7794c8 Test merge of E3SM-Project/Omega#669 (c329de826a) for testing
db19ea8426 Test merge of E3SM-Project/Omega#636 (2e45303eca) into develop (11e7cbc817)
11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping

CTest unit tests:

  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-2e45303/aurora_intelgpu/build
  • Omega: f2e71b9b4ca71b0704d7e8adc76c78e882e95686 (Omega-v0.1.0-alpha.1-6393-gf2e71b9b4c)
  • Result: All tests passed
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-2e45303/aurora_intelgpu/build/ctests.log

Coupled CIME tests

  • Cases: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-2e45303/aurora_intelgpu/coupled
  • CIME environment: /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_cime_env.sh
  • Result: All tests passed

ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu.pr636-2e45303

PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu CREATE_NEWCASE
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu XML
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SETUP
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SHAREDLIB_BUILD time=169
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu MODEL_BUILD time=409
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SUBMIT
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu RUN time=80
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu COMPARE_base_rest
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu MEMLEAK
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SHORT_TERM_ARCHIVER

PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu.pr636-2e45303

PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu CREATE_NEWCASE
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu XML
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SETUP
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SHAREDLIB_BUILD time=182
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu MODEL_BUILD time=397
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SUBMIT
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu RUN time=74
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu COMPARE_base_modpes
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu MEMLEAK
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SHORT_TERM_ARCHIVER

Build warnings

No new warnings (262 in the PR build, 265 in the baseline build).

  • Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-d8d9ed7_omega-11e7cbc/build/build_omega.log
  • PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-2e45303/aurora_intelgpu/build/build_omega.log

Notes

The omega_pr differences are the same 3 as on Chrysalis (#636 (comment)): the two QU240 realistic_global tasks match #635's norms, and only the 7 finite_volume_ppm steps of horiz_press_grad/salinity_gradient differ, since the baseline Omega rejects VerticalReconstruction: ppm.

In the first run, PGRAD_TEST failed on intelgpu in the parabolic harness fidelity check this PR adds: the kernel and the host reference differed by a relative 2.7e-10, against a tolerance of 1e-10 of the residual. That difference is 0.003 of the round-off of the terms that cancel to form the residual, so 90e807c1bc compares the two at 8 eps of that scale instead, as the linear reconstruction's fidelity check does. In the rerun, with the same kernel and reference values, the check passes.

With #669 merged, ERS_D passes on intelgpu (it crashes on develop, #666); with #670, so does VERTMIX_TEST.

Cost of the parabolic reconstruction

cached_forward runs from the first run's PR build (the fixes since change only test code), with each scheme run twice in one job; the two runs agree within 1%. Times are means from rank 0's omega.timing.0. EC30to60E2r2 (236,853 cells) runs 47 split-explicit steps on one node.

Row Task Scheme doStep (s) pressureGradTerm (s) doStep vs Centered doStep vs FV linear
intel QU240 rk4 Centered 82.3 1.03
intel QU240 rk4 FV linear 92.2 4.03 +12.1%
intel QU240 rk4 FV ppm 106.0 11.87 +28.9% +15.0%
intel QU240 split_explicit Centered 9.28 0.09
intel QU240 split_explicit FV linear 10.29 0.33 +10.9%
intel QU240 split_explicit FV ppm 11.46 0.99 +23.6% +11.4%
intelgpu QU240 rk4 Centered 12.23 0.18
intelgpu QU240 rk4 FV linear 13.17 1.22 +7.6%
intelgpu QU240 rk4 FV ppm 13.90 1.98 +13.7% +5.6%
intelgpu QU240 split_explicit Centered 2.55 0.01
intelgpu QU240 split_explicit FV linear 2.62 0.10 +2.7%
intelgpu QU240 split_explicit FV ppm 2.68 0.16 +5.3% +2.5%
intelgpu EC30to60E2r2 split_explicit Centered 4.40 0.02
intelgpu EC30to60E2r2 split_explicit FV linear 4.53 0.12 +3.0%
intelgpu EC30to60E2r2 split_explicit FV ppm 4.61 0.20 +4.8% +1.7%

With ppm, pressureGradTerm takes 2.9-3.0× as long as with linear on the CPU and 1.6× on the GPU. Runs: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr636-ae76f38/aurora_{intel,intelgpu}/ppm_perf


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Nov 1, 2026 •

Copy link
Copy Markdown
Author

Testing on pm-cpu and pm-gpu

The pm-cpu row tested the merge of #636 (ae76f3830d) into develop (11e7cbc817), with #669, #670, #671 merged in, as a4f1650e91, against Polaris' Omega submodule (11e7cbc817). The baseline ran with Polaris d8d9ed7ca3, without the changes in Polaris 37bc1c47b0 that the PR needs.

The pm-gpu row was rerun after this PR's test-only fixes, testing the merge of #636 (2e45303eca), with the same PRs merged in, as f2e71b9b4c, against the same baseline.

Row CTests omega_pr Coupled New build warnings
pm-cpu, gnu, mpich 62 of 62 passed 33 of 36 passed, 3 with baseline differences 2 of 2 passed 0
pm-gpu, gnugpu, mpich 62 of 62 passed 33 of 36 passed, 3 with baseline differences 2 of 2 passed 0
pm-cpu, gnu, mpich

Polaris omega_pr suite

  • Baseline workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-d8d9ed7_omega-11e7cbc/omega_pr
  • Baseline build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-d8d9ed7_omega-11e7cbc/build
  • PR build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-cpu_gnu/build
  • PR workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-cpu_gnu/omega_pr
  • Baseline Polaris: d8d9ed7ca3edcd1ac222015772c24e3f4761bb68 (1.1.0-609-gd8d9ed7ca)
  • PR Polaris: 37bc1c47b0dd4a621746795de774144fce36483b (1.1.0-615-g37bc1c47b)
  • Baseline Omega: 11e7cbc8176ba35bb2c660df931955e504124d17 (Omega-v0.1.0-alpha.1-6373-g11e7cbc817)
  • PR Omega: a4f1650e91ad1623aa3dd4ce2a83dcca0d7613a6 (Omega-v0.1.0-alpha.1-6391-ga4f1650e91)
  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-cpu_gnu/omega_pr/polaris_omega_pr.o60627987
  • Result:
    • Diffs (3 of 36):
    - ocean/column/horiz_press_grad/salinity_gradient
      - no norms (missing file or variable)
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/split_explicit
      - temperature:    l1=5.965e-03  l2=1.165e-04  linf=5.793e-05
      - salinity:       l1=8.833e-04  l2=1.954e-05  linf=1.483e-05
      - normalVelocity: l1=5.787e-03  l2=6.569e-05  linf=1.387e-05
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4
      - normalVelocity: l1=1.883e-01  l2=1.500e-02  linf=7.976e-03
      - temperature:    l1=2.191e-02  l2=2.376e-03  linf=1.486e-03
      - salinity:       l1=2.375e-03  l2=1.800e-04  linf=1.087e-04
Recent commits

Baseline Polaris:

d8d9ed7ca Merge pull request #936 from xylar/pr-test-resume-after-suite
8fcdbe2bb Merge pull request #934 from xylar/document-commit-message-style
84e7190dc Merge pull request #933 from xylar/pr-test-update-main
d561288c0 Merge pull request #932 from xylar/update-omega
87fa5f06c Merge pull request #847 from xylar/make-fv-pgf-default

PR Polaris:

37bc1c47b Run the finite-volume scheme with its parabolic reconstruction too
09a397b0f Run the finite-volume kernel with the parabolic reconstruction
4330c1e2e Add resting-state variants quadratic in pressure
9af03457a Add the parabolic reconstruction to the finite-volume reference
c9282e2ab Record the resting-state ratios the anchor rule moved

Baseline Omega:

11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping
8a1cfdcf93 Merge pull request #624 from andrewdnolan/omega/remove-pyyaml-workarounds
e1c6689d4d Merge pull request #645 from andrewdnolan/omega/fix-division-by-zero
27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1
fd66ecb7d1 Merge pull request #448 from xylar/omega/design/add-higher-order-hpg-design

PR Omega:

a4f1650e91 Test merge of E3SM-Project/Omega#671 (48fecaab7b) for testing
c7bcf4ef11 Test merge of E3SM-Project/Omega#670 (5e7c80513c) for testing
242b8aef1f Test merge of E3SM-Project/Omega#669 (c329de826a) for testing
dcca042f25 Test merge of E3SM-Project/Omega#636 (ae76f3830d) into develop (11e7cbc817)
11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping

CTest unit tests:

  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-cpu_gnu/build
  • Omega: a4f1650e91ad1623aa3dd4ce2a83dcca0d7613a6 (Omega-v0.1.0-alpha.1-6391-ga4f1650e91)
  • Result: All tests passed
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-cpu_gnu/build/ctests.log

Coupled CIME tests

  • Cases: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-cpu_gnu/coupled
  • CIME environment: /global/common/software/e3sm/anaconda_envs/load_latest_cime_env.sh
  • Result: All tests passed

ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu.pr636-ae76f38

PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu CREATE_NEWCASE
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu XML
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SETUP
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SHAREDLIB_BUILD time=82
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu MODEL_BUILD time=164
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SUBMIT
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu RUN time=1029
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu COMPARE_base_rest
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu MEMLEAK
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SHORT_TERM_ARCHIVER

PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu.pr636-ae76f38

PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu CREATE_NEWCASE
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu XML
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SETUP
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SHAREDLIB_BUILD time=98
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu MODEL_BUILD time=153
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SUBMIT
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu RUN time=821
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu COMPARE_base_modpes
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu MEMLEAK
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-cpu_gnu SHORT_TERM_ARCHIVER

Build warnings

No new warnings (4 in the PR build, 4 in the baseline build).

  • Baseline build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-d8d9ed7_omega-11e7cbc/build/build_omega.log
  • PR build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-cpu_gnu/build/build_omega.log
pm-gpu, gnugpu, mpich

Polaris omega_pr suite

  • Baseline workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-d8d9ed7_omega-11e7cbc/omega_pr
  • Baseline build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-d8d9ed7_omega-11e7cbc/build
  • PR build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-2e45303/pm-gpu_gnugpu/build
  • PR workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-2e45303/pm-gpu_gnugpu/omega_pr
  • Baseline Polaris: d8d9ed7ca3edcd1ac222015772c24e3f4761bb68 (1.1.0-609-gd8d9ed7ca)
  • PR Polaris: 37bc1c47b0dd4a621746795de774144fce36483b (1.1.0-615-g37bc1c47b)
  • Baseline Omega: 11e7cbc8176ba35bb2c660df931955e504124d17 (Omega-v0.1.0-alpha.1-6373-g11e7cbc817)
  • PR Omega: f2e71b9b4ca71b0704d7e8adc76c78e882e95686 (Omega-v0.1.0-alpha.1-6393-gf2e71b9b4c)
  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-2e45303/pm-gpu_gnugpu/omega_pr/polaris_omega_pr.o60641662
  • Result:
    • Diffs (3 of 36):
    - ocean/column/horiz_press_grad/salinity_gradient
      - no norms (missing file or variable)
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/split_explicit
      - temperature:    l1=5.964e-03  l2=1.165e-04  linf=5.793e-05
      - salinity:       l1=8.831e-04  l2=1.954e-05  linf=1.483e-05
      - normalVelocity: l1=5.786e-03  l2=6.569e-05  linf=1.387e-05
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4
      - normalVelocity: l1=1.883e-01  l2=1.500e-02  linf=7.976e-03
      - temperature:    l1=2.191e-02  l2=2.376e-03  linf=1.486e-03
      - salinity:       l1=2.375e-03  l2=1.800e-04  linf=1.087e-04
Recent commits

Baseline Polaris:

d8d9ed7ca Merge pull request #936 from xylar/pr-test-resume-after-suite
8fcdbe2bb Merge pull request #934 from xylar/document-commit-message-style
84e7190dc Merge pull request #933 from xylar/pr-test-update-main
d561288c0 Merge pull request #932 from xylar/update-omega
87fa5f06c Merge pull request #847 from xylar/make-fv-pgf-default

PR Polaris:

37bc1c47b Run the finite-volume scheme with its parabolic reconstruction too
09a397b0f Run the finite-volume kernel with the parabolic reconstruction
4330c1e2e Add resting-state variants quadratic in pressure
9af03457a Add the parabolic reconstruction to the finite-volume reference
c9282e2ab Record the resting-state ratios the anchor rule moved

Baseline Omega:

11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping
8a1cfdcf93 Merge pull request #624 from andrewdnolan/omega/remove-pyyaml-workarounds
e1c6689d4d Merge pull request #645 from andrewdnolan/omega/fix-division-by-zero
27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1
fd66ecb7d1 Merge pull request #448 from xylar/omega/design/add-higher-order-hpg-design

PR Omega:

f2e71b9b4c Test merge of E3SM-Project/Omega#671 (48fecaab7b) for testing
d684a6bd91 Test merge of E3SM-Project/Omega#670 (5e7c80513c) for testing
2d7a7794c8 Test merge of E3SM-Project/Omega#669 (c329de826a) for testing
db19ea8426 Test merge of E3SM-Project/Omega#636 (2e45303eca) into develop (11e7cbc817)
11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping

CTest unit tests:

  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-2e45303/pm-gpu_gnugpu/build
  • Omega: f2e71b9b4ca71b0704d7e8adc76c78e882e95686 (Omega-v0.1.0-alpha.1-6393-gf2e71b9b4c)
  • Result: All tests passed
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-2e45303/pm-gpu_gnugpu/build/ctests.log

Coupled CIME tests

  • Cases: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-2e45303/pm-gpu_gnugpu/coupled
  • CIME environment: /global/common/software/e3sm/anaconda_envs/load_latest_cime_env.sh
  • Result: All tests passed

ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu.pr636-2e45303

PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu CREATE_NEWCASE
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu XML
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SETUP
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SHAREDLIB_BUILD time=80
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu MODEL_BUILD time=353
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SUBMIT
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu RUN time=130
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu COMPARE_base_rest
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu MEMLEAK
PASS ERS_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SHORT_TERM_ARCHIVER

PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu.pr636-2e45303

PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu CREATE_NEWCASE
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu XML
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SETUP
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SHAREDLIB_BUILD time=96
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu MODEL_BUILD time=347
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SUBMIT
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu RUN time=101
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu COMPARE_base_modpes
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu MEMLEAK
PASS PEM_D.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SHORT_TERM_ARCHIVER

Build warnings

No new warnings (853 in the PR build, 853 in the baseline build).

  • Baseline build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-d8d9ed7_omega-11e7cbc/build/build_omega.log
  • PR build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-2e45303/pm-gpu_gnugpu/build/build_omega.log

Notes

The omega_pr differences are the same 3 as on Chrysalis (#636 (comment)), which explains them.

In the first run, GCC reported 2 new -Wfree-nonheap-object warnings on pm-gpu, on the vector of probe pressures in testParabolicReconstruction(), which this PR adds. 2e45303eca holds them in a fixed array instead, and the rerun has no new warnings. PGRAD_TEST passes in both runs.

Cost of the parabolic reconstruction

These are cached_forward runs from the first run's PR build (the fixes since change only test code). Each scheme ran twice in one job, and the two doStep times agree within 1%. Times are means from rank 0's omega.timing.0. QU240 runs on 1 GPU on pm-gpu. EC30to60E2r2 runs 47 split-explicit steps on 12 GPUs (3 nodes).

Row Task Scheme doStep (s) pressureGradTerm (s) doStep vs Centered doStep vs FV linear
pm-cpu QU240 rk4 Centered 163.11 2.20
pm-cpu QU240 rk4 FV linear 185.01 11.10 +13.4%
pm-cpu QU240 rk4 FV ppm 203.16 21.39 +24.6% +9.8%
pm-cpu QU240 split_explicit Centered 18.24 0.18
pm-cpu QU240 split_explicit FV linear 20.51 0.92 +12.5%
pm-cpu QU240 split_explicit FV ppm 21.95 1.77 +20.4% +7.0%
pm-gpu QU240 rk4 Centered 9.55 0.20
pm-gpu QU240 rk4 FV linear 10.30 1.00 +7.9%
pm-gpu QU240 rk4 FV ppm 11.32 2.02 +18.6% +9.9%
pm-gpu QU240 split_explicit Centered 1.50 0.02
pm-gpu QU240 split_explicit FV linear 1.56 0.08 +3.8%
pm-gpu QU240 split_explicit FV ppm 1.64 0.17 +9.3% +5.3%
pm-gpu EC30to60E2r2 split_explicit Centered 3.59 0.02
pm-gpu EC30to60E2r2 split_explicit FV linear 3.67 0.09 +2.4%
pm-gpu EC30to60E2r2 split_explicit FV ppm 3.77 0.19 +5.2% +2.7%

With ppm, pressureGradTerm takes 1.9× as long as with linear on pm-cpu and 2.0× on pm-gpu. Runs: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr636-ae76f38/pm-{cpu_gnu,gpu_gnugpu}/ppm_perf


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar and others added 2 commits November 1, 2026 09:13
…lling terms

The parabolic harness fidelity check held the host assembly to the
kernel within a relative 1e-10 of the cubic profile's tendency. That
tendency is itself a residual of terms that cancel, about 1e7 times
their round-off, so the tolerance was tighter than one rounding of
those terms. On Aurora's Intel GPUs the kernel rounds differently from
the host, and the two differed by 1.9e-19 m/s2, 0.003 of that
round-off, which failed the check.

The check now uses the absolute tolerance of the linear reconstruction's
fidelity check, 8 eps times the scale of the cancelling terms, still
about 1e-6 of the residual.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GCC 14 reported -Wfree-nonheap-object, in both precisions, on the
vector of probe pressures that testParabolicReconstruction grows for
each layer. There are at most nine, so a fixed array holds them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xylar

xylar commented Nov 1, 2026 •

Copy link
Copy Markdown
Author

Testing summary

All 5 rows have been run: Chrysalis, Aurora, Perlmutter. Every CTest and coupled test passes. Every row has the same 3 expected omega_pr diffs:

Two test-only fixes followed the first runs, and the GPU rows were rerun with them (the Aurora and Perlmutter comments above are updated):

  • 90e807c1bc: the parabolic harness fidelity check (PGRAD_TEST) failed on Aurora intelgpu, where the kernel and the reference differed by a relative 2.7e-10 of the residual, against a tolerance of 1e-10. That is 0.003 of the round-off of the cancelling terms, so the check now uses the absolute tolerance of the linear reconstruction's fidelity check, 8 eps of that scale. It passes on intelgpu now.
  • 2e45303eca: GCC 14 reported 2 new -Wfree-nonheap-object warnings on pm-gpu, from a vector in the new testParabolicReconstruction(). A fixed array replaces it, and the rerun has no new warnings.

Cost of ppm

PPM cost

With ppm, the pressure gradient takes 1.6–3× as long as with linear reconstruction. Per time step, ppm adds 7–21% to FV linear's cost on CPUs, against 11–21% for FV linear over Centered. On GPUs it adds 2–10%. On the GPU EC30to60E2r2 runs it adds only 2–3%, so ppm is about 5% slower than Centered there.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Omega

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants