Skip to content

Fix VERTMIX_TEST Rayleigh damping tests on Intel GPUs - #670

Open
xylar wants to merge 1 commit into
E3SM-Project:developfrom
xylar:omega/fix-vertmix-rayleigh-test
Open

Fix VERTMIX_TEST Rayleigh damping tests on Intel GPUs#670
xylar wants to merge 1 commit into
E3SM-Project:developfrom
xylar:omega/fix-vertmix-rayleigh-test

Conversation

@xylar

@xylar xylar commented Oct 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Fix the VERTMIX_TEST Rayleigh damping tests on Intel GPUs.

The Rayleigh damping tests in VERTMIX_TEST recomputed the initial normal velocity on the host and required layers the solve leaves unchanged to match it exactly. On aurora with intelgpu, the device value can differ in the last bit, so those layers failed. The tests now copy the initial velocity from the device instead. Only test code changes.

Fixes #668

Checklist

  • Linting

  • Building

    • CMake build does not produce any new warnings from changes in this PR

  • Testing

    aurora, intel, mpich

    • CTests Pass
    • Polaris omega_pr Pass
    • Coupled CIME tests Pass

    aurora, intelgpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass
    • Coupled CIME tests Pass

  • Provide relevant details in a comment to the PR titled Testing

🤖 Generated with Claude Code

@xylar @claude
Compare Rayleigh damping tests to device initial velocity
5e7c805 
The implicit vertical mixing tests in VertMixTest recomputed the initial
normal velocity on the host and required inactive layers to match it
exactly. On Intel GPUs, the device evaluation of the same expression can
differ in the last bit (e.g. from FMA contraction), so unchanged inactive
layers were reported as mismatches and growth errors.

Copy the initial normal velocity from the device instead and use it as
the reference for unchanged layers, the expected decay and the host
reference solve.

Fixes E3SM-Project#668

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xylar

xylar commented Oct 31, 2026

Copy link
Copy Markdown
Author

Testing

CTest unit tests:

  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Build: /lus/flare/projects/E3SM_Dec/xylar/scratch/omega_issue668/diag/build_omega/build_aurora_intelgpu
  • Omega: 5e7c80513c (the utility recorded 11e7cbc817-dirty because the build started just before the commit; the working tree was identical to 5e7c80513c, and VertMixTest.cpp.o was compiled after the source's last change)
  • Result: All tests passed (62 of 62)
  • Log: /lus/flare/projects/E3SM_Dec/xylar/scratch/omega_issue668/diag/build_omega/build_aurora_intelgpu/ctests.log

VERTMIX_TEST now runs all of the Rayleigh damping checks:

[info] [VertMixTest.cpp:1451] TestVertMixRayleigh: Decay PASS
[info] [VertMixTest.cpp:1451] TestVertMixRayleigh: LargeCoeff PASS
[info] [VertMixTest.cpp:1486] TestVertMixRayleigh: Neutral PASS
[info] [VertMixTest.cpp:1524] TestVertMixRayleigh: Combined PASS
[info] [VertMixTest.cpp:1606] ------ Vertical Mixing Unit Tests Successful ------

Reproducing and diagnosing the failure

  • With develop (11e7cbc817), built the same way, VERTMIX_TEST failed exactly as reported in VERTMIX_TEST ctest fail on Aurora with intelgpu #668: Decay FAIL with DT * Coeff = 0.060000000000000005, 3816 mismatches and 1908 sign or growth errors.
  • A temporary diagnostic build that logged the first mismatches showed that all of them were on edges with no active layers (KMin=0, KMax=-1), where the solver leaves the velocity unchanged. The unchanged device value differed from the host recomputation of initNormalVel() in the last bit, e.g. Vel=0.34199999999999997 vs. U0=0.342 and Vel=0.41400000000000003 vs. U0=0.414. The second kind (device value larger) also counted as "growth" errors, which is why there were exactly half as many sign/growth errors as mismatches.

Still to do: CTests on aurora with intel (CPU).

🤖 Generated with Claude Code

@xylar xylar added bug Something isn't working Omega CTest Aurora ALCF machine Aurora labels Oct 31, 2026
@xylar

xylar commented Oct 31, 2026

Copy link
Copy Markdown
Author

Testing on aurora

Tested the merge of #670 (5e7c80513c) into develop (11e7cbc817) as 999a67b541, against Polaris' Omega submodule (27765dd1f5).

Row CTests omega_pr Coupled New build warnings
aurora, intel, mpich 62 of 62 passed 36 of 36 passed 2 of 2 passed 0
aurora, intelgpu, mpich 62 of 62 passed 36 of 36 passed 1 of 1 passed 0
aurora, intel, mpich

Polaris omega_pr suite

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-87fa5f0_omega-27765dd/omega_pr
  • Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-87fa5f0_omega-27765dd/build
  • PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intel/build
  • PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intel/omega_pr
  • Polaris: 87fa5f06c44d513dbfe6aefc9e5872e8c8c39aee (1.1.0-601-g87fa5f06c)
  • Baseline Omega: 27765dd1f5e0e4556386de689cbba9df6a19ff7e (Omega-v0.1.0-alpha.1-6356-g27765dd1f5)
  • PR Omega: 999a67b54189f8209d64d71f4035a769e7692f48 (Omega-v0.1.0-alpha.1-6375-g999a67b541)
  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intel/omega_pr/polaris_omega_pr.o9004499
  • Result: All tests passed
Recent commits

Polaris:

87fa5f06c Merge pull request #847 from xylar/make-fv-pgf-default
457c622c5 Merge pull request #930 from xylar/aurora-coupled-capacity-queue
60c951c7b Merge pull request #825 from xylar/add-sigma-overflow
55b210d6a Merge pull request #919 from cbegeman/add-mle-coarse-omega-nightly
2face2024 Merge pull request #842 from xylar/update-land-ice-flux-options

Baseline Omega:

27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1
fd66ecb7d1 Merge pull request #448 from xylar/omega/design/add-higher-order-hpg-design
e09fcf841d Merge pull request #621 from mwarusz/omega/kpp-fixes
6af2d90296 Merge pull request #632 from xylar/omega/fix-fct-tracer-bounds
76da94a2ec Merge pull request #646 from mwarusz/omega/fix-ri-smoothing

PR Omega:

999a67b541 Test merge of E3SM-Project/Omega#670 (5e7c80513c) into develop (11e7cbc817)
11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping
8a1cfdcf93 Merge pull request #624 from andrewdnolan/omega/remove-pyyaml-workarounds
e1c6689d4d Merge pull request #645 from andrewdnolan/omega/fix-division-by-zero
27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1

CTest unit tests:

  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intel/build
  • Omega: 999a67b54189f8209d64d71f4035a769e7692f48 (Omega-v0.1.0-alpha.1-6375-g999a67b541)
  • Result: All tests passed
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intel/build/ctests.log

Coupled CIME tests

  • Cases: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intel/coupled
  • CIME environment: /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_cime_env.sh
  • Result: All tests passed

ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel.pr670-5e7c805

PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel CREATE_NEWCASE
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel XML
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SETUP
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SHAREDLIB_BUILD time=152
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel MODEL_BUILD time=136
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SUBMIT
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel RUN time=936
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel COMPARE_base_rest
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel MEMLEAK
PASS ERS_D.T62_oQU240.COMEGA-IAF.aurora_intel SHORT_TERM_ARCHIVER

PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel.pr670-5e7c805

PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel CREATE_NEWCASE
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel XML
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SETUP
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SHAREDLIB_BUILD time=163
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel MODEL_BUILD time=125
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SUBMIT
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel RUN time=781
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel COMPARE_base_modpes
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel MEMLEAK
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intel SHORT_TERM_ARCHIVER

Build warnings

No new warnings (14 in the PR build, 14 in the baseline build).

  • Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-87fa5f0_omega-27765dd/build/build_omega.log
  • PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intel/build/build_omega.log
aurora, intelgpu, mpich

Polaris omega_pr suite

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-87fa5f0_omega-27765dd/omega_pr
  • Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-87fa5f0_omega-27765dd/build
  • PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intelgpu/build
  • PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intelgpu/omega_pr
  • Polaris: 87fa5f06c44d513dbfe6aefc9e5872e8c8c39aee (1.1.0-601-g87fa5f06c)
  • Baseline Omega: 27765dd1f5e0e4556386de689cbba9df6a19ff7e (Omega-v0.1.0-alpha.1-6356-g27765dd1f5)
  • PR Omega: 999a67b54189f8209d64d71f4035a769e7692f48 (Omega-v0.1.0-alpha.1-6375-g999a67b541)
  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intelgpu/omega_pr/polaris_omega_pr.o9004776
  • Result: All tests passed
Recent commits

Polaris:

87fa5f06c Merge pull request #847 from xylar/make-fv-pgf-default
457c622c5 Merge pull request #930 from xylar/aurora-coupled-capacity-queue
60c951c7b Merge pull request #825 from xylar/add-sigma-overflow
55b210d6a Merge pull request #919 from cbegeman/add-mle-coarse-omega-nightly
2face2024 Merge pull request #842 from xylar/update-land-ice-flux-options

Baseline Omega:

27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1
fd66ecb7d1 Merge pull request #448 from xylar/omega/design/add-higher-order-hpg-design
e09fcf841d Merge pull request #621 from mwarusz/omega/kpp-fixes
6af2d90296 Merge pull request #632 from xylar/omega/fix-fct-tracer-bounds
76da94a2ec Merge pull request #646 from mwarusz/omega/fix-ri-smoothing

PR Omega:

999a67b541 Test merge of E3SM-Project/Omega#670 (5e7c80513c) into develop (11e7cbc817)
11e7cbc817 Merge pull request #587 from xylar/omega/add-rayleigh-damping
8a1cfdcf93 Merge pull request #624 from andrewdnolan/omega/remove-pyyaml-workarounds
e1c6689d4d Merge pull request #645 from andrewdnolan/omega/fix-division-by-zero
27765dd1f5 Merge pull request #493 from xylar/omega/add-finite-volume-horiz-press-grad-phase1

CTest unit tests:

  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intelgpu/build
  • Omega: 999a67b54189f8209d64d71f4035a769e7692f48 (Omega-v0.1.0-alpha.1-6375-g999a67b541)
  • Result: All tests passed
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intelgpu/build/ctests.log

Coupled CIME tests

  • Cases: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intelgpu/coupled
  • CIME environment: /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_cime_env.sh
  • Result: All tests passed

PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu.pr670-5e7c805

PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu CREATE_NEWCASE
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu XML
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SETUP
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SHAREDLIB_BUILD time=168
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu MODEL_BUILD time=560
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SUBMIT
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu RUN time=2995
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu COMPARE_base_modpes
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu MEMLEAK
PASS PEM_D.T62_oQU240.COMEGA-IAF.aurora_intelgpu SHORT_TERM_ARCHIVER

Build warnings

No new warnings (220 in the PR build, 263 in the baseline build).

  • Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-87fa5f0_omega-27765dd/build/build_omega.log
  • PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr670-5e7c805/aurora_intelgpu/build/build_omega.log

Notes

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar
xylar requested a review from brian-oneill October 31, 2026 14:18
@xylar

xylar commented Oct 31, 2026

Copy link
Copy Markdown
Author

@brian-oneill, for reporting #668, you get the dubious honor of reviewing this if you don't mind.

This was referenced Oct 31, 2026
Open
Draft
brian-oneill
brian-oneill approved these changes Nov 2, 2026
View reviewed changes

@brian-oneill brian-oneill left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Using the device initialized velocity array resolves the host/device rounding mismatch without relaxing tolerances. Just for completeness, ctests also pass on Chrysalis w/ intel, pm-cpu w/ gnu, and pm-gpu w/ gnugpu.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brian-oneill brian-oneill brian-oneill approved these changes

Assignees

@sbrus89 sbrus89

Labels

Aurora ALCF machine Aurora bug Something isn't working CTest Omega

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

VERTMIX_TEST ctest fail on Aurora with intelgpu

3 participants

@xylar @brian-oneill @sbrus89