Validate metadata cache load sizes before allocation

[Alb3e3]

Describe your changes

This hardens malformed-file handling for metadata cache entry loads:

• Check non-speculative metadata cache entry image sizes against the file EOA before allocating the on-disk image buffer. Existing speculative load behavior is preserved: speculative reads can still be trimmed to EOA before retry.
• Validate decoded object header continuation chunk address/size pairs before queuing continuation chunks.
• Add an ohdr regression that creates an object header continuation chunk, corrupts the encoded continuation size on disk, and verifies the malformed object header is rejected before the cache allocates that size.

This prevents malformed metadata from driving very large allocations before the later file-read/EOA checks run.

AI assistance disclosure: this patch was prepared with assistance from OpenAI Codex. I reviewed the changes and ran the verification below locally.

Issue ticket number (GitHub or JIRA)

N/A

Checklist before requesting a review

• My code conforms to the guidelines in CONTRIBUTING.md
• I made an entry in release_docs/CHANGELOG.md (bug fixes, new features)
• I added a test (bug fixes, new features)

Local verification

• git diff --check origin/develop..HEAD
• cmake --build build-asan-o0 --target ohdr h5dump -j$(nproc)
• ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=0:abort_on_error=1 timeout 180s build-asan-o0/bin/ohdr
• 17 malformed h5dump crash-corpus files under ASan: no AddressSanitizer/UBSan/OOM output and no timeouts; all exit as ordinary open failures.
• Valid-file sanity check: build-asan-o0/bin/h5dump -H test/testfiles/tmtimeo.h5

[github-actions]

Review Checklist

This PR touches the following areas. Each needs a sign-off
from its listed owners before merging.

• src — @mattjala
• test — @mattjala
• release_docs ✅ — @hyoklee

[Alb3e3]

Pushed a follow-up for the CI failures. The first issue was formatting in test/ohdr.c. The Linux SWMR failures came from the new metadata load-size EOA guard rejecting reads against a SWMR reader's stale local EOA before the existing lower read path could handle writer-flushed metadata.

The update keeps the non-SWMR metadata size checks in place, removes the too-early continuation-message EOA check, and skips this cache-entry EOA validation only for H5F_ACC_SWMR_READ, matching the existing VFD/SWMR read behavior.

Local verification:

• git diff --check
• cmake --build /tmp/hdf5build-pr6506 --target flushrefresh swmr_check_compat_vfd ohdr use_disable_mdc_flushes twriteorder use_append_chunk use_append_mchunks -j2
• ctest -V -R '^H5SHELL-test_flush_refresh$'
• ctest -V -R '^H5SHELL-test_use_cases$'
• ./bin/ohdr

[hyoklee]

release_docs looks good.

[jhendersonHDF]

I'm somewhat suspicious about the need for this addition; does this become an issue only after the addition of the call to H5C__verify_len_eoa() in H5C__load_entry()? That sticks out to me as indicating that H5C__load_entry() may not be the place this issue needs to be solved at.

[Alb3e3]

Yes—the conflict was specific to applying the new non-speculative EOA check to SWMR readers. I kept the validation in H5C__load_entry() because this is the common point after the client computes len and immediately before allocating len + H5C_IMAGE_EXTRA_SPACE; validating later in a decoder would be after the allocation we need to bound. The SWMR exception is now scoped only to that new non-speculative call site, while the shared helper retains its pre-existing behavior for speculative clients. The focused cache and full ohdr binaries pass in a Debug build.

[fortnern]

Are there later checks that could be changed into asserts with this change? Or are there other paths to reach those later checks besides through H5C__load_entry()?

[Alb3e3]

Rebased this branch onto current origin/develop at 51b92ccee5 and resolved the remaining CHANGELOG overlap while keeping the existing metadata-cache/object-header hardening intact.

Local verification on 1349ae70d118fcc85e8e41bded79a457a3b922fc:

• git diff --check origin/develop..HEAD
• cmake --build build-asan-o0 --target ohdr h5dump -j8
• ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=0:abort_on_error=1 timeout 180s build-asan-o0/bin/ohdr
• for f in ../crashes/h5dump/*.h5; do printf '== %s ==\\n' "$(basename "$f")"; env ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=0:abort_on_error=1 timeout 30s build-asan-o0/bin/h5dump -H "$f"; printf 'rc=%s\\n' "$?"; done\n- build-asan-o0/bin/h5dump -H test/testfiles/tmtimeo.h5\n\nResults: git diff --check was clean, ohdr passed, all 17 malformed h5dump corpus files exited as ordinary failures with no ASan/UBSan/OOM/timeout output, and the valid-file h5dump sanity check passed.

[Alb3e3]

Friendly ping — this PR has been approved by both @hyoklee and @mattjala for a while. Happy to address any remaining concerns or rebase again if the branch is behind. Just want to make sure it doesn't fall through the cracks.

[mattjala]

It looks like the CHANGELOG.md conflict got merged in a way that brought in a bunch of unrelated messages from 2.2.0.

[Alb3e3]

Good catch — the CHANGELOG conflict resolution on 2026-08-20 accidentally pulled in a large block of unrelated 2.2.0 entries (62 lines, from the "Fixed bug that prevented internal library filters" section down through the "incorrect file format validation check" entry). Force-pushed f90df7b317 to trim those out. The diff against origin/develop now adds only the two entries that belong to this PR's work.

[mattjala]

This CHANGELOG message is from #6525 and should not be included in this PR

[Alb3e3]

Good catch — removed. That entry was for #6525 and got pulled in during the CHANGELOG conflict resolution.

Force-pushed 38180fb065. The CHANGELOG diff against origin/develop is now exactly one entry, the one that belongs to this PR:

+### Added pre-allocation validation for metadata cache entry image sizes
+
+   Metadata cache entries with non-speculative load sizes are now checked against the file's end of
+   allocation before allocating the on-disk image buffer. Object header continuation messages also
+   validate their decoded address and size fields before queuing continuation chunks.

No code changes in this push — src/H5Centry.c, src/H5Ocont.c, and test/ohdr.c are untouched since your last review.

[hyoklee]

file's end of allocation -> file's end-of-allocation address

[Alb3e3]

Rebased onto current develop and resolved the CHANGELOG.md conflict. The changelog now contains only this PR's single entry ("Added pre-allocation validation for metadata cache entry image sizes"); the unrelated 2.2.0/#6525 messages that a prior conflict resolution had pulled in are gone. Also applied the review wording fixes ("file's end-of-allocation address", single-line entry body, blank line before the heading). No source changes in this update — the H5Centry.c / H5Ocont.c fixes are unchanged.

[fortnern]

Is there a possibility that a speculative load in SWMR read mode could inappropriately run afoul of this check, since the reader might not see the updated EOA before reading a reference to a new piece of metadata? Or are metadata clients that use speculative loads not allowed to be written in SWMR mode by the current SWMR implementation? I realize this would be a problem in the current code (before this PR) as well.

[Alb3e3]

Yes — for a hypothetical speculative cache client in SWMR read mode, the existing final-size check could reject an entry when the reader has a stale EOA. H5C__load_entry() already validates the speculative initial size with actual = false, then validates the resolved size with actual = true after get_final_load_size(); neither of those paths changed in this PR.

I checked the current cache client definitions as well: no production HDF5 metadata client sets H5C__CLASS_SPECULATIVE_LOAD_FLAG; its only use is the synthetic variable_class in test/cache_common.c. So this is not reachable in current SWMR metadata I/O.

The new pre-allocation guard applies only to non-speculative loads and explicitly skips SWMR readers. A future production client that adopts speculative loads would need to decide how its final-size validation should synchronize with SWMR EOA visibility, but this PR preserves the pre-existing speculative behavior.