Expose chunk-level HDF5 APIs (H5Dchunk_iter, direct chunk I/O) to Java (JNI + FFM) by mkitti · Pull Request #6547 · HDFGroup/hdf5

mkitti · 2026-07-21T09:15:59Z

Summary

Exposes HDF5's chunk-level APIs to both Java binding trees (java/src-jni JNI and java/hdf FFM/Panama), working toward full C API parity in the Java bindings:

H5Dchunk_iter, H5Dget_num_chunks, H5Dget_chunk_info (by index)
H5Dget_chunk_info_by_coord, H5Dget_chunk_storage_size, H5Dget_chunk_index_type, H5Dwrite_chunk, H5Dread_chunk (direct raw chunk I/O, bypassing the filter pipeline/hyperslab), plus the five missing H5D_CHUNK_IDX_* constants
H5Dchunk_iter_all, a bulk convenience form of H5Dchunk_iter returning every chunk's offset/filter mask/address/size as a single H5D_chunk_info_t instead of requiring a callback

Also includes a performance fix: H5D_chunk_iter_cb (JNI) was redundantly doing AttachCurrentThread/GetMethodID/NewLongArray on every chunk instead of once per H5Dchunk_iter call; resolving these once cut per-chunk callback overhead by ~2.8x at large chunk counts.

Notable design notes

H5Dread_chunk guards against a real silent-failure hazard in the underlying H5D__chunk_direct_read(): if the caller's buffer doesn't match the true on-disk chunk size, the C function still returns success but leaves the buffer untouched. Both implementations check this internally and throw IllegalArgumentException on mismatch.
H5Dchunk_iter_all's JNI implementation accumulates chunk info in a callback that is itself pure C (zero JVM crossings per chunk), converting to Java arrays only once at the end — measured ~2x faster than streaming H5Dchunk_iter at ~4000 chunks. The FFM implementation cannot replicate this: FFM's callback is necessarily an upcall stub that crosses back into the JVM on every invocation, so H5Dchunk_iter_all is offered on FFM purely for a simpler call site (no callback to author), not as a performance win — documented in the method's javadoc.
A quick benchmark (TestH5DChunkIterPerf, both trees) comparing H5Dchunk_iter/H5Dchunk_iter_all against a by-index H5Dget_chunk_info loop confirms the same scaling blowup reported in HDF5.jl#1031: the by-index loop is ~150-1400x slower at ~4000 chunks depending on which chunk-enumeration strategy it's compared against.

Test plan

Full TestH5D suite passes in both JNI (40 tests) and FFM (39 tests) builds — the count differs because testH5Dvlen_string_buffer carries @Test + @Ignore in the FFM tree (so JUnit reports it as skipped) but only @Ignore (no @Test) in the JNI tree (so it isn't discovered as a test at all); both trees exercise the same 39 active tests plus this one long-pre-existing disabled case
Each new test verified individually in isolation (JUnit Request.method), not just as part of the full suite
H5Dchunk_iter_all's bulk result cross-checked against per-chunk H5Dget_chunk_info() results, matched by offset (the two APIs aren't guaranteed to visit chunks in the same order)
H5Dread_chunk's buffer-size-mismatch guard exercised directly (expects IllegalArgumentException)
Benchmarked against a pure-C baseline (temporary, not included) to confirm the by-index blowup is a C-library property, not a binding-layer one, and that the streaming-callback overhead fix and bulk variant close most of the JNI-vs-C gap

🤖 Generated with Claude Code

…a bindings Adds native declarations and JNI glue for H5Dchunk_iter (with a new H5D_chunk_iter_cb/H5D_chunk_iter_t callback pair) plus the by-index chunk inspection functions H5Dget_num_chunks and H5Dget_chunk_info, enabling Java callers to enumerate chunks without a per-tool JNI helper. Includes JUnit coverage and a quick benchmark comparing H5Dchunk_iter against a per-index H5Dget_chunk_info loop, mirroring the scaling analysis from JuliaIO/HDF5.jl#1031 (comment).

…a bindings Adds the FFM H5.java wrappers (upcall stub via the jextract-generated H5D_chunk_iter_op_t typedef class) for H5Dchunk_iter, plus H5Dget_num_chunks and H5Dget_chunk_info for by-index chunk inspection, with matching H5D_chunk_iter_cb/H5D_chunk_iter_t callback interfaces. Includes JUnit coverage and the same quick benchmark added to the JNI tree, comparing H5Dchunk_iter against a per-index H5Dget_chunk_info loop per JuliaIO/HDF5.jl#1031 (comment).

Adds H5Dget_chunk_info_by_coord, H5Dget_chunk_storage_size, H5Dget_chunk_index_type, H5Dwrite_chunk, and H5Dread_chunk (calling the current H5Dread_chunk2 symbol) to the JNI tree, rounding out low-level raw chunk access alongside the H5Dchunk_iter/H5Dget_num_chunks/H5Dget_chunk_info added earlier this session. Also adds the five missing H5D_CHUNK_IDX_* HDF5Constants values needed to interpret H5Dget_chunk_index_type's result. H5Dread_chunk guards against a real silent-failure hazard in the underlying H5D__chunk_direct_read(): if the caller's buffer doesn't match the true on-disk chunk size, the C function still returns success but leaves the buffer untouched. Since the Java API doesn't expose the in/out buf_size parameter, the JNI glue checks it internally and throws IllegalArgumentException on a size mismatch instead of silently returning stale data. Includes JUnit coverage for each new function, including the buffer-size mismatch case.

Adds H5Dget_chunk_info_by_coord, H5Dget_chunk_storage_size, H5Dget_chunk_index_type, H5Dwrite_chunk, and H5Dread_chunk (calling the current H5Dread_chunk2 symbol) to the FFM tree, matching the JNI tree added in the same session and rounding out low-level raw chunk access alongside H5Dchunk_iter/H5Dget_num_chunks/H5Dget_chunk_info added earlier. Also adds the five missing H5D_CHUNK_IDX_* HDF5Constants values needed to interpret H5Dget_chunk_index_type's result. H5Dread_chunk guards against a real silent-failure hazard in the underlying H5D__chunk_direct_read(): if the caller's buffer doesn't match the true on-disk chunk size, the C function still returns success but leaves the buffer untouched. Since the Java API doesn't expose the in/out buf_size parameter, the wrapper checks it internally and throws IllegalArgumentException on a size mismatch instead of silently returning stale data. Includes JUnit coverage for each new function, including the buffer-size mismatch case.

…arm-up H5D_chunk_iter_cb was doing three wasteful things on every single chunk callback instead of once per H5Dchunk_iter() call: - AttachCurrentThread/DetachCurrentThread, even though the callback runs synchronously on the same (already-attached) thread that entered Java_hdf_hdf5lib_H5_H5Dchunk_1iter - GetObjectClass + GetMethodID, a name/signature lookup repeated per chunk instead of resolved once - NewLongArray, allocating a fresh JVM heap array per chunk instead of reusing one All three are now resolved/allocated once in Java_hdf_hdf5lib_H5_H5Dchunk_1iter and threaded through the callback wrapper struct. Measured ~2.8x faster at 4096 chunks (3.7ms -> 1.3ms), cutting JNI's overhead relative to an equivalent pure-C H5Dchunk_iter call from ~18x to ~6.5x. Full TestH5D suite (38 tests) still passes. Reusing the same offset array across chunks means it's only valid for the duration of a single callback invocation now (documented in the callback's javadoc) -- this actually brings JNI's contract in line with the FFM callback's MemorySegment, which was always a transient view over native memory. Also adds a warm-up pass to TestH5DChunkIterPerf before timing, so the smallest (most overhead-sensitive) sweep entry isn't dominated by one-time JIT/class-loading cost unrelated to what's being compared.

Adds a warm-up pass to TestH5DChunkIterPerf before timing, so the smallest (most overhead-sensitive) sweep entry isn't dominated by one-time JIT/ class-loading/MethodHandle-linkage cost unrelated to what's being compared. Also adds countByIndexSharedArena(), a diagnostic variant of the by-index loop that calls the raw jextract binding directly with a single reused Arena/MemorySegment set instead of H5.H5Dget_chunk_info()'s one-Arena-per-call cost. This isolated how much of H5Dget_chunk_info's FFM overhead is Arena allocation (real at small chunk counts, ~2-5x) versus the downcall itself (dominant at large chunk counts, where shared-arena and per-call-arena times converge to within ~1-2% of each other and of the equivalent JNI/pure-C numbers) -- confirming the by-index approach's blowup is a C-library property, not a binding-layer one. Documents that H5D_chunk_iter_cb's MemorySegment offset parameter is a transient view over native memory owned by the H5Dchunk_iter call and must not be retained past the callback returning -- this was always true, just not previously stated explicitly.

Adds H5Dchunk_iter_all(dataset_id, dxpl_id), a bulk convenience form of H5Dchunk_iter() that returns every chunk's offset, filter mask, address, and size as a single H5D_chunk_info_t instead of requiring the caller to author a callback. The JNI implementation accumulates chunk info in a plain C callback with zero JVM crossings per chunk (just memcpy into pre-sized native buffers, pre-sized exactly via H5Dget_num_chunks over the dataset's own dataspace), converting to Java arrays only once at the end. Measured ~2x faster than the already-optimized streaming H5Dchunk_iter at ~4000 chunks, and close to the pure-C baseline established earlier this session. H5D_chunk_info_t holds per-chunk fields as parallel primitive arrays rather than an array of per-chunk objects, avoiding one Java allocation per chunk on the way out. Includes JUnit coverage cross-checking the bulk result against the per-chunk H5Dget_chunk_info() results (matched by offset, since the two APIs aren't guaranteed to visit chunks in the same order), and an added column in TestH5DChunkIterPerf comparing all three chunk-enumeration strategies.

Adds H5Dchunk_iter_all(dataset_id, dxpl_id), matching the JNI tree added in the same session: a bulk convenience form of H5Dchunk_iter() returning every chunk's offset, filter mask, address, and size as a single H5D_chunk_info_t instead of requiring the caller to author a callback. Buffers are pre-sized exactly via H5Dget_num_chunks over the dataset's own dataspace, matching the JNI implementation's approach. Unlike the JNI implementation, this is NOT a performance optimization on FFM -- measured slower than the already-optimized streaming H5Dchunk_iter at large chunk counts, and documented as such in the method's javadoc. The C library still invokes a callback once per chunk, and in FFM that callback must be an upcall stub crossing back into the JVM on every invocation -- there is no way to give the native library a callback that runs without JVM involvement using java.lang.foreign alone, unlike JNI where the callback can be plain C. This method pays that same per-chunk upcall cost plus the extra work of copying each chunk's data into the accumulating buffers, so it's offered here purely for a simpler call site (no callback to write), not as a speedup. Includes JUnit coverage cross-checking the bulk result against the per-chunk H5Dget_chunk_info() results (matched by offset), and an added column in TestH5DChunkIterPerf comparing all chunk-enumeration strategies.

github-actions · 2026-07-21T09:16:18Z

Review Checklist

This PR touches the following areas. Each needs a sign-off
from its listed owners before merging.

java

CI's Formatting Check (clang-format 17) flagged alignment/wrapping differences in the chunk-related additions from this branch. Ran clang-format 17 (matching CI exactly, via pixi) over the changed files and committed the result -- purely whitespace, no semantic changes (verified by reviewing every diff and rebuilding + rerunning the full TestH5D suite, which still passes 39/39).

CI's Formatting Check (clang-format 17) flagged alignment/wrapping differences in the chunk-related additions from this branch. Ran clang-format 17 (matching CI exactly, via pixi) over the changed files and committed the result -- purely whitespace, no semantic changes (verified by reviewing every diff; JNI-tree equivalent rebuilt and retested to confirm the formatter didn't alter behavior).

java/src-jni/test/CMakeLists.txt's HDFTEST_COPY_FILE unconditionally requires testfiles/JUnit-<test>.txt to exist as a build dependency for every entry in HDF5_JAVA_TEST_SOURCES, including TestH5DChunkIterPerf added earlier this session. The file was never committed, so any build with BUILD_TESTING=ON failed outright at the ninja/make generation step with "missing and no known rule to make it" -- this is what was breaking CI broadly (not just the Formatting Check fixed in the previous commits). Committed as empty rather than a captured run's output: this benchmark's printed timings are inherently non-deterministic across machines, and COMPARE_TEST (config/cmake/runExecute.cmake) explicitly skips content comparison when the reference file is empty, checking only exit code -- exactly the right behavior here. Verified locally: clean rebuild no longer fails, and ctest reports "COMPARE Result: 0" / test passed.

Same issue as the JNI-tree commit: java/test/CMakeLists.txt's HDFTEST_COPY_FILE requires testfiles/JUnit-TestH5DChunkIterPerf.txt to exist as a build dependency, and it was never committed, breaking any BUILD_TESTING=ON build. Committed empty for the same reason: this benchmark's output is inherently non-deterministic across machines, and an empty reference file makes COMPARE_TEST check only exit code.

The tracked reference predated this session's new TestH5D test methods (testH5Dget_chunk_info_by_coord, testH5Dget_chunk_storage_size, testH5Dget_chunk_index_type, testH5Dwrite_chunk_and_read_chunk, testH5Dchunk_iter, testH5Dchunk_iter_all, testH5Dread_chunk_buffer_size_mismatch), so JUnit4's hash-based MethodSorters.DEFAULT ordering reshuffled the whole class's dot-progress sequence and the final "OK (32 tests)" count no longer matched, breaking JUnit-TestH5D across every CI job that builds Java tests. Regenerated from an actual local run (44/44 JUnit tests, 178/178 broader ctest passing). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Resolves conflict in java/src-jni/test/testfiles/JUnit-TestH5D.txt: develop added testH5DArray_datatype_ids_stable while this branch added several chunk-related tests. TestH5D.java itself auto-merged cleanly (both sets of new test methods are independent). Regenerated the reference file from an actual local run (40 tests total: 32 pre-existing + 1 from develop + 7 new chunk tests from this branch), verified via ctest (44/44 JUnit tests pass). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Same root cause as the JNI-side fix: the tracked reference predated this session's new TestH5D test methods (testH5Dget_chunk_info_by_coord, testH5Dget_chunk_index_type, testH5Dchunk_iter_all, testH5Dchunk_iter, testH5Dwrite_chunk_and_read_chunk, testH5Dget_chunk_storage_size, testH5Dread_chunk_buffer_size_mismatch), so JUnit4's hash-based MethodSorters.DEFAULT ordering reshuffled the whole class's dot-progress sequence and the "OK (32 tests)" count no longer matched. Regenerated from an actual scoped local FFM build/run (JDK 25 + jextract; 44/44 JUnit tests passing), matching the CI failure output exactly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

mkitti · 2026-07-22T06:34:40Z

`H5Dchunk_iter_all`: what it is, why it exists, and JNI vs FFM performance

What it is

H5Dchunk_iter_all is a convenience method (added in both the JNI and FFM
trees) that wraps H5Dchunk_iter to enumerate every chunk of a dataset
in one call and return the results as plain Java arrays, instead of requiring
the caller to write a callback:

H5D_chunk_info_t info = H5.H5Dchunk_iter_all(dataset_id, dxpl_id);
// info.getNumChunks(), info.getOffset(chunkIndex, dim), info.filterMask, info.addr, info.size

Why it's necessary

Before this PR, there were two ways to enumerate a dataset's chunks from
Java:

H5Dget_chunk_info in a loop over 0..H5Dget_num_chunks-1 — simple
array-based ergonomics, but algorithmically poor: each call does an
independent chunk-index lookup, so enumerating all chunks this way costs
O(n²) (or worse — see the indx_ms column below, which blows up
super-linearly with a v1 B-tree index). This is exactly the pathology
reported for HDF5.jl
that motivated this PR's benchmark in the first place.
H5Dchunk_iter directly — the C library streams every chunk with a
single index traversal (O(n) total), but only via a callback, which is
awkward for the common "just give me all the chunks" case and (for FFM
specifically) has real per-chunk crossing cost — see below.

H5Dchunk_iter_all closes that gap: array-based ergonomics and the
library's linear-time traversal, by driving H5Dchunk_iter internally and
accumulating into arrays for the caller.

Performance: methodology

Benchmarked with TestH5DChunkIterPerf.java (both trees): a chunked
2-D dataset with a 16×16 chunk shape, swept over 32×32 through
1024×1024 (4 to 4096 chunks), default library version bounds (v1 B-tree
chunk index — see the java/CLAUDE.md note on chunk index defaults), one
untimed warm-up dataset run first to absorb JIT/class-loading/native-linkage
cost, then each approach timed via System.nanoTime(). Two independent runs
each; numbers below are the average (they agreed within noise on both
trees). All times in milliseconds.

Results

chunks	JNI `iter_ms` (streaming)	JNI `all_ms` (bulk)	FFM `iter_ms` (streaming)	FFM `all_ms` (bulk)	`indx_ms` (by-index loop, either tree)†
4	0.012	0.011	0.377	0.543	~0.02
16	0.022	0.016	0.488	0.728	~0.05
64	0.036	0.019	0.531	1.470	~0.28
256	0.130	0.057	1.829	4.426	~7
1024	0.308	0.155	2.276	8.166	~50
4096	1.168	0.535	4.089	11.819	~700

† indx_ms (looping H5Dget_chunk_info by index) is shown once since it's
dominated by the C-library algorithm, not the binding layer — both trees
match to within normal run-to-run noise. It grows super-linearly and is
already ~130-600× slower than either chunk-iterate approach by 4096 chunks,
confirming this is a library/index-structure effect, not a JNI/FFM artifact.

Interpretation

JNI: H5Dchunk_iter_all is a genuine speedup, ~1.9-2.3× faster than
streaming H5Dchunk_iter from a few hundred chunks up. This is real and
architectural: JNI's callback trampoline is plain C, so the bulk variant's
callback (H5D_chunk_iter_all_cb) does zero JNI/JVM crossings per chunk —
just a native accumulate — with all n chunks converted to Java arrays
once, at the end, instead of once per chunk.
FFM: H5Dchunk_iter_all is slower than streaming H5Dchunk_iter,
by about 1.4-3.6× depending on size — the opposite direction from JNI.
This is also architectural, not a missed optimization: in FFM, the
callback handed to the C library must be a Java upcall stub (there is
no way to give H5Dchunk_iter a pure-native callback via
java.lang.foreign alone), so it pays the same per-chunk JVM-crossing
cost as streaming H5Dchunk_iter, plus extra per-chunk work: the
mandatory .reinterpret() on the zero-length offset MemorySegment
jextract hands the upcall, plus a MemorySegment.copy and three
setAtIndex writes to stash each chunk's data before the final bulk
conversion. There's no callback-side win available to claim back.
FFM is slower than JNI in absolute terms for both variants, as
expected for a downcall/upcall-per-chunk cost model vs. plain JNI, with
the gap most pronounced for the bulk variant (~22× at 4096 chunks) since
that's exactly where FFM's structural overhead compounds and JNI's does
not.
Despite FFM's H5Dchunk_iter_all not being a perf win there, it's kept in
both trees for API symmetry/ergonomics — it's documented in the FFM
javadoc and java/CLAUDE.md as ergonomics-only on that side, not a
performance claim.
The actually load-bearing result for the original HDF5.jl-style
concern is chunk-iterate (either variant, either tree) vs. by-index
looping, not JNI vs. FFM — that gap (700ms vs ~1ms at 4096 chunks) is the
one that matters for any caller enumerating a large number of chunks.

Implementation: 24e9353 (JNI), 96d25ed (FFM). Benchmark source:
java/src-jni/test/TestH5DChunkIterPerf.java /
java/test/TestH5DChunkIterPerf.java.

Copilot

Pull request overview

This PR expands the HDF5 Java bindings (both java/src-jni JNI and java/hdf FFM/Panama trees) to expose chunk-level dataset APIs, including chunk iteration, chunk metadata queries, and direct raw chunk I/O, with accompanying tests and a JNI callback-overhead optimization.

Changes:

Adds Java bindings for chunk enumeration (H5Dchunk_iter, H5Dchunk_iter_all) and chunk metadata queries (H5Dget_num_chunks, H5Dget_chunk_info*, H5Dget_chunk_storage_size, H5Dget_chunk_index_type).
Adds direct raw chunk I/O APIs (H5Dwrite_chunk, H5Dread_chunk) plus missing H5D_CHUNK_IDX_* constants.
Adds new/updated JUnit tests (including a perf/diagnostic benchmark test) and integrates them into the Java test CMake targets.

Reviewed changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
java/test/TestH5DChunkIterPerf.java	Adds an FFM-side JUnit benchmark/diagnostic test comparing chunk enumeration strategies.
java/test/TestH5D.java	Adds FFM-side functional tests for chunk iteration, bulk iteration, metadata queries, and direct chunk I/O.
java/test/testfiles/JUnit-TestH5DChunkIterPerf.txt	Adds reference file (empty to avoid stdout comparison) for the new perf test.
java/test/testfiles/JUnit-TestH5D.txt	Updates expected JUnit output to include the newly added TestH5D tests.
java/test/CMakeLists.txt	Registers `TestH5DChunkIterPerf` in the FFM Java test build/run list.
java/src-jni/test/TestH5DChunkIterPerf.java	Adds a JNI-side JUnit benchmark test comparing chunk enumeration strategies.
java/src-jni/test/TestH5D.java	Adds JNI-side functional tests for chunk iteration, bulk iteration, metadata queries, and direct chunk I/O.
java/src-jni/test/testfiles/JUnit-TestH5DChunkIterPerf.txt	Adds reference file (empty to avoid stdout comparison) for the new perf test.
java/src-jni/test/testfiles/JUnit-TestH5D.txt	Updates expected JUnit output to include the newly added TestH5D tests.
java/src-jni/test/CMakeLists.txt	Registers `TestH5DChunkIterPerf` in the JNI Java test build/run list.
java/src-jni/jni/h5dImp.h	Declares new JNI entry points for chunk iteration, chunk metadata queries, and direct chunk I/O.
java/src-jni/jni/h5dImp.c	Implements new JNI bindings, adds optimized chunk-iter callback handling, and implements bulk chunk iteration accumulation.
java/src-jni/jni/h5Constants.c	Exposes additional `H5D_CHUNK_IDX_*` constants to the JNI constants layer.
java/src-jni/hdf/hdf5lib/structs/H5D_chunk_info_t.java	Adds JNI-side `H5D_chunk_info_t` struct wrapper for bulk chunk-iteration results.
java/src-jni/hdf/hdf5lib/HDF5Constants.java	Adds new public `H5D_CHUNK_IDX_*` constants (JNI tree).
java/src-jni/hdf/hdf5lib/H5.java	Adds JNI method declarations and Javadoc for new chunk-level APIs.
java/src-jni/hdf/hdf5lib/CMakeLists.txt	Adds new callback interfaces and struct wrapper to the JNI Java build sources.
java/src-jni/hdf/hdf5lib/callbacks/H5D_chunk_iter_t.java	Adds JNI callback operator-data marker interface for chunk iteration.
java/src-jni/hdf/hdf5lib/callbacks/H5D_chunk_iter_cb.java	Adds JNI callback interface for chunk iteration.
java/hdf/hdf5lib/structs/H5D_chunk_info_t.java	Adds FFM-side `H5D_chunk_info_t` struct wrapper for bulk chunk-iteration results.
java/hdf/hdf5lib/HDF5Constants.java	Adds new public `H5D_CHUNK_IDX_*` constants (FFM tree).
java/hdf/hdf5lib/H5.java	Adds FFM wrappers for chunk iteration, bulk iteration, chunk metadata queries, and direct chunk I/O.
java/hdf/hdf5lib/CMakeLists.txt	Adds new callback interfaces and struct wrapper to the FFM Java build sources.
java/hdf/hdf5lib/callbacks/H5D_chunk_iter_t.java	Adds FFM callback operator-data marker interface for chunk iteration.
java/hdf/hdf5lib/callbacks/H5D_chunk_iter_cb.java	Adds FFM callback interface for chunk iteration (upcall stub).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+/**
+ * Data class for link callback for H5Dchunk_iter.
+ *
+ */


+/**
+ * Data class for link callback for H5Dchunk_iter.
+ *
+ */


+/**
+ * Information class for link callback for H5Dchunk_iter.
+ *
+ */


+/**
+ * Information class for link callback for H5Dchunk_iter.
+ *
+ */


+            assertEquals("testH5Dchunk_iter_all: H5Dget_num_chunks agrees", info.getNumChunks(),
+                         nchunks_by_index);
+
+            for (int i = 0; i < nchunks_by_index; i++) {


+            assertEquals("testH5Dchunk_iter_all: H5Dget_num_chunks agrees", info.getNumChunks(),
+                         nchunks_by_index);
+
+            for (int i = 0; i < nchunks_by_index; i++) {


 Time:  XXXX

-OK (33 tests)
+OK (40 tests)


+     * invoked from a library-created worker thread). */
+    JNIEnv *cbenv          = wrapper->env;
+    jobject visit_callback = wrapper->visit_callback;
+    void   *op_data        = (void *)wrapper->op_data;


- Fix copy/paste Javadoc ("link callback" -> chunk iterator callback) in both trees' H5D_chunk_iter_t/H5D_chunk_iter_cb interfaces - Use a long loop counter in TestH5D's by-index chunk-info cross-check to avoid truncating a chunk count above Integer.MAX_VALUE - Keep op_data as jobject in H5D_chunk_iter_cb (JNI) instead of casting it to void* before passing it to CallIntMethod

mkitti · 2026-07-23T01:44:12Z

I do not think the one CI failure is related to my changes.

mkitti · 2026-08-05T00:01:01Z

After thinking about H5Dchunk_iter_all I wonder if this should be implemented in such a way that other bindings could use it.

github-actions · 2026-09-04T06:23:20Z

This pull request has had no activity for 30 days and has been marked stale. Push a commit or comment to keep it open, or it will be flagged for maintainer review.

mkitti · 2026-09-04T17:38:41Z

@jhendersonHDF any thoughts about this? I am thinking of splititng this this apart and moving H5Dchunk_iter_all to its own pull rqeuest.

jhendersonHDF

Hi @mkitti,

I haven't really had a chance to look over this and will be out for the next two weeks, but I'll leave a couple initial comments below.

As far as H5Dchunk_iter_all(), it's something that might be easier to review in a separate PR; the wrappers over existing C functions are easy enough to review and would make a fairly small PR. My thought on H5Dchunk_iter_all() is that it's something that might become unnecessary eventually. I wouldn't mind having it in the Java bindings, but we've had requests in the past to consider switching over to an iterator based approach for functions that currently take callbacks. If we do so, that would likely get the performance benefits of H5Dchunk_iter_all(), avoiding the callback overhead, and make it fairly straightforward to implement something close to an H5Dchunk_iter_all() but potentially in a more flexible way. For large datasets, I'm thinking something like batched blocks of retrievals instead of allocating all the memory upfront.

jhendersonHDF · 2026-09-04T17:51:41Z

    /** */
    public static final int H5D_CHUNK_IDX_BTREE = H5D_CHUNK_IDX_BTREE();
    /** */
+    public static final int H5D_CHUNK_IDX_SINGLE = H5D_CHUNK_IDX_SINGLE();


I would consider dropping these unless they're necessary. They're really only useful for the H5Dget_chunk_index_type() function and that is meant to be an internal API (if one can even call an API internal). It's not really a function meant to be called or used for anything.

jhendersonHDF · 2026-09-04T18:25:54Z

This seems useful enough, but since there isn't anything actually being tested for correctness here it should probably be part of some performance testing framework that reports to a dashboard rather than the main Java testing. Otherwise, it's likely that any change in results will just get missed.

mkitti added 8 commits July 20, 2026 20:01

github-project-automation Bot added this to HDF5 - TRIAGE & TRACK Jul 21, 2026

github-project-automation Bot moved this to To be triaged in HDF5 - TRIAGE & TRACK Jul 21, 2026

mkitti and others added 7 commits July 21, 2026 13:04

Merge branch 'develop' into h5dchunk_java

c8609f3

mkitti marked this pull request as ready for review July 22, 2026 06:39

mkitti requested a review from jhendersonHDF as a code owner July 22, 2026 06:39

Copilot AI review requested due to automatic review settings July 22, 2026 06:39

mkitti requested a review from mattjala as a code owner July 22, 2026 06:39

Copilot started reviewing on behalf of mkitti July 22, 2026 06:41 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

github-actions Bot removed the request for review from mattjala July 22, 2026 21:19

Merge remote-tracking branch 'origin/develop' into h5dchunk_java

58d84d9

brtnfld assigned jhendersonHDF Aug 18, 2026

hyoklee added this to the Backlog milestone Aug 28, 2026

github-actions Bot added the stale label Sep 4, 2026

jhendersonHDF reviewed Sep 4, 2026

View reviewed changes

github-actions Bot removed the stale label Sep 5, 2026

Navigation Menu

Uh oh!

Uh oh!

Expose chunk-level HDF5 APIs (H5Dchunk_iter, direct chunk I/O) to Java (JNI + FFM) - #6547

Expose chunk-level HDF5 APIs (H5Dchunk_iter, direct chunk I/O) to Java (JNI + FFM)#6547
mkitti wants to merge 18 commits into
HDFGroup:developfrom
mkitti:h5dchunk_java

mkitti commented Jul 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

mkitti commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mkitti commented Jul 23, 2026

Uh oh!

mkitti commented Aug 5, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

mkitti commented Sep 4, 2026

Uh oh!

jhendersonHDF left a comment

Uh oh!

jhendersonHDF Sep 4, 2026

Uh oh!

jhendersonHDF Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

mkitti commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notable design notes

Test plan

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review Checklist

Uh oh!

mkitti commented Jul 22, 2026

H5Dchunk_iter_all: what it is, why it exists, and JNI vs FFM performance

What it is

Why it's necessary

Performance: methodology

Results

Interpretation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mkitti commented Jul 23, 2026

Uh oh!

mkitti commented Aug 5, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

mkitti commented Sep 4, 2026

Uh oh!

jhendersonHDF left a comment

Choose a reason for hiding this comment

Uh oh!

jhendersonHDF Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

jhendersonHDF Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mkitti commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

`H5Dchunk_iter_all`: what it is, why it exists, and JNI vs FFM performance