Skip to content

fix: cap byte[] allocations at MAX_ARRAY_LENGTH (Integer.MAX_VALUE - 8) - #163

Merged
bernardladenthin merged 3 commits into
mainfrom
fix/max-array-length
Sep 30, 2026
Merged

bernardladenthin merged 3 commits into
mainfrom
fix/max-array-length

Conversation

@bernardladenthin

Copy link
Copy Markdown
Owner

Summary

  • Fix: with more than 2 GB buffered, trim() allocated new byte[Integer.MAX_VALUE]. HotSpot always rejects Integer.MAX_VALUE and Integer.MAX_VALUE - 1 ("Requested array size exceeds VM limit"), regardless of heap size (verified on JDK 21).
  • New MAX_ARRAY_LENGTH = Integer.MAX_VALUE - 8, the same value as jdk.internal.util.ArraysSupport.SOFT_MAX_ARRAY_LENGTH. It is mirrored because that class is JDK-internal and absent on Java 8. maxAllocationSize defaults to it, and setMaxAllocationSize clamps larger values to it.
  • Visible change: getMaxAllocationSize() now defaults to Integer.MAX_VALUE - 8.
  • Docs: OpenJML findings, and links to the upstream fix for the Atomic*.jml RAC IllegalAccessError. The rm workaround in setup-openjml stays until an OpenJML release ships the fix.

Test plan

  • Affected unit / integration tests pass locally (mvn test, 9 new test cases: constant value, default, clamp boundaries at MAX_ARRAY_LENGTH ± 1, Integer.MAX_VALUE - 1, Integer.MAX_VALUE, Long.MAX_VALUE)
  • PIT: 100 %, no survivors
  • Spotless and Javadoc (failOnWarnings, doclint all) clean
  • OpenJML ESC: 14/14 clean proofs; RAC suite 285/285 green
  • CI is green on this branch
  • Docs / CHANGELOG updated where applicable

Related issues / PRs

Refs OpenJML/Specs#29, OpenJML/OpenJML#982

Checklist

  • I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
  • My commits follow Conventional Commits
  • No security-sensitive changes (if there are, I have notified the maintainer privately per SECURITY.md)

🤖 Generated with Claude Code

https://claude.ai/code/session_018mLGtytvtJU7aNshBpnrB5

bernardladenthin and others added 3 commits September 30, 2026 23:25
HotSpot rejects byte[] of Integer.MAX_VALUE and Integer.MAX_VALUE - 1
("Requested array size exceeds VM limit") on any heap size. With more
than 2 GB buffered, trim() allocated exactly such an array.

- MAX_ARRAY_LENGTH mirrors jdk.internal.util.ArraysSupport
  .SOFT_MAX_ARRAY_LENGTH (JDK-internal, absent on Java 8).
- maxAllocationSize defaults to it; setMaxAllocationSize clamps above it.
- Tests: constant value, default, clamp boundaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018mLGtytvtJU7aNshBpnrB5
…OpenJML#982

- Invariant RAC crash ("no enclosing instance") not minimally
  reproducible; invariants stay omitted defensively.
- Atomic*.jml RAC IllegalAccessError reported upstream; the rm in
  setup-openjml stays until an OpenJML release ships the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018mLGtytvtJU7aNshBpnrB5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018mLGtytvtJU7aNshBpnrB5
@bernardladenthin
bernardladenthin merged commit 32dd672 into main Sep 30, 2026
11 of 16 checks passed
@bernardladenthin
bernardladenthin deleted the fix/max-array-length branch September 30, 2026 21:26

This branch had an error being deployed

1 failed deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant