Hardening for RSA-PSS verify, AES-GMAC init, JNI exceptions, debug logging

[cconlon]

This PR fixes 5 Fenrir issues:

• F-12165: validate the AES-GMAC tag length at Mac.init() instead of silently truncating
• F-12166: preserve pending JNI exceptions and never use an unresolved field ID in getNativeStruct()
• F-12167: RSA-PSS verify returns false for an invalid signature instead of throwing
• F-12168: correct the OCSP-response success-sentinel doc comment (WOLFSSL_SUCCESS is 1, not 0)
• F-12189: stop dumping keys, plaintext, secrets, and RNG output via debug LogHex in FIPS and non-FIPS paths

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Hardening changes across JCE/JNI and tests to make crypto behavior safer and more predictable (RSA-PSS verify semantics, AES-GMAC tag handling, JNI exception safety, and removal of sensitive debug hex dumps).

Changes:

• Add tests and implementation changes so RSA-PSS verify returns false for invalid signatures rather than throwing.
• Validate AES-GMAC tag length at Mac.init() and ensure rejected init does not mutate existing MAC state.
• Preserve pending JNI exceptions / avoid using unresolved field IDs, fix an OCSP doc comment sentinel, and remove sensitive LogHex dumps.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/test/java/com/wolfssl/wolfcrypt/test/RsaTest.java	Extends RSA-PSS tests for corrupted signatures and salt-length discovery behavior.
src/test/java/com/wolfssl/provider/jce/test/WolfCryptMacTest.java	Adds AES-GMAC tag-length validation and “rejected init preserves state” tests.
src/main/java/com/wolfssl/wolfcrypt/Rsa.java	Updates RSA-PSS verify Javadoc to reflect new error/return semantics.
src/main/java/com/wolfssl/provider/jce/WolfCryptMac.java	Validates GMAC tag length before mutating fields and avoids silent truncation.
jni/jni_wolfssl_cert_manager.c	Corrects OCSP-response success sentinel in the doc comment.
jni/jni_rsa.c	Adjusts RSA-PSS verify JNI behavior to map invalid signature failures to false.
jni/jni_rng.c	Removes debug hex dump of RNG output.
jni/jni_native_struct.c	Preserves pending JNI exceptions and avoids unresolved jfieldID usage.
jni/jni_hmac.c	Removes debug hex dumps of HMAC input/output.
jni/jni_fips.c	Removes debug hex dumps of sensitive material (keys/plaintext/secrets/RNG output) in FIPS paths.
jni/jni_ecc.c	Removes debug hex dumps of ECC private/export/shared-secret outputs.
jni/jni_dh.c	Removes debug hex dumps of DH private/secret material.
jni/jni_des3.c	Removes debug hex dumps of DES3 input/output buffers.
jni/jni_aescmac.c	Removes debug hex dumps of CMAC inputs/outputs and verify MAC bytes.
jni/jni_aes.c	Removes debug hex dumps of AES input/output buffers.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.