openssl compat: cut EVP, EVP_PKEY and PEM overhead, and fix defects found

Performance:

 - Size WOLFSSL_SHA3_CTX's holder in pointers, not bytes. It reserved
   sizeof(wc_Sha3) pointers for a 440 byte object, so WOLFSSL_Hasher goes
   3520 -> 448 bytes and WOLFSSL_EVP_MD_CTX 3552 -> 944. A static assert now
   catches the same mistake.
 - Resolve digests and ciphers by table pointer before comparing names, and
   return the table's own pointer from the accessors. EVP_sha256()
   47.4 -> 1.2ns, EVP_MD_type 64.7 -> 4.3ns, EVP_CIPHER_nid 101.2 -> 8.1ns.
 - Copy only the fields past the hash union in EVP_MD_CTX_copy_ex, and leave
   the cipher union alone in EVP_CIPHER_CTX_init.
 - Set up an EVP_PKEY's RNG, an RSA key's blinding RNG and an EVP_PKEY's DER
   encoding on first use rather than at creation. Seeding a DRBG cost more
   than everything else about making a key put together.
   EVP_PKEY_new + free 13366 -> 34ns, RSA_new + free 13748 -> 230ns,
   EVP_PKEY_set1_RSA 13994 -> 45ns. Certificate parsing roughly halves as a
   result: d2i_X509 45978 -> 17198ns, X509_dup 45945 -> 17116ns, both now
   faster than OpenSSL. set1_DSA still encodes immediately, as WOLFSSL_DSA is
   not reference counted and the pkey only borrows the caller's key.
 - Keep the decoded public key on the X509 and hand out a reference, as
   OpenSSL does. X509_get_pubkey + free 27692 -> 9.8ns, against 9.6ns.
 - Give public key operations the shared global RNG instead of seeding one per
   call, where the instance locks itself or the build is single threaded.
   WOLFSSL_PK_LOCAL_RNG forces an RNG per call. EVP_DigestSign with ECDSA
   P-256 32291 -> 17008ns, against 18852ns.
 - Encode base64 without a call per output character, and hand whole blocks to
   Base64_Encode() rather than one 48 byte line at a time. EVP_EncodeUpdate
   6446 -> 1855ns per 4KB. Skip the call into Base64_SkipNewline() for
   characters needing no skipping: EVP_DecodeUpdate 37059 -> 16194ns. What
   remains there is the constant time character decode, which is deliberate;
   wc_PemToDer() already picks the table decoder by PEM type.
 - Search cipher_tbl before the alias table in EVP_get_cipherbyname(), since
   over half the aliases only spell a name already there in another case, and
   compare without the platform's strcasecmp: 330 -> 67ns. The comparison is
   ASCII only, so it no longer follows the locale and works on platforms whose
   XSTRCASECMP is strcmp.
 - Find the PEM footer in a memory BIO's buffer and read up to it, rather than
   reading a byte at a time through the whole input. Loading a certificate
   from a file that carries a text dump alongside it 100311 -> 23301ns,
   against 33006ns.

Fixes:

 - AES-CCM through EVP now behaves as OpenSSL's does: the payload length comes
   from the first Update() carrying payload, AAD before the length is refused,
   and Final() has nothing left to do. Of 376 observables compared against
   OpenSSL - the tag and IV length matrices, the defaults, length inference
   over 48 sizes, and an encrypt/decrypt matrix with the tag then corrupted -
   375 agree. The one that differs, EVP_CIPHER_iv_length() reporting
   CCM_NONCE_MIN_SZ where OpenSSL reports 12, is unchanged by this work.
 - EVP_get_cipherbyname(NULL) and EVP_get_digestbyname(NULL) dereferenced the
   name; OpenSSL returns NULL.
 - EVP_EncodeUpdate() wrote the block it was holding and then wrote over it,
   so a caller feeding data in pieces got wrong bytes, including bytes never
   written.
 - EVP_PKEY_keygen() on DH read the parameters after replacing the key object
   they came from, which is the same object when a caller passes its own key
   back in.
 - EVP_PKEY_set1_EC_KEY() with an empty EC_KEY now succeeds. It reported
   failure only as a side effect of encoding eagerly; OpenSSL returns 1 and
   the empty key is caught when the encoding is asked for.
 - Remove three GMULT macros naming functions that do not exist.

Portability:

 - Add XMEMCHR, with wc_memchr() for builds naming their own string functions
   through STRING_USER that have no memchr among them.

Tests:

 - Known answer vectors for the base64 encoders, generated independently of
   wolfSSL. testwolfcrypt passed with the encoder deliberately corrupted, as
   it only checked return codes.
 - EVP_EncodeUpdate() over every chunking of ten sizes straddling the 48 byte
   block, and PEM_read_bio_X509() reading several certificates from one BIO
   while leaving nothing behind after one.
 - EVP_get_cipherbyname() over canonical names, alternative spellings, every
   casing of each, and NULL.
 - One RSA private key operation per freshly decoded key: sharing a key lets
   the first operation provide the blinding RNG and hides a missing one in the
   rest. Plus EVP_PKEY_set1_DSA() followed by the caller releasing its key.
 - The reference counting behind the cached X509 public key: outliving the
   certificate and the caller, replacement, and duplication.
 - wc_memchr(), guarded by USE_WOLF_MEMCHR so it runs where that function is
   built.

Author: SparkiDev

src/internal.c

@@ -5750,6 +5750,9 @@ static void FreeX509Contents(WOLFSSL_X509* x509)
         XFREE(x509->pubKey.buffer, x509->heap, DYNAMIC_TYPE_PUBLIC_KEY);
         x509->pubKey.buffer = NULL;
     }
+#ifdef WOLFSSL_X509_PUBKEY_CACHE
+    wolfssl_x509_free_pubkey_evp(x509);
+#endif
     FreeDer(&x509->derCert);
     XFREE(x509->sig.buffer, x509->heap, DYNAMIC_TYPE_SIGNATURE);
     x509->sig.buffer = NULL;

src/pk.c

@@ -6292,6 +6292,10 @@ int wolfSSL_PEM_write_bio_PrivateKey(WOLFSSL_BIO* bio, WOLFSSL_EVP_PKEY* key,
         #ifndef NO_DH
             case WC_EVP_PKEY_DH:
                 /* Write using generic API with DH type. */
+                if (EvpPkeyEnsureDer(key) != WOLFSSL_SUCCESS) {
+                    ret = WOLFSSL_FAILURE;
+                    break;
+                }
                 ret = der_write_to_bio_as_pem((byte*)key->pkey.ptr,
                     key->pkey_sz, bio, DH_PRIVATEKEY_TYPE);
                 break;
@@ -7469,6 +7473,8 @@ int pkcs8_encrypt(WOLFSSL_EVP_PKEY* pkey,
 #endif /* WOLFSSL_HAVE_MLDSA */
             {
                 /* Encrypt private into buffer. */
+                if (EvpPkeyEnsureDer(pkey) != WOLFSSL_SUCCESS)
+                    return WOLFSSL_FAILURE;
                 ret = TraditionalEnc(
                     (byte*)pkey->pkey.ptr + pkey->pkcs8HeaderSz,
                     (word32)pkey->pkey_sz - pkey->pkcs8HeaderSz,
@@ -7535,6 +7541,9 @@ int pkcs8_encode(WOLFSSL_EVP_PKEY* pkey, byte* key, word32* keySz)
             if (keySz == NULL)
                 return BAD_FUNC_ARG;
 
+            if (EvpPkeyEnsureDer(pkey) != WOLFSSL_SUCCESS)
+                return BAD_FUNC_ARG;
+
             if (key == NULL) {
                 *keySz = (word32)pkey->pkey_sz;
                 return LENGTH_ONLY_E;
@@ -7614,6 +7623,9 @@ int pkcs8_encode(WOLFSSL_EVP_PKEY* pkey, byte* key, word32* keySz)
         ret = NOT_COMPILED_IN;
     }
 
+    if ((ret >= 0) && (EvpPkeyEnsureDer(pkey) != WOLFSSL_SUCCESS)) {
+        ret = BAD_FUNC_ARG;
+    }
     if (ret >= 0) {
         /* Encode private key in PKCS#8 format. */
         ret = wc_CreatePKCS8Key(key, keySz, (byte*)pkey->pkey.ptr +

src/pk_rsa.c

@@ -236,6 +236,130 @@ void wolfSSL_RSA_free(WOLFSSL_RSA* rsa)
  * @return  RSA key on success.
  * @return  NULL on failure.
  */
+#if !defined(HAVE_FIPS) && defined(WC_RSA_BLINDING)
+/* Serialises the one time set up of a key's blinding RNG, so that two threads
+ * arriving at a private key operation together cannot both create one and
+ * leave the loser's leaked. */
+static wolfSSL_Mutex rsaRngMutex
+    WOLFSSL_MUTEX_INITIALIZER_CLAUSE(rsaRngMutex);
+#ifdef WOLFSSL_HAVE_RSA_RNG_MUTEX
+static int rsaRngMutexValid = 0;
+
+/* Create the lock serialising blinding RNG set up.
+ *
+ * Called from wolfSSL_Init(), which is already serialised against itself, so
+ * that no caller has to create the mutex on first use. */
+int wolfssl_rsa_rng_mutex_init(void)
+{
+    if (rsaRngMutexValid == 0) {
+        if (wc_InitMutex(&rsaRngMutex) != 0) {
+            WOLFSSL_MSG("Bad Init Mutex rsa rng");
+            return BAD_MUTEX_E;
+        }
+        rsaRngMutexValid = 1;
+    }
+    return 0;
+}
+
+/* Destroy the lock serialising blinding RNG set up. Called from
+ * wolfSSL_Cleanup(). */
+void wolfssl_rsa_rng_mutex_free(void)
+{
+    if (rsaRngMutexValid == 1) {
+        (void)wc_FreeMutex(&rsaRngMutex);
+        rsaRngMutexValid = 0;
+    }
+}
+#endif /* WOLFSSL_HAVE_RSA_RNG_MUTEX */
+#endif /* !HAVE_FIPS && WC_RSA_BLINDING */
+
+/* Provide the RNG that blinds private key operations, creating it if this is
+ * the first time it has been needed.
+ *
+ * Deferred rather than done in wolfSSL_RSA_new_ex() because seeding a DRBG
+ * costs more than everything else about creating the key put together, and
+ * public key work - verifying a signature, carrying a certificate's public
+ * key - never reaches this.
+ *
+ * @param [in, out] rsa  RSA key.
+ * @return  The RNG set on the wolfCrypt key on success.
+ * @return  NULL when no RNG could be provided.
+ */
+WC_RNG* wolfssl_rsa_ensure_rng(WOLFSSL_RSA* rsa)
+{
+#if !defined(HAVE_FIPS) && defined(WC_RSA_BLINDING)
+    WC_RNG* rng;
+    int haveMutex;
+
+    if ((rsa == NULL) || (rsa->internal == NULL)) {
+        return NULL;
+    }
+
+    /* rngInited is read under the lock, not before it: an unlocked read would
+     * race with another thread publishing the flag, and it shares a storage
+     * unit with the other bits of the key. Taking the lock on the already set
+     * up path costs nothing beside the private key operation that follows. */
+#ifdef WOLFSSL_HAVE_RSA_RNG_MUTEX
+    /* wolfSSL_Init() creates it; without that there is nothing to lock, and
+     * the same single threaded assumption applies as to the global RNG. */
+    haveMutex = (rsaRngMutexValid == 1);
+#else
+    haveMutex = 1;
+#endif
+    if (haveMutex && (wc_LockMutex(&rsaRngMutex) != 0)) {
+        WOLFSSL_MSG("Bad Lock Mutex rsa rng");
+        return NULL;
+    }
+
+    /* Already provided; hand back what the wolfCrypt key is holding. */
+    if (rsa->rngInited) {
+        rng = ((RsaKey*)rsa->internal)->rng;
+        if (haveMutex) {
+            wc_UnLockMutex(&rsaRngMutex);
+        }
+        return rng;
+    }
+
+    /* Create a local RNG. */
+    rng = (WC_RNG*)XMALLOC(sizeof(WC_RNG), rsa->heap, DYNAMIC_TYPE_RNG);
+    if ((rng != NULL) && (wc_InitRng_ex(rng, rsa->heap, rsa->devId) != 0)) {
+        WOLFSSL_MSG("InitRng failure, attempting to use global RNG");
+        XFREE(rng, rsa->heap, DYNAMIC_TYPE_RNG);
+        rng = NULL;
+    }
+
+    rsa->ownRng = 1;
+    if (rng == NULL) {
+        /* Get the wolfSSL global RNG - not thread safe. */
+        rng = wolfssl_get_global_rng();
+        rsa->ownRng = 0;
+    }
+    if (rng == NULL) {
+        /* Couldn't create global either. */
+        WOLFSSL_ERROR_MSG("wolfSSL_RSA no WC_RNG for blinding");
+        if (haveMutex) {
+            wc_UnLockMutex(&rsaRngMutex);
+        }
+        return NULL;
+    }
+
+    /* Set the local or global RNG into the wolfCrypt RSA key. */
+    (void)wc_RsaSetRNG((RsaKey*)rsa->internal, rng);
+    /* Won't fail as key and rng are not NULL. */
+    rsa->rngInited = 1;
+
+    if (haveMutex) {
+        wc_UnLockMutex(&rsaRngMutex);
+    }
+
+    return rng;
+#else
+    (void)rsa;
+    return NULL;
+#endif /* !HAVE_FIPS && WC_RSA_BLINDING */
+}
+
+
 WOLFSSL_RSA* wolfSSL_RSA_new_ex(void* heap, int devId)
 {
     WOLFSSL_RSA* rsa = NULL;
@@ -285,37 +409,14 @@ WOLFSSL_RSA* wolfSSL_RSA_new_ex(void* heap, int devId)
             rsaKeyInited = 1;
         }
     }
-    #if !defined(HAVE_FIPS) && defined(WC_RSA_BLINDING)
     if (!err) {
-        WC_RNG* rng;
+        /* The RNG used to blind private key operations is not set up here.
+         * Seeding it dominates the cost of creating an RSA object, and a key
+         * that is only ever used to verify a signature or to carry a public
+         * key never needs it. wolfssl_rsa_ensure_rng() provides it on the
+         * first private key operation instead. */
+        rsa->devId = devId;
 
-        /* Create a local RNG. */
-        rng = (WC_RNG*)XMALLOC(sizeof(WC_RNG), heap, DYNAMIC_TYPE_RNG);
-        if ((rng != NULL) && (wc_InitRng_ex(rng, heap, devId) != 0)) {
-            WOLFSSL_MSG("InitRng failure, attempting to use global RNG");
-            XFREE(rng, heap, DYNAMIC_TYPE_RNG);
-            rng = NULL;
-        }
-
-        rsa->ownRng = 1;
-        if (rng == NULL) {
-            /* Get the wolfSSL global RNG - not thread safe. */
-            rng = wolfssl_get_global_rng();
-            rsa->ownRng = 0;
-        }
-        if (rng == NULL) {
-            /* Couldn't create global either. */
-            WOLFSSL_ERROR_MSG("wolfSSL_RSA_new no WC_RNG for blinding");
-            err = 1;
-        }
-        else {
-            /* Set the local or global RNG into the wolfCrypt RSA key. */
-            (void)wc_RsaSetRNG(key, rng);
-            /* Won't fail as key and rng are not NULL. */
-        }
-    }
-    #endif /* !HAVE_FIPS && WC_RSA_BLINDING */
-    if (!err) {
         /* Set wolfCrypt RSA key into RSA key. */
         rsa->internal = key;
         /* Data from external RSA key has not been set into internal one. */
@@ -2419,8 +2520,9 @@ WC_RNG* WOLFSSL_RSA_GetRNG(WOLFSSL_RSA* rsa, WC_RNG** tmpRng, int* initTmpRng)
         *initTmpRng = 0;
 
     #if !defined(HAVE_FIPS) && defined(WC_RSA_BLINDING)
-        /* Use wolfCrypt RSA key's RNG if available/set. */
-        rng = ((RsaKey*)rsa->internal)->rng;
+        /* Use wolfCrypt RSA key's RNG, setting it up if this is its first
+         * use. */
+        rng = wolfssl_rsa_ensure_rng(rsa);
     #endif
     }
     if ((!err) && (rng == NULL) && (tmpRng != NULL)) {
@@ -3638,6 +3740,15 @@ int wolfSSL_RSA_private_decrypt(int len, const unsigned char* from,
         }
     }
 
+    /* This takes the RNG off the wolfCrypt key rather than being passed one,
+     * so the key's blinding RNG has to be in place before the call. */
+#if !defined(HAVE_FIPS) && defined(WC_RSA_BLINDING)
+    if ((ret == 0) && (wolfssl_rsa_ensure_rng(rsa) == NULL)) {
+        WOLFSSL_ERROR_MSG("No RNG for RSA blinding");
+        ret = WOLFSSL_FATAL_ERROR;
+    }
+#endif
+
     if (ret == 0) {
         /* Use wolfCrypt to private-decrypt with RSA key.
          * Size of 'to' buffer must be size of RSA key */

src/ssl.c

@@ -347,6 +347,22 @@ WC_RNG* wolfssl_make_global_rng(void)
  * for RSA, DH, ECC and DSA for BN. */
 WC_RNG* wolfssl_make_rng(WC_RNG* rng, int* local);
 
+/* Whether the shared global RNG is handed to the operations that need one, in
+ * preference to giving each call an RNG of its own.
+ *
+ * Setting up an RNG seeds a DRBG, which costs more than many of the operations
+ * that ask for one: an ECDSA signature spends more time on it than on the
+ * signature. Sharing the one the library already has avoids that, and is only
+ * done where it is safe to share: either the instance takes its own lock
+ * whenever it is used, or there are no other threads to share it with.
+ *
+ * Define WOLFSSL_PK_LOCAL_RNG to go back to an RNG per call regardless.
+ */
+#if !defined(WOLFSSL_PK_LOCAL_RNG) && \
+    (defined(WC_RNG_HAVE_AUTO_LOCK) || defined(SINGLE_THREADED))
+    #define WOLFSSL_PK_GLOBAL_RNG
+#endif
+
 /* Make a random number generator or get global if possible.
  *
  * Global may not be available and NULL will be returned.
@@ -361,7 +377,20 @@ WC_RNG* wolfssl_make_rng(WC_RNG* rng, int* local)
     WC_RNG* ret = NULL;
 #ifdef WOLFSSL_SMALL_STACK
     int freeRng = 0;
+#endif
 
+#ifdef WOLFSSL_PK_GLOBAL_RNG
+    /* Hand back the shared one, which is already seeded. */
+    ret = wolfssl_make_global_rng();
+    if (ret != NULL) {
+        /* Not the caller's to free. */
+        *local = 0;
+        return ret;
+    }
+    /* No global to be had, so fall through and make one. */
+#endif
+
+#ifdef WOLFSSL_SMALL_STACK
     /* Allocate RNG object . */
     if (rng == NULL) {
         rng = (WC_RNG*)XMALLOC(sizeof(WC_RNG), NULL, DYNAMIC_TYPE_RNG);
@@ -2540,6 +2569,20 @@ int wolfSSL_Init(void)
         }
 #endif
 
+        /* The compatibility layer's caches each need a global mutex, made here
+         * because a mutex created on first use cannot be made safe. */
+#ifdef WOLFSSL_HAVE_X509_PUBKEY_MUTEX
+        if ((ret == WOLFSSL_SUCCESS) &&
+                (wolfssl_x509_pubkey_mutex_init() != 0)) {
+            ret = BAD_MUTEX_E;
+        }
+#endif
+#ifdef WOLFSSL_HAVE_RSA_RNG_MUTEX
+        if ((ret == WOLFSSL_SUCCESS) && (wolfssl_rsa_rng_mutex_init() != 0)) {
+            ret = BAD_MUTEX_E;
+        }
+#endif
+
     #ifdef WC_RNG_SEED_CB
         wc_SetSeed_Cb(WC_GENERATE_SEED_DEFAULT);
     #endif
@@ -3831,6 +3874,15 @@ int wolfSSL_Cleanup(void)
     #endif
 #endif
 
+/* Outside the global RNG block above: wolfSSL_Init() creates these two
+ * whatever that is configured to. */
+#ifdef WOLFSSL_HAVE_X509_PUBKEY_MUTEX
+    wolfssl_x509_pubkey_mutex_free();
+#endif
+#ifdef WOLFSSL_HAVE_RSA_RNG_MUTEX
+    wolfssl_rsa_rng_mutex_free();
+#endif
+
 #ifdef HAVE_EX_DATA_CRYPTO
     crypto_ex_cb_free(crypto_ex_cb_ctx_session);
     crypto_ex_cb_ctx_session = NULL;

src/ssl_crypto.c

@@ -801,6 +801,15 @@ int wolfSSL_SHA3_224_Init(WOLFSSL_SHA3_224_CTX* sha3_224)
 {
     /* Ensure WOLFSSL_SHA3_224_CTX is big enough for wolfCrypt wc_Sha3. */
     WOLFSSL_ASSERT_SIZEOF_GE(WOLFSSL_SHA3_224_CTX, wc_Sha3);
+    /* And no bigger than it needs to be. The holder is an array of pointers,
+     * so a size in bytes used as the element count inflates it eightfold -
+     * which nothing else notices, because it is still big enough. This
+     * context is the largest member of the union in WOLFSSL_EVP_MD_CTX, so
+     * the cost lands on every digest context in the compatibility layer.
+     * Measured on the holder rather than the context, because ALIGN16 rounds
+     * the context's size up and that tail padding is not waste to report. */
+    wc_static_assert(sizeof(((WOLFSSL_SHA3_224_CTX*)0)->holder) <
+        sizeof(wc_Sha3) + sizeof(void*));
 
     WOLFSSL_ENTER("SHA3_224_Init");