Merge pull request #11215 from SparkiDev/tls13_rfc9846

Conformance to RFC 9846: New version of TLS 1.3 specification

Author: philljj

ChangeLog.md

@@ -170,6 +170,23 @@
   configuration now sees neither the prototype nor the `client_cert_cb`
   typedef instead of failing to build; no other configuration changes.
 
+* **Behavioral change (`wolfSSL_write_early_data` and the AEAD key usage
+  limit)**: RFC 9846, Section 5.5 adds that "it is not possible to perform a
+  KeyUpdate for early data; therefore, implementations MUST NOT exceed the
+  limits when sending early data".  Reaching the limit mid-early-data
+  previously drove the ordinary rekey path, which emitted a KeyUpdate while
+  the client was still in `CLIENT_HELLO_COMPLETE` - before the handshake had
+  finished, where a conforming peer must reject it.  The write now fails
+  instead, returning `WOLFSSL_FATAL_ERROR` with `wolfSSL_get_error()`
+  reporting `TOO_MUCH_EARLY_DATA`.  The write stays all-or-nothing, matching
+  every other non-`partialWrite` return from `SendData()`.  The budget also
+  now stops one record short, because an accepted 0-RTT exchange still owes
+  the server an `EndOfEarlyData`, and RFC 9846 Section 2.3 sends that under the
+  same early traffic keys.  Callers that hit this should abandon early data on
+  the connection rather than resume the send from an offset.  Reaching the
+  limit needs roughly 23.7 million early data records on one connection, so no
+  practical caller is affected.
+
 ## Fixes
 
 * **Fix (sniffer could not decrypt Encrypt-Then-MAC or X25519 sessions)**: the
@@ -266,6 +283,43 @@
   than withdrawing them, matching what a failure to allocate the `ecc_key`
   already did.  Only affects builds with `WOLFSSL_BLIND_PRIVATE_KEY`.
 
+* **Fix (fatal-level `user_canceled` closed a TLS 1.3 connection)**: RFC 9846,
+  Section 6.1 states that this alert "generally has AlertLevel=warning" and
+  that "receiving implementations SHOULD continue to read data from the peer
+  until a 'close_notify' is received".  wolfSSL already exempted
+  `user_canceled` from the TLS 1.3 rule that all error alerts are fatal, but
+  both `DoAlert()` and `DoProcessAlertRecord()` acted on the AlertLevel byte
+  before reaching those exemptions, so a peer sending the alert at fatal level
+  tore the connection down and invalidated the session.  The level byte
+  carries no meaning in TLS 1.3, and the alert is now ignored whichever level
+  the peer used.  TLS 1.2 and earlier are unchanged: a fatal-level alert
+  remains fatal there.
+
+* **Fix (key update cap turned a peer's `update_requested` into a fatal
+  error)**: RFC 9846, Section 4.7.3 adds that a sender at the 2^48-1 key
+  update cap "MUST NOT send its own KeyUpdate ... and SHOULD instead ignore
+  the 'update_requested' flag".  Responding to a peer's request went through
+  the ordinary send path, which refuses at the cap with `BAD_STATE_E`, and
+  that error propagated out and killed the connection.  The request is now
+  dropped and the connection continues on its current keys until the Section
+  5.5 data limits force it closed.  An application-initiated
+  `wolfSSL_update_keys()` at the cap still reports `BAD_STATE_E`; the rule
+  applies only to responding to a peer.
+
+* **Fix (malformed extension aborted without sending `decode_error`)**: RFC
+  9846, Section 4.3 adds that trailing data in an extension is forbidden and
+  that "receivers MUST abort the handshake with a 'decode_error' alert if
+  there is data left over after parsing the structure".  The extension parsers
+  detect malformed structures, but around a third of them report it as the
+  wolfCrypt `BUFFER_E` rather than `BUFFER_ERROR`, and only `BUFFER_ERROR` was
+  mapped to an alert.  `TranslateErrorToAlert()` returned `invalid_alert` for
+  `BUFFER_E`, which every caller treats as "send nothing", so the handshake
+  aborted correctly but silently and the peer saw only a dropped connection.
+  Both codes now map to `decode_error`.  This affects `pre_shared_key`,
+  `psk_key_exchange_modes`, `early_data`, `cookie`, `post_handshake_auth` and
+  the certificate type extensions, and more generally any malformed handshake
+  message reported with `BUFFER_E`.
+
 # wolfSSL Release 5.9.2 (Jun 23, 2026)
 
 Release 5.9.2 has been developed according to wolfSSL's development and QA

configure.ac

@@ -6069,7 +6069,7 @@ fi
 
 # OLD TLS
 AC_ARG_ENABLE([oldtls],
-    [AS_HELP_STRING([--enable-oldtls],[Enable old TLS versions < 1.2 (default: disabled)])],
+    [AS_HELP_STRING([--enable-oldtls],[Enable old TLS versions < 1.2, which RFC 8996 and RFC 9846 forbid negotiating (default: disabled)])],
     [ ENABLED_OLD_TLS=$enableval ],
     [ ENABLED_OLD_TLS=no ]
     )

doc/dox_comments/header_files/ssl.h

@@ -2584,6 +2584,11 @@ int  wolfSSL_shutdown(WOLFSSL* ssl);
 
     \param ssl pointer to the SSL session, created with wolfSSL_new().
 
+    \note RFC 9846, Section 6.1 requires this alert to be followed by a
+    close notify, which is why the shutdown is part of this call. On the
+    receiving side the alert is not itself an error: a TLS 1.3 peer keeps
+    reading until the close notify arrives, whatever AlertLevel was used.
+
     _Example_
     \code
     int ret = 0;
@@ -14770,6 +14775,13 @@ int  wolfSSL_require_psk(WOLFSSL* ssl);
 
     \return BAD_FUNC_ARG if ssl is NULL or not using TLS v1.3.
     \return WANT_WRITE if the writing is not ready.
+    \return BAD_STATE_E if the connection has already performed the maximum
+    number of key updates. RFC 9846, Section 4.7.3 caps a TLS 1.3 sender at
+    2^48-1 key updates; beyond that the connection must be closed rather than
+    rekeyed. Note that a KeyUpdate arriving from the peer with
+    request_update set is ignored once this cap is reached, rather than
+    failing the connection, so only an application-initiated update reports
+    this error.
     \return WOLFSSL_SUCCESS if successful.
 
     _Example_
@@ -14800,7 +14812,8 @@ int  wolfSSL_update_keys(WOLFSSL* ssl);
     is received.
 
     \param [in] ssl a pointer to a WOLFSSL structure, created using wolfSSL_new().
-    \param [out] required   0 when no key update response required. 1 when no key update response required.
+    \param [out] required   0 when no key update response is required. 1 when
+    a key update response from the peer is still outstanding.
 
     \return 0 on successful.
     \return BAD_FUNC_ARG if ssl is NULL or not using TLS v1.3.
@@ -15622,7 +15635,15 @@ int  wolfSSL_set_max_early_data(WOLFSSL* ssl, unsigned int sz);
     \return SIDE_ERROR if called with a server.
     \return BAD_STATE_E if invoked without a valid session or without a valid
     PSK cb
-    \return WOLFSSL_FATAL_ERROR if the connection is not made.
+    \return WOLFSSL_FATAL_ERROR if the connection is not made, or if the
+    AEAD key usage limit would be exceeded by this write, in which case
+    wolfSSL_get_error() reports TOO_MUCH_EARLY_DATA. A KeyUpdate cannot be
+    performed while sending early data (RFC 9846, Section 5.5), so no further
+    early data can be sent on this connection. The write is all-or-nothing as
+    usual: a value less than sz is never returned. Note that a failed call may
+    still have put records on the wire before the limit was reached, so treat
+    the connection as unusable for early data rather than resuming the send
+    from an offset.
     \return the amount of early data written in bytes if successful.
 
     _Example_

src/dtls.c

@@ -88,6 +88,16 @@ void DtlsResetState(WOLFSSL* ssl)
     ssl->keys.dtls_sequence_number_hi = 0;
     ssl->keys.dtls_sequence_number_lo = 0;
 
+    /* Forget any alert this object sent for the ClientHello being abandoned.
+     * DoClientHello() can send a fatal alert on the stateless path and then
+     * swallow the error (DtlsIgnoreError) so the object stays up waiting for
+     * the next ClientHello. A leftover alert_fatal in the history makes the
+     * "already sent a more specific fatal alert" guards suppress every later
+     * alert on this object, so a single malformed ClientHello from a spoofed
+     * address would mute alerts for every peer that follows. */
+    ssl->alert_history.last_tx.code  = -1;
+    ssl->alert_history.last_tx.level = -1;
+
     /* Reset states */
     ssl->options.serverState = NULL_STATE;
     ssl->options.clientState = NULL_STATE;

src/internal.c

@@ -104,7 +104,8 @@
  * WOLFSSL_TLS13_NO_PEEK_HANDSHAKE_DONE:
  *                  Disable peek returning WANT_READ for tickets       default: off
  * WOLFSSL_TLS13_IGNORE_AEAD_LIMITS:
- *                  Ignore AEAD message limits from RFC 8446           default: off
+ *                  Ignore AEAD message limits from RFC 9846 5.5, which
+ *                  makes observing them a MUST                        default: off
  * WOLFSSL_DTLS13_SEND_MOREACK_DEFAULT:
  *                  Send more ACKs by default in DTLS 1.3              default: off
  *
@@ -23994,10 +23995,41 @@ static void LogAlert(int type)
 #endif /* DEBUG_WOLFSSL */
 }
 
+/* RFC 9846 Section 6.1 exempts "user_canceled" from tearing the connection
+ * down whatever AlertLevel the peer used, because the level byte carries no
+ * meaning in TLS 1.3.
+ *
+ * The exemption needs a connection that is demonstrably running TLS 1.3, and
+ * neither obvious flag is enough on its own. ssl->version holds the highest
+ * version this side offered until the peer's choice is known, and
+ * options.tls1_3 is set from the session by wolfSSL_set_session() before any
+ * ServerHello, so a downgrade-capable client resuming a TLS 1.3 session has
+ * both set while the peer may still pick TLS 1.2. Exempting on either would
+ * swallow a pre-1.3 peer's fatal alert and leave the caller waiting for a
+ * handshake that is never coming.
+ *
+ * Requiring the record to have been decrypted settles it: in TLS 1.3 every
+ * record after the ServerHello is encrypted, so a decrypted alert can only
+ * have arrived on a connection whose version is already agreed - which is the
+ * only situation RFC 9846 Section 6.1 is describing. keys.decryptedCur is a
+ * property of the record in hand, not a latch: GetRecordHeader() clears it for
+ * each new record and only a successful decrypt sets it, so an earlier
+ * encrypted record cannot lend its status to a later plaintext alert.
+ *
+ * ssl   The SSL/TLS object.
+ * code  Alert description received.
+ * returns 1 when the alert must be ignored rather than acted on.
+ */
+static int AlertIsExemptUserCanceled(const WOLFSSL* ssl, int code)
+{
+    return ssl->options.tls1_3 && ssl->keys.decryptedCur &&
+           (code == user_canceled);
+}
+
 /* process alert, return level */
 #ifndef NO_SESSION_CACHE
 /* RFC 5246 Section 7.2.2: a TLS 1.2 session whose connection is terminated by a
- * fatal alert MUST be invalidated so it cannot be resumed. (TLS 1.3 RFC 8446
+ * fatal alert MUST be invalidated so it cannot be resumed. (TLS 1.3 RFC 9846
  * Section 6.2 only requires closing the connection, but evicting here too is
  * sound defense-in-depth.) Evict the cached session (which also drops any
  * associated ticket). Acts on an established connection or an in-progress
@@ -24011,7 +24043,7 @@ static void InvalidateSessionOnFatalAlert(WOLFSSL* ssl)
         return;
     /* Don't evict on an unauthenticated record: a TLS 1.3 plaintext alert
      * received under encryption (current record not decrypted) is rejected (or
-     * ignored) by DoAlert, and the teardown alert routes back here. RFC 8446
+     * ignored) by DoAlert, and the teardown alert routes back here. RFC 9846
      * 6.2 doesn't require TLS 1.3 eviction; TLS 1.2 alerts are plaintext so are
      * unaffected. */
     if (IsAtLeastTLSv1_3(ssl->version) && IsEncryptionOn(ssl, 0) &&
@@ -24075,10 +24107,16 @@ static int DoAlert(WOLFSSL* ssl, byte* input, word32* inOutIdx, int* type)
     {
         ssl->alert_history.last_rx.code = code;
         ssl->alert_history.last_rx.level = level;
-        if (level == alert_fatal) {
+        /* RFC 9846 Section 6.1: "user_canceled" only "generally" has
+         * AlertLevel=warning, and a receiver SHOULD keep reading until
+         * "close_notify" arrives. The level byte is meaningless in TLS 1.3,
+         * so do not let a peer that sends the alert at fatal level tear the
+         * connection down. */
+        if (level == alert_fatal &&
+                !AlertIsExemptUserCanceled(ssl, code)) {
             ssl->options.isClosed = 1;  /* Don't send close_notify */
         }
-        /* RFC 8446 Section 6.2: In TLS 1.3, all error alerts are implicitly
+        /* RFC 9846 Section 6.2: In TLS 1.3, all error alerts are implicitly
          * fatal regardless of the AlertLevel byte. */
         if (IsAtLeastTLSv1_3(ssl->version) &&
                 code != close_notify && code != user_canceled) {
@@ -24130,12 +24168,17 @@ static int DoAlert(WOLFSSL* ssl, byte* input, word32* inOutIdx, int* type)
         }
 #ifndef NO_SESSION_CACHE
         /* Validated fatal alert: invalidate the session so it can't be resumed
-         * (RFC 5246 7.2.2; in TLS 1.3 all error alerts are fatal, RFC 8446
-         * 6.2). */
-        if (*type != close_notify &&
-                (level == alert_fatal ||
-                 (IsAtLeastTLSv1_3(ssl->version) && *type != user_canceled)))
+         * (RFC 5246 7.2.2; in TLS 1.3 all error alerts are fatal, RFC 9846
+         * 6.2). "close_notify" is not an error, and "user_canceled" is exempt
+         * in TLS 1.3 at any AlertLevel (RFC 9846 6.1). */
+        if (IsAtLeastTLSv1_3(ssl->version)) {
+            if (*type != close_notify &&
+                    !AlertIsExemptUserCanceled(ssl, *type))
+                InvalidateSessionOnFatalAlert(ssl);
+        }
+        else if (level == alert_fatal && *type != close_notify) {
             InvalidateSessionOnFatalAlert(ssl);
+        }
 #endif
     }
     return level;
@@ -24985,7 +25028,9 @@ static int DoProcessAlertRecord(WOLFSSL* ssl)
     WOLFSSL_MSG("got ALERT!");
     ret = DoAlert(ssl, ssl->buffers.inputBuffer.buffer,
                   &ssl->buffers.inputBuffer.idx, &type);
-    if (ret == alert_fatal)
+    /* RFC 9846 Section 6.1: keep reading past a TLS 1.3 "user_canceled" until
+     * "close_notify" arrives, whatever AlertLevel the peer used. */
+    if (ret == alert_fatal && !AlertIsExemptUserCanceled(ssl, type))
         return FATAL_ERROR;
     else if (ret < 0)
         return ret;
@@ -25001,7 +25046,7 @@ static int DoProcessAlertRecord(WOLFSSL* ssl)
     if (type == decrypt_error)
         return FATAL_ERROR;
 
-    /* RFC 8446 Section 6.2: In TLS 1.3, all error alerts MUST
+    /* RFC 9846 Section 6.2: In TLS 1.3, all error alerts MUST
      * be treated as fatal regardless of the AlertLevel byte.
      * Only close_notify (handled above) and user_canceled
      * are exempt. */
@@ -28634,7 +28679,7 @@ int IsSCR(WOLFSSL* ssl)
     !defined(WOLFSSL_TLS13_IGNORE_AEAD_LIMITS)
 /*
  * Enforce limits specified in
- * https://www.rfc-editor.org/rfc/rfc8446#section-5.5
+ * https://www.rfc-editor.org/rfc/rfc9846#section-5.5
  */
 static int CheckTLS13AEADSendLimit(WOLFSSL* ssl)
 {
@@ -28701,6 +28746,44 @@ static int CheckTLS13AEADSendLimit(WOLFSSL* ssl)
                 ssl->keys.sequence_number_lo);
     }
 
+#ifdef WOLFSSL_EARLY_DATA
+    /* RFC 9846 Section 5.5: a KeyUpdate cannot be performed for early data, so
+     * a sender MUST NOT exceed the limits while sending it. There is no way to
+     * rekey here - the handshake has not finished, so a KeyUpdate would be out
+     * of order - and the write has to fail instead.
+     *
+     * Stop one record early. If the server accepts the early data the client
+     * still owes it an EndOfEarlyData, and Section 2.3 has that message go out
+     * under the same 0-RTT traffic keys. Spending the last record on
+     * application data would leave that message to overrun the limit.
+     *
+     * seq is the number the next record will use, so it also counts the
+     * records already sent under this key. Refusing once seq + 1 reaches the
+     * limit stops application data at seq == limit - 2 and keeps the final
+     * slot, seq == limit - 1, free for the EndOfEarlyData. */
+    if (ssl->options.side == WOLFSSL_CLIENT_END &&
+            ssl->earlyData != no_early_data &&
+            ssl->earlyData != done_early_data) {
+        w64wrapper afterThis = seq;
+
+        w64Increment(&afterThis);
+        /* cppcheck-suppress uninitvar
+         * (false positive from cppcheck-2.13.0: every switch arm above either
+         * sets limit or returns) */
+        if (w64GTE(afterThis, limit)) {
+            WOLFSSL_MSG("AEAD limit reached while sending early data");
+            /* Deliberately not recorded here. The caller decides whether this
+             * is an error: a write whose last record took the final slot has
+             * completed and returns success, and recording it now would leave
+             * TOO_MUCH_EARLY_DATA in the OpenSSL-compatibility error queue for
+             * a call that reported no failure. SendData() records it on the
+             * path that does fail. */
+            return TOO_MUCH_EARLY_DATA;
+        }
+        return 0;
+    }
+#endif
+
     if (w64GTE(seq, limit)) { /* cppcheck-suppress uninitvar
                                * (false positive from cppcheck-2.13.0)
                                */
@@ -28966,6 +29049,27 @@ int SendData(WOLFSSL* ssl, const void* data, size_t sz)
 #if defined(WOLFSSL_TLS13) && !defined(WOLFSSL_TLS13_IGNORE_AEAD_LIMITS)
         if (IsAtLeastTLSv1_3(ssl->version)) {
             ret = CheckTLS13AEADSendLimit(ssl);
+        #ifdef WOLFSSL_EARLY_DATA
+            /* This check runs at the top of every iteration, including the
+             * one after the last record, so it also fires when the write is
+             * already complete and its final record happened to reach the
+             * limit. Nothing is left to send, so report the success: without
+             * this the caller would see WOLFSSL_FATAL_ERROR for a write that
+             * fully landed.
+             *
+             * A write stopped part way through is a failure, not a short
+             * return. Short returns here are gated on partialWrite, and
+             * wolfSSL_get_error(ssl, ret) reports nothing for a positive ret,
+             * so a short count would be indistinguishable from a complete one
+             * to a caller following the documented contract. */
+            if (ret == WC_NO_ERR_TRACE(TOO_MUCH_EARLY_DATA)) {
+                if (sent == (word32)sz) {
+                    break;
+                }
+                /* Failing for real, so now it is worth recording. */
+                WOLFSSL_ERROR_VERBOSE(ret);
+            }
+        #endif
             if (ret != 0) {
                 ssl->error = ret;
                 return WOLFSSL_FATAL_ERROR;
@@ -38414,7 +38518,14 @@ const byte* MaskCurve25519PeerKey(const byte* pub, word32 pubSz,
     int TranslateErrorToAlert(int err)
     {
         switch (err) {
+            /* RFC 9846 Section 4.3 requires a "decode_error" alert when an
+             * extension has data left over after its structure is parsed, and
+             * Section 6.2 defines the alert for any field out of range or
+             * message of incorrect length. The extension parsers report those
+             * as either BUFFER_ERROR or the wolfCrypt BUFFER_E; both must map
+             * here, or the handshake aborts silently with no alert sent. */
             case WC_NO_ERR_TRACE(BUFFER_ERROR):
+            case WC_NO_ERR_TRACE(BUFFER_E):
                 return decode_error;
             case WC_NO_ERR_TRACE(EXT_NOT_ALLOWED):
             case WC_NO_ERR_TRACE(PEER_KEY_ERROR):