🔖 Release 3.1.3

Ahmed TAHRI · Ahmed TAHRI · commit b9bd9b314b89 · 2023-10-19T09:10:29.000+02:00
**Fixed** - **oheaders** from a Response contains `Set-Cookie` entries when it should not. - Static type checker not accepting **list\[str\]** in values for argument **param**. - Static type checker not accepting **Iterable\[bytes\]** for **data**. - Function proxy_bypass_registry for Windows may be fooled by insufficient control on our end. Patch taken from idle upstream PR psf#6302 - SSLError message related to the certificate revocation could print `None` instead of `unspecified` for the reason. **Changed** - Allow setting `None` in max_size for **SharableLimitedDict** to remove limits. - Using `RLock` instead of `Lock` in **SharableLimitedDict**, and **InMemoryRevocationStatus** classes. **Misc** - Missing assert statements for test test_header_validation. - Unrelated warnings are now silent in our test suite. - Unexpected warning now trigger an error in our test suite. - Removed `tests.compat`. - Removed `test-readme`, `flake8`, and `publish` from Makefile. **Added** - Extra-dist install `http3` to force install HTTP/3 support in your environment if not present. - Extra-dist install `ocsp` to force install certificate revocation support in your environment if not present.
diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml
@@ -16,8 +16,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12-dev", "pypy-3.8", "pypy-3.9"]
-        os: [ubuntu-22.04, macOS-latest, windows-latest]
+        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "pypy-3.8", "pypy-3.9", "pypy-3.10"]
+        os: [ubuntu-latest, macOS-latest, windows-latest]
         include:
           # pypy-3.7 on Windows and Mac OS currently fails trying to compile
           # cryptography. Moving pypy-3.7 to only test linux.
diff --git a/HISTORY.md b/HISTORY.md
@@ -1,6 +1,32 @@
 Release History
 ===============
 
+3.1.3 (2023-10-19)
+------------------
+
+**Fixed**
+- **oheaders** from a Response contains `Set-Cookie` entries when it should not.
+- Static type checker not accepting **list\[str\]** in values for argument **param**.
+- Static type checker not accepting **Iterable\[bytes\]** for **data**.
+- Function proxy_bypass_registry for Windows may be fooled by insufficient control on our end.
+  Patch taken from idle upstream PR https://github.com/psf/requests/pull/6302
+- SSLError message related to the certificate revocation could print `None` instead of `unspecified` for the reason.
+
+**Changed**
+- Allow setting `None` in max_size for **SharableLimitedDict** to remove limits.
+- Using `RLock` instead of `Lock` in **SharableLimitedDict**, and **InMemoryRevocationStatus** classes.
+
+**Misc**
+- Missing assert statements for test test_header_validation.
+- Unrelated warnings are now silent in our test suite.
+- Unexpected warning now trigger an error in our test suite.
+- Removed `tests.compat`.
+- Removed `test-readme`, `flake8`, and `publish` from Makefile.
+
+**Added**
+- Extra-dist install `http3` to force install HTTP/3 support in your environment if not present.
+- Extra-dist install `ocsp` to force install certificate revocation support in your environment if not present.
+
 3.1.2 (2023-10-16)
 ------------------
 
diff --git a/Makefile b/Makefile
@@ -5,22 +5,10 @@ test:
 	# This runs all of the tests on all supported Python versions.
 	tox -p
 ci:
-	python -m pytest tests --junitxml=report.xml
-
-test-readme:
-	python setup.py check --restructuredtext --strict && ([ $$? -eq 0 ] && echo "README.rst and HISTORY.rst ok") || echo "Invalid markup in README.rst or HISTORY.rst!"
-
-flake8:
-	python -m flake8 src/requests
+	python -m pytest tests --verbose --junitxml=report.xml
 
 coverage:
-	python -m pytest --cov-config .coveragerc --verbose --cov-report term --cov-report xml --cov=src/requests tests
-
-publish:
-	python -m pip install 'twine>=1.5.0'
-	python setup.py sdist bdist_wheel
-	twine upload dist/*
-	rm -fr build dist .egg requests.egg-info
+	python -m pytest --cov-config .coveragerc --verbose --cov-report term --cov-report xml --cov=niquests tests
 
 docs:
 	cd docs && make html
diff --git a/docs/user/advanced.rst b/docs/user/advanced.rst
@@ -1209,3 +1209,22 @@ This would mean that attempting to request ``https://cloudflare.com/a/b`` will b
 over QUIC.
 
 .. warning:: You cannot specify another hostname for security reasons.
+
+Increase the default Alt-Svc cache size
+---------------------------------------
+
+When a server yield its support for HTTP/3 over QUIC, the information
+is stored within a local thread safe in-memory storage.
+
+That storage is limited to 12,288 entries by default, and you can override this
+by passing a custom ``QuicSharedCache`` instance like so::
+
+    import niquests
+
+    cache = niquests.structures.QuicSharedCache(max_size=128_000)
+    session = niquests.Session(quic_cache_layer=cache)
+
+
+.. note:: Passing ``None`` to max size actually permit the cache to grow indefinitely. This is unwise and can lead to significant RAM usage.
+
+When the cache is full, the oldest entry is removed.
diff --git a/pyproject.toml b/pyproject.toml
@@ -48,7 +48,13 @@ dependencies = [
 
 [project.optional-dependencies]
 socks = [
-  "PySocks>=1.5.6, !=1.5.7",
+    "PySocks>=1.5.6, !=1.5.7",
+]
+http3 = [
+    "qh3<1.0.0,>=0.11.3"
+]
+ocsp = [
+    "cryptography<42.0.0,>=41.0.0"
 ]
 
 [project.urls]
@@ -91,3 +97,11 @@ addopts = "--doctest-modules"
 doctest_optionflags = "NORMALIZE_WHITESPACE ELLIPSIS"
 minversion = "6.2"
 testpaths = ["tests"]
+filterwarnings = [
+    "error",
+    '''ignore:'parse_authorization_header' is deprecated and will be removed:DeprecationWarning''',
+    '''ignore:The 'set_digest' method is deprecated and will be removed:UserWarning''',
+    '''ignore:Passing bytes as a header value is deprecated and will:DeprecationWarning''',
+    '''ignore:'JSONIFY_PRETTYPRINT_REGULAR' config key is deprecated and will be removed:DeprecationWarning''',
+    '''ignore:unclosed .*:ResourceWarning''',
+]
diff --git a/requirements-dev.txt b/requirements-dev.txt
@@ -1,5 +1,5 @@
 -e .[socks]
-pytest>=2.8.0,<=6.2.5
+pytest>=2.8.0,<=7.4.2
 pytest-cov
 pytest-httpbin==2.0.0
 httpbin==0.10.1
diff --git a/setup.cfg b/setup.cfg
@@ -2,4 +2,3 @@
 ignore = E203, E501, W503
 per-file-ignores =
     src/niquests/__init__.py:E402, F401
-    tests/compat.py:F401
diff --git a/src/niquests/__version__.py b/src/niquests/__version__.py
@@ -9,9 +9,9 @@
 __url__: str = "https://niquests.readthedocs.io"
 
 __version__: str
-__version__ = "3.1.2"
+__version__ = "3.1.3"
 
-__build__: int = 0x030102
+__build__: int = 0x030103
 __author__: str = "Kenneth Reitz"
 __author_email__: str = "me@kennethreitz.org"
 __license__: str = "Apache-2.0"
diff --git a/src/niquests/_typing.py b/src/niquests/_typing.py
@@ -16,8 +16,8 @@
 )
 #: List of formats accepted for URL queries parameters. (e.g. /?param1=a&param2=b)
 QueryParameterType: typing.TypeAlias = typing.Union[
-    typing.List[typing.Tuple[str, str]],
-    typing.Mapping[str, str],
+    typing.List[typing.Tuple[str, typing.Union[str, typing.List[str]]]],
+    typing.Mapping[str, typing.Union[str, typing.List[str]]],
     bytes,
     str,
 ]
@@ -32,6 +32,7 @@
     bytearray,
     typing.IO,
     BodyFormType,
+    typing.Iterable[bytes],
 ]
 #: HTTP Headers can be represented through three ways. 1) typical dict, 2) internal insensitive dict, and 3) list of tuple.
 HeadersType: typing.TypeAlias = typing.Union[
@@ -123,9 +124,7 @@
 _HV = typing.TypeVar("_HV")
 
 HookCallableType: typing.TypeAlias = typing.Callable[
-    [
-        _HV,
-    ],
+    [_HV],
     typing.Optional[_HV],
 ]
 
diff --git a/src/niquests/extensions/_ocsp.py b/src/niquests/extensions/_ocsp.py
@@ -53,7 +53,9 @@ def _str_fingerprint_of(certificate: Certificate) -> str:
 def _infer_issuer_from(certificate: Certificate) -> Certificate | None:
     issuer: Certificate | None = None
 
-    for der_cert in wassima.root_der_certificates() + _SharedStaplingCache.issuers:
+    for der_cert in (
+        wassima.root_der_certificates() + _SharedRevocationStatusCache.issuers
+    ):
         if isinstance(der_cert, Certificate):
             possible_issuer = der_cert
         else:
@@ -197,7 +199,7 @@ def __init__(self, max_size: int = 2048):
         self._store: dict[str, ocsp.OCSPResponse] = {}
         self._issuers: list[Certificate] = []
         self._timings: list[datetime.datetime] = []
-        self._access_lock: threading.Lock = threading.Lock()
+        self._access_lock = threading.RLock()
         self.hold: bool = False
 
     @property
@@ -295,7 +297,7 @@ def save(
                 self._timings.pop(0)
 
 
-_SharedStaplingCache = InMemoryRevocationStatus()
+_SharedRevocationStatusCache = InMemoryRevocationStatus()
 
 
 def verify(
@@ -327,17 +329,17 @@ def verify(
 
     # this feature, by default, is reserved for a reasonable usage.
     if not strict:
-        mean_rate_sec = _SharedStaplingCache.rate()
-        cache_count = len(_SharedStaplingCache)
+        mean_rate_sec = _SharedRevocationStatusCache.rate()
+        cache_count = len(_SharedRevocationStatusCache)
 
         if cache_count >= 10 and mean_rate_sec <= 1.0:
-            _SharedStaplingCache.hold = True
+            _SharedRevocationStatusCache.hold = True
 
-        if _SharedStaplingCache.hold:
+        if _SharedRevocationStatusCache.hold:
             return
 
     peer_certificate = load_der_x509_certificate(conn_info.certificate_der)
-    cached_response = _SharedStaplingCache.check(peer_certificate)
+    cached_response = _SharedRevocationStatusCache.check(peer_certificate)
 
     if cached_response is not None:
         issuer_certificate = _infer_issuer_from(peer_certificate)
@@ -352,7 +354,7 @@ def verify(
                 r.ocsp_verified = False
                 raise SSLError(
                     f"""Unable to establish a secure connection to {r.url} because the certificate has been revoked
-                    by issuer ({cached_response.revocation_reason}).
+                    by issuer ({cached_response.revocation_reason or "unspecified"}).
                     You should avoid trying to request anything from it as the remote has been compromised.
                     See https://en.wikipedia.org/wiki/OCSP_stapling for more information."""
                 )
@@ -479,14 +481,16 @@ def verify(
 
         ocsp_resp = ocsp.load_der_ocsp_response(ocsp_http_response.content)
 
-        _SharedStaplingCache.save(peer_certificate, issuer_certificate, ocsp_resp)
+        _SharedRevocationStatusCache.save(
+            peer_certificate, issuer_certificate, ocsp_resp
+        )
 
         if ocsp_resp.response_status == ocsp.OCSPResponseStatus.SUCCESSFUL:
             if ocsp_resp.certificate_status == ocsp.OCSPCertStatus.REVOKED:
                 r.ocsp_verified = False
                 raise SSLError(
                     f"""Unable to establish a secure connection to {r.url} because the certificate has been revoked
-                    by issuer ({ocsp_resp.revocation_reason}).
+                    by issuer ({ocsp_resp.revocation_reason or "unspecified"}).
                     You should avoid trying to request anything from it as the remote has been compromised.
                     See https://en.wikipedia.org/wiki/OCSP_stapling for more information."""
                 )
diff --git a/src/niquests/models.py b/src/niquests/models.py
@@ -514,7 +514,7 @@ def prepare_body(
                         form_data_boundary = None
 
                     body = self._encode_params(
-                        data,
+                        data,  # type: ignore[arg-type]
                         boundary_for_multipart=form_data_boundary,
                     )
                     if isinstance(data, str) or hasattr(data, "read"):
@@ -1108,7 +1108,9 @@ def oheaders(self) -> Headers:
         'text/html'
         """
         if self.raw:
-            return parse_it(self.raw)
+            headers = parse_it(self.raw)
+            headers -= "Set-Cookie"
+            return headers
         return parse_it(self.headers)
 
     @property
diff --git a/src/niquests/structures.py b/src/niquests/structures.py
@@ -105,10 +105,10 @@ def get(self, key, default=None):
 
 
 class SharableLimitedDict(typing.MutableMapping):
-    def __init__(self, max_size: int) -> None:
+    def __init__(self, max_size: int | None) -> None:
         self._store: typing.MutableMapping[typing.Any, typing.Any] = {}
         self._max_size = max_size
-        self._lock = threading.Lock()
+        self._lock = threading.RLock()
 
     def __delitem__(self, __key) -> None:
         with self._lock:
@@ -124,7 +124,7 @@ def __iter__(self) -> typing.Iterator:
 
     def __setitem__(self, key, value):
         with self._lock:
-            if len(self._store) >= self._max_size:
+            if self._max_size and len(self._store) >= self._max_size:
                 self._store.popitem()
 
             self._store[key] = value
diff --git a/src/niquests/utils.py b/src/niquests/utils.py
@@ -86,6 +86,9 @@ def proxy_bypass_registry(host) -> bool:
         # '<local>' string by the localhost entry and the corresponding
         # canonical entry.
         proxyOverride = proxyOverride.split(";")
+        # Sometime empty value can land in evaluated proxyOverride.split(";").
+        # We clear them out before doing any regexp match.
+        proxyOverride = filter(None, proxyOverride)
         # now check if we match one of the registry values.
         for test in proxyOverride:
             if test == "<local>":
diff --git a/tests/__init__.py b/tests/__init__.py
@@ -1,16 +1,3 @@
 """Requests test package initialisation."""
 
 from __future__ import annotations
-
-import warnings
-
-try:
-    from urllib3.exceptions import SNIMissingWarning
-
-    # urllib3 1.x sets SNIMissingWarning to only go off once,
-    # while this test suite requires it to always fire
-    # so that it occurs during test_niquests.test_https_warnings
-    warnings.simplefilter("always", SNIMissingWarning)
-except ImportError:
-    # urllib3 2.0 removed that warning and errors out instead
-    SNIMissingWarning = None
diff --git a/tests/compat.py b/tests/compat.py
diff --git a/tests/test_requests.py b/tests/test_requests.py
diff --git a/tests/test_utils.py b/tests/test_utils.py
