Speed up parsing of contiguous input: numbers, strings, and UTF-8 (with optional simdutf) by nlohmann · Pull Request #5283 · nlohmann/json

nlohmann · 2026-07-20T09:27:47Z

What & why

The parser reads input one character at a time through the input adapter and converts numbers with strtod/strtoull. For the common case of contiguous byte input (std::string, std::vector<char>, string literals, const char* ranges), the per-character get()/add() overhead and the locale/errno-heavy number conversion dominate. This PR adds fast paths for that case, borrowing ideas from the simdjson / fast_float ecosystem, while keeping the library header-only with a C++11 fallback on every path. Streaming inputs (files, std::istream, wide strings, user adapters) are untouched and keep the byte-at-a-time scanner. Every fast path defers to the byte-path scanner on anything malformed, so parse errors keep the same message and the same byte offset; the one deliberate exception is the reported column after a newline, described under Public API impact below.

The changes (each independent and reversible)

Number conversion — replace strtoull/strtoll/strtod over the validated digit buffer with dedicated parsers: hand-rolled integer parsing with overflow→float fallback (matching the old semantics exactly), and Clinger's exact fast path for double, bit-identical to strtod on its subset and declining otherwise. float/long double keep strtof/strtold.
SWAR string scanning — scan 8 bytes at a time with a uint64_t word test to bulk-append ordinary characters. Pure C++11, no intrinsics.
UTF-8 bulk validation — fold well-formed multibyte sequences into the bulk run (validated with the same ranges as the byte-path switch), so non-ASCII text no longer drops to the slow per-byte path.
Contiguous-container pointer adapter — route std::string & friends through a pointer-based adapter so 1–3 apply to json::parse(std::string) in C++11–17 (previously only const char*/literals, and C++20 contiguous iterators, qualified). The pointer keeps the container's element type, so char_type — and all behavior, including binary formats from std::vector — is unchanged. Detection requires the container's own value_type to be the single-byte element data() points at, so a type whose size() counts something else keeps the iterator-based adapter instead of being read as [data(), data() + size()) bytes.
Contiguous number fast path — parse the whole number token straight from the buffer (no per-byte get()/add()), materializing token_buffer in one copy with the locale decimal point substituted, then reusing the shared convert_number() tail.
Optional simdutf backend (JSON_USE_SIMDUTF, off by default) — since simdutf ships a .cpp and uses runtime dispatch, it is not vendored; when defined, bulk UTF-8 validation routes through simdutf::validate_utf8 (the user supplies/links simdutf), otherwise the C++11 scalar validator is used. The library stays header-only either way. simdutf requires C++17 — its header rejects older standards with an #error — so the backend is gated on JSON_HAS_CPP_17 and the macro is inert below that, which keeps it safe to set project-wide in a mixed-standard build.
New byte-level headers — the value conversion and string scanning helpers live in two new focused headers, detail/input/number_parse.hpp (302 lines) and detail/input/string_scan.hpp (241 lines), as free functions operating on raw bytes, so lexer.hpp keeps only the state machine and the adapter-facing glue. lexer.hpp itself grows (1760 → 2025 lines) with the bulk scanners and the number fast path; the only code moved out of it is the ~30-line strtoull/strtoll/strtod block replaced in item 1.
Sized sentinels — the bulk paths apply to any iterator/sentinel pair whose remaining length is computable in O(1), not just same-type pairs, so std::counted_iterator + std::default_sentinel_t reaches them too (per @gregmarr's review, matching Extend memcpy fast path to sized sentinels (e.g. std::counted_iterator) #5268).

Measured gains

Cumulative, json::parse / accept(std::string), C++11, g++ 13 -O3, vs the v3.12.0 baseline (representative synthetic data):

dataset	`parse`	`accept`
floats	1.6×	1.8×
ints	1.2×	1.3×
strings (twitter-like)	1.3×	1.7×
long ASCII strings	8.5×	18×
dense CJK	2.4×	2.3×
mixed international	2.1×	2.2×

accept()/SAX/validation see the largest gains (not allocation-bound); full-DOM parse() gains on object-heavy input are capped by DOM allocation, which is a separate concern. With JSON_USE_SIMDUTF, dense-UTF-8 validation has further headroom (~4.7 GB/s ceiling measured with an SSE validator; ~7× the scalar path).

Public API impact

No breaking changes to the documented public API. parse(), accept(), sax_parse(), and the from_*() signatures are unchanged.

Two internal nlohmann::detail changes are worth recording:

detail::input_adapter(container) now returns a pointer-based adapter for contiguous byte containers, so detail::string_input_adapter_type changes from iterator_input_adapter<std::string::iterator> to iterator_input_adapter<const char*>. Source-compatible (these types are normally deduced), but the mangled names change, which matters to anyone shipping a precompiled wrapper that names them.
iterator_input_adapter gains supports_bulk_scan / bulk_data() / bulk_remaining() / bulk_skip(), and supports_seek is relaxed from "same-type iterator/sentinel" to "distance computable in O(1)".

One deliberate behavior change: error column after a newline

scan_number() reads the character that terminates a number and then ungets it, so the reported column is the one reached after the number's last character. When that terminator is a newline, get() has already cleared chars_read_current_line and unget() could only restore lines_read, leaving the column at 0 — a number followed by a newline reported a different position than the same number followed by a space.

unget() now remembers the column the newline was read at, so both terminators report the same position:

json::parse("[01\n]");             // was: line 1, column 0    now: line 1, column 3
json::parse("[01 ]");              //      line 1, column 3         line 1, column 3  (unchanged)

json::parse("null -2.5e3\nfalse"); // was: line 1, column 0    now: line 1, column 11
json::parse("null -2.5e3 false");  //      line 1, column 11        line 1, column 11 (unchanged)

Only the column changes — message text, byte offset, and accept/reject are identical, and the contiguous and streaming routes agree. 22 of 481k differential documents are affected, all of this shape.

Verification

Differential testing against the merge base across all routes (std::string / std::istream / std::vector / accept), C++11 and C++17, clang/libc++ and g++/libstdc++, in the C and de_DE/fr_FR locales:

481k mixed documents: identical values, accept/reject, byte offsets, and message text; zero disagreement between the contiguous and streaming routes across 298,333 error documents.
900k hard floating-point cases (subnormals, 15–25 significant digits, the Clinger boundary at |scale| = 22/23, significands near 2^53, over/underflow edges): bit-identical to strtod, both with std::from_chars active under libstdc++ and under -ffast-math.
JSON_DIAGNOSTIC_POSITIONS start/end offsets: identical over 60k documents.
ASan + UBSan clean over the full corpus and over counted-iterator inputs whose count ends before the underlying buffer does.
Millions of differential-fuzz documents (arbitrary bytes; overlong/surrogate/truncated UTF-8; overflow-range integers, long digit strings, %.17g doubles; escapes) parse and accept/reject identically; the JSON_USE_SIMDUTF build matches too.
Full test suite: 109/109, warning-clean on g++ and clang, C++11/17/20, in both build configurations; clang-tidy clean on the new headers; make check-amalgamation clean.
With JSON_TestSimdutf=ON, the whole suite (162 tests, built at C++11 and C++17) passes against the simdutf backend, and its results are byte-identical to the scalar validator over the differential corpus.

Notes for reviewers / remaining work

Dedicated tests/ cases cover the number fast path (including an exhaustive grammar-parity sweep over the number alphabet), the string and UTF-8 bulk scanners (exhaustive contiguous-vs-streaming parity, special bytes at every offset of the SWAR stride, and the boundaries of every accepted UTF-8 range), contiguous-container detection, the counted-iterator bulk paths (including counts that end before the underlying buffer), and contiguous-vs-streaming error-position parity.
JSON_TestSimdutf (off by default) builds the unit tests against simdutf, fetched with FetchContent; ci_test_simdutf runs the suite in that configuration and is part of the ci_cmake_options matrix. It warns and falls back to the scalar validator when no tested standard can reach simdutf. The library target and the installed CMake package are deliberately untouched: making nlohmann_json link simdutf::simdutf would add a find_dependency(simdutf) to the exported package, which is a separate decision.
The JSON_USE_SIMDUTF doc page says "Added in version 3.13.0", matching the other pages documenting unreleased features — adjust if this lands in a different release.

The changes are described in detail, both the what and why.
If applicable, an existing issue is referenced. (none)
The code coverage remained at 100%. A test case for every new line of code. (targeted tests now cover the number fast path, the string/UTF-8 bulk scanners, contiguous-container detection, the counted-iterator paths, and error-position parity; a full coverage run has not been done)
If applicable, the documentation is updated. (JSON_USE_SIMDUTF macro page, nav entry, and supported-macros overview)
The source code is amalgamated by running make amalgamate.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA

wilkolbrzym-coder · 2026-08-18T20:14:54Z

Quick review notes – PR #5283

No critical bugs found – the implementation is correct. The pointer arithmetic in scan_number_bulk_contiguous() is safe because bulk_data() is only called after the current character has been consumed.
SWAR string scanner & UTF‑8 validation – correctly implemented; the UTF‑8 range checks exactly match the byte‑by‑byte scanner, so error messages and positions stay identical.
Number parsers – handle overflow correctly and fall back to float when needed, preserving legacy behaviour.
Simdutf integration – opt‑in and safe; on validation failure it falls back to the scalar path.

What's missing (should be addressed before merge):

Dedicated unit tests for the bulk string/UTF‑8 scanning and for the container adapter (e.g., std::vector<uint8_t>). Currently covered only by fuzzing – explicit test cases would lock in the behaviour.
CMake option for JSON_USE_SIMDUTF – not included yet; users must manually add the header and link the library.
Support for counted iterators – the adapter uses std::distance(current, end) and assumes iterators of the same type. Niche case, worth noting but not a blocker.

…simdjson world) The number scanner converted its already-validated digit buffer with std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno machinery and dominate number-heavy parsing (strtod runs at ~6 M/s). Replace them with dedicated parsers over the validated buffer: - parse_integer_unsigned / parse_integer_signed: accumulate digits with overflow detection, falling back to the float path on overflow exactly as the strtoull/strtoll round-trip check did. Overflow behavior is unchanged for narrower or wider custom number types. - parse_float_fast: Clinger's exact fast path for `double` (<=19 significant digits, |exp10| <= 22, significand < 2^53), where significand * 10^exp is exact under IEEE round-to-nearest. This is the same fast path used by fast_float/simdjson. It is bit-identical to strtod on this subset and declines (falling back to strtod) otherwise. Only `double` uses it; float and long double keep std::strtof/std::strtold via a templated overload. Measured on representative data (g++ 13, -O3): - integers: DOM parse +11%, SAX +25-34% - floats: DOM parse +37%, SAX +70% (clang: float DOM ~1.9x) No dependencies added; header-only and C++11-clean. Existing parser, lexer, conversion and deserialization unit tests pass unchanged; a 3M-value random-double fuzz matches strtod bit-for-bit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

scan_string() read the input one character at a time through the input adapter and classified every byte with a large switch. For contiguous byte buffers we can instead scan 8 bytes at a time with a SWAR word test that finds the first byte needing individual handling (the closing quote, an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append the ordinary run in one go. - input adapters expose supports_bulk_scan / bulk_data / bulk_remaining / bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges (raw pointers in every standard; std::string/std::vector/std::array and friends additionally in C++20 via std::contiguous_iterator). - the lexer gains a bulk_scan capability (gated on lazy_token_string so bypassing the per-character capture cannot lose error diagnostics) and a scan_string_bulk() fast path; streaming/wide/user adapters are unchanged and keep the byte-at-a-time scanner. The run contains no newline (all bytes < 0x20 are treated as special), so position bookkeeping stays exact, and error tokens are still reconstructed lazily from the consumed byte range. The SWAR special-byte test is pure uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean. Measured on representative data, pointer input, g++ 13 -O3 (string values discarded by accept() see the largest gains): long ASCII strings: DOM +4.5x, SAX +14x, accept +17x (to ~2 GB/s) short strings: DOM +15%, SAX +62%, accept +85% escape-heavy: DOM +31%, SAX +26%, accept +28% Same-input parity verified: 200k randomized documents (escapes, multibyte UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR path and the streaming byte path; unit lexer/parser/diagnostic-position/ deserialization/conversions suites pass unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

…I text) The SWAR bulk string path stopped at the first non-ASCII byte and handed every multibyte character to the byte-at-a-time scanner, whose per-byte get()/next_byte_in_range()/add() machinery runs at roughly half the speed of validating straight from the buffer. As a result, dense non-ASCII text (CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII. Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a non-ASCII lead byte, validates one sequence with validate_one_utf8() - which mirrors scan_string()'s per-byte switch ranges exactly (rejecting overlong forms, surrogates, and out-of-range code points) - and appends it in place, continuing until the closing quote, an escape, a control byte, or an ill-formed sequence. All error handling still defers to the byte path, so error messages and positions are byte-for-byte unchanged. Because only well-formed content is fast-pathed and every rejection falls through to the existing scanner, behavior is identical; the win is purely throughput. Measured on pointer input (accept, string values discarded): content g++ 13 clang 18 dense CJK 277 -> 648 ~605 MB/s (~2.3x) dense emoji 299 -> 857 ~702 MB/s (~2.6-2.9x) mixed 90% ASCII 246 -> 331 ~334 MB/s (~1.35x) pure ASCII unchanged (~3.2 / 4.1 GB/s) Verified: 2,000,000 randomized documents built from arbitrary bytes (overlong, surrogate, truncated, out-of-range sequences) accept/reject and parse identically via the contiguous path and the streaming byte path; lexer/parser/diagnostic-position/deserialization/conversions suites pass unchanged. Pure C++11, no intrinsics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

…ths in C++11) json::parse(std::string) - the most common entry point - did not benefit from the contiguous fast paths (bulk string scanning, UTF-8 bulk validation, memcpy for binary formats) in C++11..17: std::string::iterator is a library wrapper, not a raw pointer, and pre-C++20 there is no portable way to prove it contiguous, so supports_bulk_scan was false. Only raw pointers, string literals, and C-arrays (and, in C++20, anything modelling std::contiguous_iterator) took the fast path. Detect contiguous single-byte containers (std::string, std::vector<char>, std::vector<std::uint8_t>, std::string_view, ...) via is_contiguous_byte_ container and route them through an iterator_input_adapter built from data()/data()+size(). The generic iterator-based container overload is constrained to exclude these, so the two overloads are disjoint and there is no ambiguity (a plain competing overload loses to the greedy forwarding-reference container overload on reference binding, and a factory partial-specialization is ambiguous - both were tried and rejected). The pointer keeps the container's own element type, so char_type - and therefore all parsing behavior - is byte-for-byte identical to the iterator path (const char* for std::string, const std::uint8_t* for std::vector<std::uint8_t>); only the raw pointer additionally turns on the fast paths. Lifetimes are unchanged: the container outlives the adapter for the full parse expression, exactly as the iterators it replaces did. Measured, C++11, json::parse/accept(std::string), g++ 13: long ASCII strings: accept 201 -> 3200 MB/s (~16x), parse 174 -> 1444 dense CJK: accept 263 -> 697 MB/s (~2.6x) short strings: accept 163 -> 243 MB/s (~1.5x) Verified: char_type preserved for std::string (char) and std::vector<std::uint8_t> (uint8_t); CBOR/MsgPack round-trips from std::vector<std::uint8_t> unchanged; 1,000,000 randomized documents accept and parse identically via std::string and via std::istream; deserialization/user-defined-input/parser/lexer/conversions/diagnostic- position suites pass (20,480 assertions); warning-clean on g++ and clang in C++11/17/20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

…UTF) The bulk string scanner validates UTF-8 straight from a contiguous buffer. The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8 validation (easy to get subtly wrong - a from-scratch SSE attempt rejected valid CJK), wire in the vetted simdutf library behind an opt-in switch. simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes <simdutf.h> and routes the bulk validator through simdutf::validate_utf8; the project supplies and links simdutf. Undefined (the default), nothing external is included and the portable C++11 scalar path is used, so the library stays header-only and its baseline behavior is unchanged. Design keeps behavior identical either way: - scan_string_bulk() now finds the run up to the next quote/escape/control byte (non-ASCII allowed) and validates it in one shot; on the rare validation failure it recomputes the exact valid prefix with the scalar helper, so ill-formed input still falls through to the byte path and is reported at the same position with the same message. - the per-sequence scalar path is factored into scalar_string_bulk_run() and is the default backend; the refactor is behavior-preserving and does not change scalar throughput. Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse identically across 2,000,000 arbitrary-byte documents and 1,000,000 mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte path); lexer/parser/diagnostic-position/deserialization suites pass under both configurations (20,188 assertions with the backend enabled); warning-clean on g++ and clang, C++11 and C++20, both configurations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

scan_number() reads a number one character at a time through the input adapter (get()) and appends each byte to token_buffer (add()) before converting. For contiguous input, the per-character get()/add() overhead dominates: it is roughly two thirds of the time spent on number-heavy parsing, far more than the value conversion itself. Add scan_number_bulk_contiguous(), which parses the whole number token straight from the input buffer: it validates and classifies the extent with the same grammar as scan_number()'s state machine, materializes token_buffer in one copy (substituting the locale decimal point exactly as scan_number() does), advances the adapter, and reuses the shared convert_number() tail. On anything it does not recognize as a well-formed number it makes no state change and returns token_type::uninitialized, so the caller falls back to scan_number(), which then produces the exact diagnostic. Errors and their positions are therefore unchanged. The conversion tail is factored out of scan_number() into convert_number() so both scanners share it; the fast path is selected by tag dispatch on the existing bulk_scan capability, so streaming/wide/user adapters are unaffected. Measured on pointer input, g++ 13 -O3: - integers: parse +65%, accept +98% - floats: parse +39%, accept +70% Verified: 2,000,000 randomized number documents (including overflow-range integers, long digit strings and %.17g doubles) parse identically via the contiguous path and the streaming byte path, matching value, type and round-trip text; the locale suite and existing parser/lexer/conversions/ deserialization tests pass; a new "lexer number fast path" test checks contiguous-vs-streaming parity, token classification, and that malformed numbers are rejected identically. Pure C++11, no intrinsics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

lexer.hpp had grown by ~600 lines of byte-level helpers that have no dependency on the lexer's template parameters and clutter the state machine. Move them, unchanged, into two focused headers as free functions in namespace detail: - number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now templated on the number type) and parse_float_fast (Clinger's exact double fast path, with the decimal point passed as an argument instead of read from a lexer member). - string_scan.hpp: the SWAR string helpers (is_string_special, swar_string_special, find_string_special, validate_one_utf8, scalar_string_bulk_run) and the backend-dispatched string_bulk_run, including the optional simdutf include and find_string_delimiter. lexer.hpp now includes these and calls the free functions; the methods that touch lexer state (scan_string, scan_number, scan_string_bulk, scan_number_bulk_contiguous, convert_number) stay put. This is a pure code move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the now-unused <cstdint>/<cstring>/<limits> includes are removed, and the free-function form makes the SWAR helpers reusable elsewhere (e.g. the serializer's string escaping). Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and 2,000,000 arbitrary-byte-string differential-fuzz documents parse identically to before; lexer/parser/conversions/deserialization/locale/ diagnostic-position suites pass (20,576 assertions); warning-clean on g++ and clang in C++11/17/20; the amalgamation regenerates and passes check-amalgamation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

- number_parse.hpp: use std::array for the powers-of-ten table (avoid-c-arrays) and `auto` for the cast-initialized result (modernize-use-auto), matching the codebase style (cf. the serializer's utf8d table). Indexing casts keep the -Wsign-conversion build clean. - add JSON_USE_SIMDUTF to the mkdocs navigation so the macro page is reachable. No behavior change; clang-tidy is clean on the new headers and the amalgamation is regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The contiguous number fast path materialized token_buffer with token_buffer.assign(data, len), but string_t is only required to provide the minimal interface the rest of the lexer uses (push_back, append, clear, operator[], ...). Custom string types such as the test's alt_string do not implement assign(), so scan_number_bulk_contiguous() failed to compile for them (unit-alt-string), breaking the gcc/clang standards and old-compiler CI jobs. reset() already clears token_buffer, so fill it with append() - which alt_string and std::string both provide and which the string fast path already relies on - instead of assign(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

…rence The CI clang-tidy (newer than the locally available version) reported two additional checks on the new code: - readability-math-missing-parentheses: parenthesize the (a * b) + c digit accumulations in number_parse.hpp. - cppcoreguidelines-missing-std-forward: the contiguous-byte-container input_adapter overload took a forwarding reference but only reads data()/size() and never forwards it. It is already disjoint from the generic container overload via SFINAE, so a plain const& is correct and clearer (and keeps the container alive for the whole parse just as before). No behavior change; char_type and routing are unchanged (std::string and std::vector<std::uint8_t> still take the pointer adapter with char/uint8_t char_type), CBOR/MsgPack round-trips and the 2M number fuzz still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The Clinger fast path is exact only for the "easy" subset (<=19 significant digits, |exp10| <= 22); high-precision and scientific floats fall through to strtod, where the failed Clinger attempt actually makes parsing a net loss. std::from_chars implements the Eisel-Lemire algorithm in modern standard libraries: locale-independent, correctly rounded, and fast over the whole value range. convert_number() now tries parse_float_from_chars() first (guarded by __cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is used only when it consumes the entire token; a partial parse means a non-'.' locale decimal point, and an under-/overflow (result_out_of_range) also declines - in both cases the existing strtod fallback supplies the exact value and the well-defined +/-inf/0 the parser expects, side-stepping the P4168 divergence between implementations. float and long double now get the fast path too (Clinger was double-only). Measured, C++17, g++ 13 -O3, json::parse/accept: - canada-style floats: ~unchanged (Clinger already covered them) - high-precision (17 digits): parse 2.1x, accept 2.5x - scientific (17 digits + exp): parse 3.6x, accept 4.1x Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value - including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow - to bit-identical results; 2M number-fuzz clean; conversions/deserialization/ locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy clean; warning-clean on g++ and clang in C++11/17/20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

libstdc++ 15 defines __cpp_lib_to_chars even in C++14 mode (via bits/version.h pulled in by other headers), but <charconv> is only included under JSON_HAS_CPP_17. That made parse_float_from_chars() reference std::from_chars without the header in C++14 builds, breaking gcc-latest, icpx, and the offline-testdata jobs. Gate the use on JSON_HAS_CPP_17 && __cpp_lib_to_chars so it matches the include condition exactly; C++11/14 always take the scalar fallback. Verified by forcing __cpp_lib_to_chars in a C++14 build: the guard suppresses std::from_chars and it compiles. C++17 behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The contiguous number fast path uses a Clinger-style exact algorithm (significand * 10^scale in double arithmetic), which is only correctly rounded when double operations are evaluated in true 53-bit precision. On the x87 FPU used by 32-bit x86 (FLT_EVAL_METHOD == 2) the single multiply/divide is computed in 80-bit and then double-rounded to double, so a small fraction of values land 1 ULP off. This surfaced as test-cbor_cpp11 and test-msgpack_cpp11 failing on the mingw (x86) job for regression/floats.json: the C++17 builds pass because they take the correctly-rounded std::from_chars path, while C++11 falls back to parse_float_fast(). A 5M-sample check over shortest round-trip decimals reproduces it: 0 divergences with 53-bit doubles, ~1 in 25 000 with 80-bit intermediates; declining to std::strtod fixes all of them. Guard parse_float_fast() on FLT_EVAL_METHOD so it declines whenever the platform evaluates doubles in extended precision, letting the caller use the correctly-rounded std::from_chars / std::strtod path instead. On mainstream x86-64/ARM64 (FLT_EVAL_METHOD == 0) the fast path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

GCC 7 sets __cplusplus to the C++17 value under -std=gnu++1z, so JSON_HAS_CPP_17 is defined, but its libstdc++ ships no <charconv> header (added in GCC 8; floating-point from_chars in GCC 11). The unconditional "#if defined(JSON_HAS_CPP_17) #include <charconv>" therefore failed to compile there: "fatal error: charconv: No such file or directory" in the ci_test_compilers_gcc (7) job. Wrap the include in __has_include(<charconv>), mirroring the library's existing handling of <version> and <filesystem> in macro_scope.hpp. When the header is absent, __cpp_lib_to_chars stays undefined and parse_float_from_chars() takes its scalar fallback, so the from_chars use site (already gated on __cpp_lib_to_chars) is never reached. GCC 8-10, which have <charconv> but no floating-point from_chars, are unaffected: they include the header but still take the fallback. GCC 11+ is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The contiguous number fast path never reads the character that terminates a number token, while scan_number() reads it and then ungets it. When that character is a newline, get() has already cleared chars_read_current_line, and unget() could only restore lines_read - leaving the column at 0. The two paths therefore reported different columns for the same document: json::parse("[01\n]") -> line 1, column 3 json::parse(stringstream) -> line 1, column 0 Remember the column the newline was read at so unget() can restore it. Both paths now report the position the offending token actually starts at, which also fixes the pre-existing column-0 artifact for streaming input. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

supports_bulk_scan required IteratorType and SentinelType to be the same type, which excluded std::counted_iterator paired with std::default_sentinel_t - the combination #5268 had already enabled for the memcpy fast path. Such input fell back to the byte-at-a-time scanner even though it is contiguous and its remaining length is computable in O(1). Factor the "distance is computable in O(1)" test into sentinel_is_sized and use it for iterator_is_contiguous, supports_seek, and supports_bulk_scan alike, and share the std::ranges::distance/std::distance dispatch through a remaining_count() helper. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The macro was only listed in the API macro index; add it to the supported macros overview alongside the other JSON_USE_* macros, and note that it selects between two definitions of the same inline function and so must be defined identically in every translation unit. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Sized sentinels newly reach the bulk string/number scanners and the seek-based token reconstruction, so exercise both: - diagnostics that quote the offending token, which are rebuilt from the consumed input via copy_consumed_range() - inputs whose count ends before the underlying buffer does, including a closing quote that exists only behind the count, a cut inside an 8-byte SWAR stride, and a cut inside a UTF-8 sequence Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The comment offered "a C++20 sentinel or counted_iterator" as examples of a SentinelType, but std::counted_iterator is the IteratorType - the sentinel it pairs with is std::default_sentinel_t. #5268 corrected the same wording in the API documentation and left the code comment behind. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

json::parse is declared warn_unused_result, and CHECK_THROWS_WITH_AS evaluates its expression as a discarded statement, so the assertion broke the -Werror builds (GCC -Werror=unused-result, MSVC C4834 under /WX). Compare against the helper that already captures the message instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The JSON number grammar is encoded twice: as the scan_number() state machine and as the contiguous fast path. The fast path declining on anything it does not recognize keeps most divergence harmless, but if it ever accepted something the state machine rejects the result would be a silent correctness bug, and the existing test only pinned a hand-written list of numbers. Enumerate every string of length 1..4 over "01.eE+-" (2800 tokens) and require both paths to agree on the parsed value and on the exact error message. Verified to fail if the fast path's grammar is perturbed. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

An integer token does not need token_buffer: the number_integer and number_unsigned SAX callbacks take only the value, and the overflow diagnostic rebuilds the text from the input via get_token_string(). Convert straight from the input buffer and materialize the token only for the floating-point tail, which still needs a NUL-terminated buffer for strtod. JSON_DIAGNOSTIC_POSITIONS derives a number's start position from get_string().size(), so the copy is kept when that is enabled. The integer dispatch is factored into convert_integer() and shared with convert_number(), so both scanners keep using one implementation. Integer-heavy input, 400k values, -O3: parse accept gcc 16 +14% +23% clang +15% +22% Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

These paths had no dedicated tests and rested on differential fuzzing only. Add three sections, all comparing the contiguous scanner against the byte-at-a-time one on the parsed value and on the exact error message: - every string of length 1..3 over an alphabet of ordinary ASCII, both specials, a control byte, escape characters, UTF-8 lead and continuation bytes, and a byte that is never valid - each at offset 0 and offset 9, so the bulk scanner sees them with and without a run behind them - every kind of run-ending byte at each offset across two 8-byte SWAR words, so multibyte sequences also straddle the word boundary - the boundaries of every range validate_one_utf8() recognizes: shortest and longest encodings, overlongs, both ends of the surrogate block, U+10FFFF and just past it, and truncated sequences Verified to fail if the bulk validator accepts surrogates, and if the SWAR word test stops detecting control characters. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The pinned astyle expands a braced-init-list used as a range-for range onto several lines; hoist the two offsets into a named vector instead, which reads better and leaves nothing for astyle to reformat. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Two problems in the tests added for the bulk scanners, both found by CI: - the inner `const json j` in the counted-iterator diagnostics shadowed the one declared at test-case scope, which -Wshadow rejects on GCC and clang and C4456 rejects on MSVC under /WX; rename them - the new parity checks parse deliberately invalid input, which calls std::abort() rather than throwing when JSON_NOEXCEPTION is defined, so they would have crashed the no-exception build; guard them the way the other tests do json::accept() does not abort, so the UTF-8 range assertions stay compiled without exceptions and keep covering validate_one_utf8() there. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Making supports_bulk_scan depend on iterator_is_contiguous meant the trait is now instantiated for every adapter, not only when get_elements() is called. On standard libraries with an incomplete <ranges> that is fatal: libstdc++ 10 evaluates std::contiguous_iterator<std::counted_iterator<T*>> by calling std::to_address, which needs an operator-> its counted_iterator does not have, so satisfaction checking is a hard error rather than false. Reported by clang 14 + libstdc++ 10. JSON_HAS_RANGES already encodes exactly this ("libstdc++ < 11 has incomplete C++20 ranges", #4440), so require it for the C++20 branch. Affected toolchains fall back to the pointer-only test and the byte-at-a-time scanner, which parses identically, just without the bulk fast paths. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

- give the helper lambdas an explicit std::string return type and return braced initializer lists (modernize-return-braced-init-list) - replace the C-style array of test cases with a std::vector (modernize-avoid-c-arrays) - silence pro-type-member-init on the two brace-initialized aggregates; default member initializers would stop them being aggregates in C++11 Signed-off-by: Niels Lohmann <mail@nlohmann.me>

clang-tidy reads "[\"\\ud834\"]" as a literal better written raw, and the two literals written next to each other in "[\"a\x01""b\"]" as a missing comma. The concatenation was there to stop the hex escape swallowing the following character; build those documents from explicit bytes instead and use raw strings elsewhere. The byte sequences are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

@gregmarr

@gregmarr spotted that convert_integer() was inserted between convert_number() and its doc block, leaving convert_integer() with two stacked blocks and convert_number() with none. Comment only; no code change. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

#5386 reformatted this example on develop and extended the CI format check to cover docs/mkdocs/docs/examples, which this branch predates. astyle rewrites "json & /*parsed*/" to "json& /*parsed*/" per --align-reference=type; the result is byte-identical to develop's copy. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both arms of the `#if`. The simdutf arm already falls through when validation fails, so a single return after the `#endif` says the same thing. The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which string_scan.hpp already does under the same guard; users only have to put the header on the include path and link the library, not include it themselves. Set the version history entry to 3.13.0, matching the other macro pages documenting unreleased features. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The comment claimed the reported column is the one the offending token starts at. It is the column reached after the token's last character - which is the actual point of the unget() change: a number terminated by a newline now reports what the same number terminated by a space always did. Assert that equality directly, and add a multi-character token where the start and end columns differ, so the invariant cannot be read off a single-character example. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me>

scan_number_bulk_contiguous() converts an integer token straight from the input buffer. When the value does not fit, it materializes token_buffer and calls convert_number(), which tried the very same integer conversion again before falling back to floating point. Recording the outcome in number_type skips the second attempt. The resulting token type and value are unchanged: convert_number() reached the float tail either way. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me>

is_contiguous_byte_container accepted any type with a data() returning a pointer to a single-byte integral plus a size(). That is duck typing: the two members say nothing about size() counting the units data() points at. A type where it does not - fixed-size records, say - was routed to the pointer-based adapter and parsed as [data(), data() + size()) bytes, silently truncating input the iterator-based adapter had read in full: struct record_buffer { using value_type = std::array<char, 4>; std::string bytes; const char* data() const; // raw bytes std::size_t size() const; // in records const char* begin() const; const char* end() const; }; json::parse(record_buffer{"[1,2,3,4,5]"}); // parse error at column 3 Requiring the container's own value_type to be that same element type ties the two together. Every contiguous standard container satisfies it, so std::string, std::vector<char>, std::array<char, N> and std::string_view keep the fast path; anything else falls back to the iterator-based adapter, which is always correct. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me>

simdutf.h rejects anything below C++17 with an #error, so defining JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a message about simdutf being unavailable - it failed to compile at all, taking the library's C++11 support with it. Nothing caught this because no build ever compiled that path. Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp gates std::from_chars. Below C++17 the macro now has no effect and the scalar validator runs; it accepts and rejects exactly the same input, so the macro is safe to set project-wide even when some translation units use an older standard. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me>

JSON_USE_SIMDUTF was documented and shipped but never compiled by anything in the repository, so nothing held the backend to the behavior the docs promise. Add JSON_TestSimdutf (OFF by default), which fetches simdutf and defines JSON_USE_SIMDUTF for every test target, and a ci_test_simdutf target that runs the whole suite in that configuration. Because simdutf needs C++17, the suite is built at C++11 as well, so one job covers both the scalar fallback with the macro defined and simdutf itself. The dependency hangs off test_main, whose usage requirements every test target inherits. The library target and the installed CMake package are deliberately untouched: making nlohmann_json link simdutf would put a find_dependency() in the exported package, which is a separate decision. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me>

simdutf needs C++17: without it the dependency does not even compile, and with a C++17 compiler but no C++17-or-later standard under test it builds and then goes unused. Either way the option silently did nothing useful, or broke the configure step outright. Resolve the tested standards first, then check them: when none of them can reach simdutf, skip the dependency and say so, naming which of the two reasons applies and how to fix it. The tests then run against the scalar validator, which is what would have happened anyway. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me>

github-actions Bot added documentation L labels Jul 20, 2026

nlohmann force-pushed the claude/parser-performance-research-4hkdf8 branch from c724f41 to 84939c1 Compare July 20, 2026 09:51

github-actions Bot added the tests label Jul 20, 2026

nlohmann changed the title ~~Speed up parsing: fast number/string/UTF-8 paths for contiguous input (with optional simdutf)~~ Speed up parsing of contiguous input: numbers, strings, and UTF-8 (with optional simdutf) Jul 20, 2026

nlohmann mentioned this pull request Jul 20, 2026

Speed up dump(), and keep it from overflowing the stack #5285

Draft

5 tasks

gregmarr reviewed Jul 28, 2026

View reviewed changes

Comment thread include/nlohmann/detail/input/input_adapters.hpp Outdated

nlohmann force-pushed the claude/parser-performance-research-4hkdf8 branch from 4183304 to c4a3b4d Compare August 4, 2026 06:58

nlohmann and others added 20 commits August 19, 2026 20:26

Amalgamate source code

ebc563e

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

nlohmann force-pushed the claude/parser-performance-research-4hkdf8 branch from c4a3b4d to 509494e Compare August 19, 2026 18:54

nlohmann added 6 commits August 19, 2026 22:44

Amalgamate source code

3db2b5d

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

This comment was marked as resolved.

Sign in to view

nlohmann added 5 commits August 20, 2026 03:02

Amalgamate source code

a5e94f4

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

gregmarr reviewed Aug 20, 2026

View reviewed changes

Comment thread include/nlohmann/detail/input/lexer.hpp

This comment was marked as resolved.

Sign in to view

nlohmann marked this pull request as ready for review August 25, 2026 06:14

nlohmann commented Aug 28, 2026

View reviewed changes

Comment thread docs/mkdocs/docs/api/macros/json_use_simdutf.md Outdated

Comment thread include/nlohmann/detail/input/string_scan.hpp Outdated

github-actions Bot added CI CMake labels Aug 28, 2026

nlohmann and others added 7 commits August 28, 2026 14:59

nlohmann force-pushed the claude/parser-performance-research-4hkdf8 branch from 622e91e to d8cfdc9 Compare August 28, 2026 15:07

Navigation Menu

Uh oh!

Speed up parsing of contiguous input: numbers, strings, and UTF-8 (with optional simdutf) - #5283

Speed up parsing of contiguous input: numbers, strings, and UTF-8 (with optional simdutf)#5283
nlohmann wants to merge 40 commits into
developfrom
claude/parser-performance-research-4hkdf8

nlohmann commented Jul 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

wilkolbrzym-coder commented Aug 18, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

nlohmann commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What & why

The changes (each independent and reversible)

Measured gains

Public API impact

One deliberate behavior change: error column after a newline

Verification

Notes for reviewers / remaining work

Uh oh!

Uh oh!

wilkolbrzym-coder commented Aug 18, 2026

Quick review notes – PR #5283

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nlohmann commented Jul 20, 2026 •

edited

Loading