Skip to content

Commit 84939c1

Browse files
nlohmannclaude
nlohmann
and
claude
committed
Add a contiguous fast path for scanning numbers
scan_number() reads a number one character at a time through the input adapter (get()) and appends each byte to token_buffer (add()) before converting. For contiguous input, the per-character get()/add() overhead dominates: it is roughly two thirds of the time spent on number-heavy parsing, far more than the value conversion itself. Add scan_number_bulk_contiguous(), which parses the whole number token straight from the input buffer: it validates and classifies the extent with the same grammar as scan_number()'s state machine, materializes token_buffer in one copy (substituting the locale decimal point exactly as scan_number() does), advances the adapter, and reuses the shared convert_number() tail. On anything it does not recognize as a well-formed number it makes no state change and returns token_type::uninitialized, so the caller falls back to scan_number(), which then produces the exact diagnostic. Errors and their positions are therefore unchanged. The conversion tail is factored out of scan_number() into convert_number() so both scanners share it; the fast path is selected by tag dispatch on the existing bulk_scan capability, so streaming/wide/user adapters are unaffected. Measured on pointer input, g++ 13 -O3: - integers: parse +65%, accept +98% - floats: parse +39%, accept +70% Verified: 2,000,000 randomized number documents (including overflow-range integers, long digit strings and %.17g doubles) parse identically via the contiguous path and the streaming byte path, matching value, type and round-trip text; the locale suite and existing parser/lexer/conversions/ deserialization tests pass; a new "lexer number fast path" test checks contiguous-vs-streaming parity, token classification, and that malformed numbers are rejected identically. Pure C++11, no intrinsics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>
1 parent 190f4b6 commit 84939c1

3 files changed

Lines changed: 350 additions & 6 deletions

File tree

‎include/nlohmann/detail/input/lexer.hpp‎

Lines changed: 139 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1769,12 +1769,26 @@ class lexer : public lexer_base<BasicJsonType>
17691769
// we are done scanning a number)
17701770
unget();
17711771

1772+
return convert_number(number_type);
1773+
}
1774+
1775+
/*!
1776+
@brief convert the number text in token_buffer to its value and token type
1777+
1778+
The digit sequence in token_buffer has already been validated (by the
1779+
scan_number() state machine or by the contiguous fast path) and holds the
1780+
locale decimal point in place of '.'. Integers are parsed first and fall
1781+
back to floating point on overflow. This is shared so both scanners produce
1782+
identical results.
1783+
*/
1784+
token_type convert_number(token_type number_type)
1785+
{
17721786
const char* const num_begin = token_buffer.data();
17731787
const char* const num_end = num_begin + token_buffer.size();
17741788

17751789
// try to parse integers first and fall back to floats; the digit
1776-
// sequence has already been validated by the state machine above, so
1777-
// a dedicated parser can avoid the locale/errno overhead of strtoull
1790+
// sequence has already been validated, so a dedicated parser can avoid
1791+
// the locale/errno overhead of strtoull
17781792
if (number_type == token_type::value_unsigned)
17791793
{
17801794
if (parse_integer_unsigned(num_begin, num_end, value_unsigned))
@@ -1807,6 +1821,128 @@ class lexer : public lexer_base<BasicJsonType>
18071821
return token_type::value_float;
18081822
}
18091823

1824+
/*!
1825+
@brief contiguous fast path for scanning a number
1826+
1827+
Parses the whole number token straight from the input buffer, avoiding the
1828+
per-character get()/add() of scan_number(). On success it fills token_buffer
1829+
(with the locale decimal point substituted, as scan_number() does) and
1830+
returns the token type. On anything it does not fully recognize as a
1831+
well-formed number it makes no state change and returns
1832+
token_type::uninitialized, so the caller falls back to scan_number(), which
1833+
then produces the exact diagnostic. @a current is the first digit or the
1834+
leading minus (already read); the remaining bytes are taken from the adapter.
1835+
*/
1836+
token_type scan_number_bulk_contiguous()
1837+
{
1838+
// a pending unget offsets the buffer position from current; fall back
1839+
if (next_unget)
1840+
{
1841+
return token_type::uninitialized;
1842+
}
1843+
const std::size_t rem = ia.bulk_remaining();
1844+
if (rem == 0)
1845+
{
1846+
// the first digit is the last input byte; let scan_number() finish
1847+
return token_type::uninitialized;
1848+
}
1849+
// the byte before the next unread one is current (contiguous input)
1850+
const char* const data = reinterpret_cast<const char*>(ia.bulk_data()) - 1;
1851+
const std::size_t avail = rem + 1;
1852+
1853+
// validate + classify the number extent (mirrors scan_number()'s grammar)
1854+
std::size_t i = 0;
1855+
std::size_t dot_index = std::string::npos;
1856+
token_type number_type = token_type::value_unsigned;
1857+
if (data[0] == '-')
1858+
{
1859+
number_type = token_type::value_integer;
1860+
i = 1;
1861+
if (i >= avail)
1862+
{
1863+
return token_type::uninitialized;
1864+
}
1865+
}
1866+
if (data[i] == '0')
1867+
{
1868+
++i;
1869+
}
1870+
else if (data[i] >= '1' && data[i] <= '9')
1871+
{
1872+
++i;
1873+
while (i < avail && data[i] >= '0' && data[i] <= '9')
1874+
{
1875+
++i;
1876+
}
1877+
}
1878+
else
1879+
{
1880+
return token_type::uninitialized;
1881+
}
1882+
if (i < avail && data[i] == '.')
1883+
{
1884+
number_type = token_type::value_float;
1885+
dot_index = i;
1886+
++i;
1887+
if (i >= avail || !(data[i] >= '0' && data[i] <= '9'))
1888+
{
1889+
return token_type::uninitialized;
1890+
}
1891+
while (i < avail && data[i] >= '0' && data[i] <= '9')
1892+
{
1893+
++i;
1894+
}
1895+
}
1896+
if (i < avail && (data[i] == 'e' || data[i] == 'E'))
1897+
{
1898+
number_type = token_type::value_float;
1899+
++i;
1900+
if (i < avail && (data[i] == '+' || data[i] == '-'))
1901+
{
1902+
++i;
1903+
}
1904+
if (i >= avail || !(data[i] >= '0' && data[i] <= '9'))
1905+
{
1906+
return token_type::uninitialized;
1907+
}
1908+
while (i < avail && data[i] >= '0' && data[i] <= '9')
1909+
{
1910+
++i;
1911+
}
1912+
}
1913+
const std::size_t len = i;
1914+
1915+
// materialize the token exactly as scan_number() would, substituting the
1916+
// locale decimal point so convert_number()'s strtof fallback stays valid
1917+
reset();
1918+
token_buffer.assign(data, len);
1919+
if (dot_index != std::string::npos)
1920+
{
1921+
token_buffer[dot_index] = static_cast<typename string_t::value_type>(decimal_point_char);
1922+
decimal_point_position = dot_index;
1923+
}
1924+
1925+
// consume the remaining bytes of the number (current was already read)
1926+
ia.bulk_skip(len - 1);
1927+
position.chars_read_total += (len - 1);
1928+
position.chars_read_current_line += (len - 1);
1929+
1930+
return convert_number(number_type);
1931+
}
1932+
1933+
/// contiguous input: try the number fast path, else the byte-path scanner
1934+
token_type scan_number_dispatch(std::true_type /*bulk*/)
1935+
{
1936+
const token_type t = scan_number_bulk_contiguous();
1937+
return (t != token_type::uninitialized) ? t : scan_number();
1938+
}
1939+
1940+
/// streaming input: always use the byte-path scanner
1941+
token_type scan_number_dispatch(std::false_type /*bulk*/)
1942+
{
1943+
return scan_number();
1944+
}
1945+
18101946
/*!
18111947
@param[in] literal_text the literal text to expect
18121948
@param[in] length the length of the passed literal text
@@ -2161,7 +2297,7 @@ class lexer : public lexer_base<BasicJsonType>
21612297
case '7':
21622298
case '8':
21632299
case '9':
2164-
return scan_number();
2300+
return scan_number_dispatch(std::integral_constant<bool, bulk_scan> {});
21652301

21662302
// end of input (the null byte is needed when parsing from
21672303
// string literals)

‎single_include/nlohmann/json.hpp‎

Lines changed: 139 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9532,12 +9532,26 @@ class lexer : public lexer_base<BasicJsonType>
95329532
// we are done scanning a number)
95339533
unget();
95349534

9535+
return convert_number(number_type);
9536+
}
9537+
9538+
/*!
9539+
@brief convert the number text in token_buffer to its value and token type
9540+
9541+
The digit sequence in token_buffer has already been validated (by the
9542+
scan_number() state machine or by the contiguous fast path) and holds the
9543+
locale decimal point in place of '.'. Integers are parsed first and fall
9544+
back to floating point on overflow. This is shared so both scanners produce
9545+
identical results.
9546+
*/
9547+
token_type convert_number(token_type number_type)
9548+
{
95359549
const char* const num_begin = token_buffer.data();
95369550
const char* const num_end = num_begin + token_buffer.size();
95379551

95389552
// try to parse integers first and fall back to floats; the digit
9539-
// sequence has already been validated by the state machine above, so
9540-
// a dedicated parser can avoid the locale/errno overhead of strtoull
9553+
// sequence has already been validated, so a dedicated parser can avoid
9554+
// the locale/errno overhead of strtoull
95419555
if (number_type == token_type::value_unsigned)
95429556
{
95439557
if (parse_integer_unsigned(num_begin, num_end, value_unsigned))
@@ -9570,6 +9584,128 @@ class lexer : public lexer_base<BasicJsonType>
95709584
return token_type::value_float;
95719585
}
95729586

9587+
/*!
9588+
@brief contiguous fast path for scanning a number
9589+
9590+
Parses the whole number token straight from the input buffer, avoiding the
9591+
per-character get()/add() of scan_number(). On success it fills token_buffer
9592+
(with the locale decimal point substituted, as scan_number() does) and
9593+
returns the token type. On anything it does not fully recognize as a
9594+
well-formed number it makes no state change and returns
9595+
token_type::uninitialized, so the caller falls back to scan_number(), which
9596+
then produces the exact diagnostic. @a current is the first digit or the
9597+
leading minus (already read); the remaining bytes are taken from the adapter.
9598+
*/
9599+
token_type scan_number_bulk_contiguous()
9600+
{
9601+
// a pending unget offsets the buffer position from current; fall back
9602+
if (next_unget)
9603+
{
9604+
return token_type::uninitialized;
9605+
}
9606+
const std::size_t rem = ia.bulk_remaining();
9607+
if (rem == 0)
9608+
{
9609+
// the first digit is the last input byte; let scan_number() finish
9610+
return token_type::uninitialized;
9611+
}
9612+
// the byte before the next unread one is current (contiguous input)
9613+
const char* const data = reinterpret_cast<const char*>(ia.bulk_data()) - 1;
9614+
const std::size_t avail = rem + 1;
9615+
9616+
// validate + classify the number extent (mirrors scan_number()'s grammar)
9617+
std::size_t i = 0;
9618+
std::size_t dot_index = std::string::npos;
9619+
token_type number_type = token_type::value_unsigned;
9620+
if (data[0] == '-')
9621+
{
9622+
number_type = token_type::value_integer;
9623+
i = 1;
9624+
if (i >= avail)
9625+
{
9626+
return token_type::uninitialized;
9627+
}
9628+
}
9629+
if (data[i] == '0')
9630+
{
9631+
++i;
9632+
}
9633+
else if (data[i] >= '1' && data[i] <= '9')
9634+
{
9635+
++i;
9636+
while (i < avail && data[i] >= '0' && data[i] <= '9')
9637+
{
9638+
++i;
9639+
}
9640+
}
9641+
else
9642+
{
9643+
return token_type::uninitialized;
9644+
}
9645+
if (i < avail && data[i] == '.')
9646+
{
9647+
number_type = token_type::value_float;
9648+
dot_index = i;
9649+
++i;
9650+
if (i >= avail || !(data[i] >= '0' && data[i] <= '9'))
9651+
{
9652+
return token_type::uninitialized;
9653+
}
9654+
while (i < avail && data[i] >= '0' && data[i] <= '9')
9655+
{
9656+
++i;
9657+
}
9658+
}
9659+
if (i < avail && (data[i] == 'e' || data[i] == 'E'))
9660+
{
9661+
number_type = token_type::value_float;
9662+
++i;
9663+
if (i < avail && (data[i] == '+' || data[i] == '-'))
9664+
{
9665+
++i;
9666+
}
9667+
if (i >= avail || !(data[i] >= '0' && data[i] <= '9'))
9668+
{
9669+
return token_type::uninitialized;
9670+
}
9671+
while (i < avail && data[i] >= '0' && data[i] <= '9')
9672+
{
9673+
++i;
9674+
}
9675+
}
9676+
const std::size_t len = i;
9677+
9678+
// materialize the token exactly as scan_number() would, substituting the
9679+
// locale decimal point so convert_number()'s strtof fallback stays valid
9680+
reset();
9681+
token_buffer.assign(data, len);
9682+
if (dot_index != std::string::npos)
9683+
{
9684+
token_buffer[dot_index] = static_cast<typename string_t::value_type>(decimal_point_char);
9685+
decimal_point_position = dot_index;
9686+
}
9687+
9688+
// consume the remaining bytes of the number (current was already read)
9689+
ia.bulk_skip(len - 1);
9690+
position.chars_read_total += (len - 1);
9691+
position.chars_read_current_line += (len - 1);
9692+
9693+
return convert_number(number_type);
9694+
}
9695+
9696+
/// contiguous input: try the number fast path, else the byte-path scanner
9697+
token_type scan_number_dispatch(std::true_type /*bulk*/)
9698+
{
9699+
const token_type t = scan_number_bulk_contiguous();
9700+
return (t != token_type::uninitialized) ? t : scan_number();
9701+
}
9702+
9703+
/// streaming input: always use the byte-path scanner
9704+
token_type scan_number_dispatch(std::false_type /*bulk*/)
9705+
{
9706+
return scan_number();
9707+
}
9708+
95739709
/*!
95749710
@param[in] literal_text the literal text to expect
95759711
@param[in] length the length of the passed literal text
@@ -9924,7 +10060,7 @@ class lexer : public lexer_base<BasicJsonType>
992410060
case '7':
992510061
case '8':
992610062
case '9':
9927-
return scan_number();
10063+
return scan_number_dispatch(std::integral_constant<bool, bulk_scan> {});
992810064

992910065
// end of input (the null byte is needed when parsing from
993010066
// string literals)

0 commit comments

Comments
 (0)