fix: restore the character that terminates a number

[nlohmann]

Fixes #5340. Draft: this changes observable behavior of operator>> and sax_parse(strict = false), so it is a candidate for the next major release rather than a patch release. The version numbers in the docs assume 4.0.0 and need adjusting if that changes.

The bug

A number is the only JSON value whose end can only be detected by looking at the character that follows it. lexer::scan_number() reads that character and calls lexer::unget(), but unget() is deliberately simulated — it rewinds only the lexer's own bookkeeping (chars_read_total, chars_read_current_line, token_string), not the input. input_stream_adapter::get_character() consumed via sbumpc(), so the terminating character stayed consumed:

std::istringstream input("1true");
json j1;
input >> j1;   // j1 == 1, but the stream now starts at "rue"

This is invisible to parse()/accept(), which require the input to end after the value anyway. It is only observable where the caller keeps using the input: operator>> and sax_parse(strict = false).

The fix

Following @gregmarr's suggestion, the adapter now peeks instead of consuming, rather than consuming and trying to put the character back afterwards.

• input_stream_adapter::get_character() returns sb->sgetc() and remembers that a character is pending; the next call steps over it with sbumpc() before peeking again. get_elements() and the destructor commit the pending character the same way, so every other user of the adapter — and the stream the caller gets back — sees the position it saw before.
• input_stream_adapter::release_lookahead() drops the pending character instead of committing it, leaving it in the stream. The adapter advertises this with supports_lookahead, detected with the same is_detected tag-dispatch idiom already used for supports_seek; other adapters compile the call away to a no-op.
• lexer::release_lookahead() forwards a pending simulated unget to the adapter and clears next_unget, so the character is read from the input again rather than replayed from current. A pending unget of EOF needs no special case: reaching EOF leaves no lookahead to release.
• parser calls it on the three non-strict paths (both parse() branches and sax_parse), folded into the existing if (strict && …) checks.

The invariant that makes this safe: get() clears next_unget whenever it consumes it, so a pending unget always refers to the most recent adapter read, and there is never more than one.

Why not sungetc()

The first version of this PR consumed with sbumpc() and put the character back with sungetc(). That is best-effort: sungetc() fails when the streambuf has no putback position, and the character stays consumed. Peeking cannot fail — the character is never consumed in the first place, so no putback position is required. The no_putback_streambuf test (a streambuf with no get area whose pbackfail() always fails) covers exactly that case and now passes.

Verification

input	before	after
1true	rue ❌	true ✅
1 true	true ❌	true ✅
1[2]	2] ❌	[2] ✅
1{}	} ❌	{} ✅
1"a"	a" ❌	"a" ✅
-0.5e3x	`` ❌	x ✅

Note 1 true was wrong before too — the issue's table lists it as fine because the swallowed byte happened to be whitespace, but the position was still off by one.

• New tests in tests/src/unit-deserialization.cpp (stream position after extraction (#5340)): number terminators for every following value type, self-delimiting values, a number at end of input, repeated extraction of 1true[2]3"x"{"a":4}5, sax_parse(strict = false), strict parsing still rejecting trailing data, and the streambuf that cannot put back. 8 assertions in this section fail against develop (and the repeated-extraction subcase throws).
• unit-deserialization, unit-class_lexer, unit-class_parser, unit-class_parser_diagnostic_positions, unit-user_defined_input, unit-regression2, and unit-disabled_exceptions pass against both include/ and single_include/ — 20,411 assertions each, 0 failures.
• Parse error messages and reported positions were diffed across 13 malformed inputs (string and stream paths) and are byte-identical to develop.

Public API

No breaking changes to the public API surface: no signature, type, or name changes; nothing added to or removed from the public interface. lexer::release_lookahead(), input_stream_adapter::release_lookahead(), and input_stream_adapter::supports_lookahead are new members in detail::.

There is a behavior change, which is why this is a draft targeting the next major release:

• operator>> and sax_parse(strict = false) on a std::istream now leave the stream one byte earlier when the parsed value was a number. Code that relied on the terminating byte being swallowed will observe it again.
• Anyone who worked around the bug by inserting whitespace separators is unaffected — whitespace is still skipped on the next extraction.
• parse(), accept(), and all non-stream inputs (strings, iterators, containers, FILE*) are unchanged.

Follow-up

docs/mkdocs/docs/api/operator_gtgt.md and sax_parse.md say "version 4.0.0"; these need updating if the change lands elsewhere. The admonition added in #5343 is replaced here by the version note.

---

• The changes are described in detail, both the what and why.
• An existing issue is referenced.
• All new code is covered by tests (new section in unit-deserialization.cpp; verified to fail without the fix).
• The documentation is updated.
• make amalgamate was run; single_include matches include (+132/−5, no unrelated reformatting).

---

This pull request was written by Claude Code.

[gregmarr]

When sungetc() fails
sungetc() can fail if the streambuf has no putback position. Then the character stays consumed — which is exactly today's behavior, so a best-effort unget is never worse than the status quo. restore_pending_unget() reports this rather than asserting. Covered by a test using a streambuf whose pbackfail always fails.

Have you looked at using peek() to get the next character and then advancing the stream if appropriate, rather than getting the next character and having to do unget? I haven't looked to see if it's feasible, just curious if you already had done this.