Add line/column to tokenizer errors

[kaidokert]

Tracks the line and column numbers in tokenizer, and propagates error source info properly. Note, size_t is used and can overflow in extreme contexts.

Fixes #68

Summary by Sourcery

Track line and column positions in the JSON tokenizer and surface them in error reporting while keeping existing error semantics.

New Features:

• Include line and column metadata on tokenizer errors and expose them via Display/Debug output for easier location of parse failures.

Enhancements:

• Track line and column counters within the tokenizer loop using saturating arithmetic to avoid panics on extreme input sizes.
• Adjust tokenizer control flow and helpers to propagate positional context (line/column) whenever constructing errors.
• Provide a custom PartialEq implementation for tokenizer errors that preserves existing equality behavior while allowing additional metadata fields.

Tests:

• Update existing tokenizer and conformance tests to accommodate the new error signature while preserving prior expectations.
• Add unit and API-level tests to verify correct line/column tracking across various multiline, indented, and edge-case inputs and ensure error messages include this information.

Summary by CodeRabbit

• Improvements

  • Error reporting now includes byte offset plus line and column, yielding richer, line/column-aware parse diagnostics for multiline inputs and edge cases (trailing commas, invalid tokens, unexpected content).
  • Tokenization and parsing consistently propagate location info so error messages precisely identify where issues occur.

• Tests

  • Added tests validating line/column propagation and multiline error scenarios to ensure accurate, actionable diagnostics.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

The tokenizer now records absolute byte position plus line and column in a new public Position type and threads it through Tokenizer, ParseContext, and Error. Many error kinds were added/adjusted and tests updated to assert line/column-aware error reporting.

Changes

Cohort / File(s)	Summary
Tokenizer core & Error picojson/src/ujson/tokenizer/mod.rs	Add public Position { pos, line, column }; Tokenizer gains position and drops total_consumed; Error now stores Position; Error::new signature changed; Display/Debug include line/column; ErrKind expanded with many new variants.
ParseContext & container logic picojson/src/ujson/tokenizer/mod.rs (ParseContext methods)	after_comma changed to Option<(u8, Position)>; enter_object/exit_object/enter_array/exit_array and related methods accept and propagate Position; container transition/error sites updated.
Parsing flow & tokenization picojson/src/ujson/tokenizer/mod.rs (various functions)	Tokenization and parse loop update Position on bytes/newlines; error creation sites now pass Position; public parsing entrypoints (parse_full, parse_chunk, finish, start_token, etc.) propagate Position.
Public API & types picojson/src/ujson/tokenizer/mod.rs (exports)	Position is public; signatures for Tokenizer, ParseContext, and Error updated to be position-aware; public methods adjusted accordingly.
Tests picojson/tests/api_test_errors.rs	Added test_error_includes_line_and_column_info and test_multiline_error_tracking to validate line/column propagation and error messages.

Sequence Diagram(s)

sequenceDiagram
    participant Client as Client
    participant Tokenizer as Tokenizer
    participant ParseContext as ParseContext
    participant ErrorFactory as Error

    Client->>Tokenizer: parse_full / feed bytes
    Tokenizer->>Tokenizer: advance bytes, update Position (pos,line,column)
    Tokenizer->>ParseContext: enter/exit/check (with Position)
    ParseContext-->>Tokenizer: Ok or Err(kind, Position)
    Tokenizer->>ErrorFactory: Error::new(kind, char?, Position)
    ErrorFactory-->>Client: return ParseError/Error containing Position

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• Panic notes #21: Modifies trailing-comma handling and how after_comma records byte+position (closely related changes).
• Error handling #48: Changes Error representation and parser error propagation (overlaps with Error/ParseError edits here).

Poem

🐰 I hop through bytes and count each line,

my whiskers mark the column and the sign,
When parsing stumbles, I point the place,
a carrot-shaped clue — pos, line, column traced,
hooray, now errors wear a tiny rabbit face!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 78.08% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Add line/column to tokenizer errors' directly and clearly summarizes the primary change: adding line and column tracking to tokenizer error reporting.
Linked Issues check	✅ Passed	The PR fully addresses issue #68 requirements: Position type with line/column fields added, errors propagate positional metadata, line/column tracked during tokenization, and saturating arithmetic used to handle overflow risk.
Out of Scope Changes check	✅ Passed	All changes are directly related to implementing line/column tracking in tokenizer errors as specified in issue #68; no unrelated modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello @kaidokert, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the diagnostic capabilities of the JSON tokenizer by integrating line and column information into all generated error messages. This enhancement provides a more user-friendly experience when encountering parsing failures, as the exact location of the syntax error within the input document can now be precisely identified. The changes involve updating internal state management, error propagation, and output formatting, all while maintaining backward compatibility for existing test suites.

Highlights

• Enhanced Error Reporting: Tokenizer errors now include precise line and column numbers, in addition to the byte position, making debugging malformed JSON significantly easier.
• Internal State Tracking: The Tokenizer struct now tracks the current line and column during parsing, updating these values for each processed byte and resetting the column on newline characters.
• Error Struct and API Updates: The Error struct has been extended with line and column fields, and its constructor (Error::new) and display/debug formatting have been updated to utilize this new information. Existing internal methods that generate errors now accept these new parameters.
• Test Compatibility: A custom PartialEq implementation was added for the Error struct to ensure that existing tests, which only asserted on error kind, character, and position, continue to pass without modification.
• Comprehensive Testing: New dedicated test cases have been added to thoroughly validate the accuracy of line and column tracking across various scenarios, including single-line, multi-line, indented, and edge-case errors.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[sourcery-ai]

Hey - I've left some high level feedback:

• The repeated threading of line and column through many internal methods (enter_*, exit_*, check_trailing_comma, start_token, etc.) makes the signature noise quite high; consider introducing a small Position { pos, line, column } helper or storing line/column in ParseContext so callers only pass a single value or nothing at all.
• Instead of using line == 0 as a sentinel to decide which Display/Debug format to print, it may be clearer and more type-safe to store Option<(usize, usize)> for line/column and pattern match on that, which also avoids having to pass dummy 0, 0 values in tests.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The repeated threading of `line` and `column` through many internal methods (`enter_*`, `exit_*`, `check_trailing_comma`, `start_token`, etc.) makes the signature noise quite high; consider introducing a small `Position { pos, line, column }` helper or storing line/column in `ParseContext` so callers only pass a single value or nothing at all.
- Instead of using `line == 0` as a sentinel to decide which `Display`/`Debug` format to print, it may be clearer and more type-safe to store `Option<(usize, usize)>` for line/column and pattern match on that, which also avoids having to pass dummy `0, 0` values in tests.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[gemini-code-assist]

Code Review

This pull request is a great enhancement, adding line and column numbers to tokenizer errors for better diagnostics. The implementation is solid, including the use of saturating_add to prevent overflows and adding comprehensive tests for the new functionality. The custom PartialEq for Error is a clever way to avoid breaking existing tests.

I have a couple of suggestions to further improve the accuracy of error reporting and the internal design of the Error struct. Please see my detailed comments.

[kaidokert]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a significant improvement, adding line and column numbers to tokenizer errors to make debugging easier. The implementation is thorough, correctly tracking positions, propagating them to the Error type, and updating display formats. The use of saturating_add for position counters is a robust choice. This PR also implicitly fixes a bug where trailing comma errors reported an incorrect position, further enhancing error diagnostics. The new tests are comprehensive. I've provided a couple of suggestions for minor code cleanup and to make a test more idiomatic.

[kaidokert]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a great enhancement, adding line and column numbers to tokenizer errors, which will significantly improve the debugging experience. The implementation is thorough, including updates to the tokenizer's core logic, error structures, and extensive test coverage for the new position tracking.

I have a few suggestions to further improve robustness:

• Using saturating_add for byte position calculation to prevent overflow issues.
• Revisiting the AVR-specific padding in the Error struct to ensure it fully avoids the problematic size range.
• Considering a more conventional handling of \r and \r\n line endings for more intuitive column reporting.

Overall, this is a well-executed and valuable change.

[kaidokert]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a great enhancement, adding line and column numbers to tokenizer errors, which will significantly improve the debugging experience for parsing issues. The implementation is thorough, covering error propagation, comprehensive new tests, and robust handling of potential overflows using saturating_add. My feedback focuses on a few areas to further improve the code: ensuring a platform-specific workaround is effective and making some of the new tests more robust and idiomatic.