Skip to content

[ty] Introduce shared primitives for parsing backticks in docstrings - #26928

Merged
lerebear merged 2 commits into
mainfrom
lerebear/push-krurnqpmzmum
Jul 24, 2026
Merged

[ty] Introduce shared primitives for parsing backticks in docstrings#26928
lerebear merged 2 commits into
mainfrom
lerebear/push-krurnqpmzmum

Conversation

@lerebear

@lerebear lerebear commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

This refactors the code span parsing we do in docstrings around a cursor-based scanner and explicit ranged backtick runs and spans. Ultimately, this serves two purposes:

  1. To remove duplicated offset arithmetic from inline-link and trailing-parenthetical parsing.
  2. To expose an iterator over backtick-delimited fragments of text for downstream normalization.

Test plan

See included tests.

@astral-sh-bot astral-sh-bot Bot added server Related to the LSP server ty Multi-file analysis & type inference labels Jul 17, 2026
Base automatically changed from lerebear/push-tuwwquqwmyus to main July 17, 2026 05:14
@lerebear
lerebear force-pushed the lerebear/push-krurnqpmzmum branch 11 times, most recently from 9593c21 to 4436ab1 Compare July 22, 2026 22:40
@lerebear lerebear changed the title [ty] Model docstring backtick spans explicitly [ty] Introduce shared primitives for parsing backticks in docstrings Jul 23, 2026
@lerebear
lerebear requested a review from MichaReiser July 23, 2026 03:54
@lerebear
lerebear force-pushed the lerebear/push-krurnqpmzmum branch from 4436ab1 to cd9fb36 Compare July 23, 2026 03:57
@lerebear
lerebear marked this pull request as ready for review July 23, 2026 05:10
@lerebear
lerebear requested a review from a team as a code owner July 23, 2026 05:10

@MichaReiser MichaReiser left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

Codex claims that the backtick lexer is overly strict and doesn't correctly recover from invalid backticks

`unmatched ``code``

should be

vec![
    ("text", "`unmatched "),
    ("span", "code"),
]

Escaping only escapes the first backtick, not all of them

\``code`

Expected

vec![
    ("text", r"\`"),
    ("span", "code"),
]

Comment thread crates/ty_ide/src/docstring/document/syntax.rs Outdated
Comment thread crates/ty_ide/src/docstring/document/syntax.rs Outdated
Comment thread crates/ty_ide/src/docstring/document/syntax.rs Outdated
Comment thread crates/ty_ide/src/docstring/document/syntax.rs
Comment thread crates/ty_ide/src/docstring/document/syntax.rs Outdated
Comment thread crates/ty_ide/src/docstring/document/syntax.rs Outdated
@lerebear
lerebear force-pushed the lerebear/push-krurnqpmzmum branch 2 times, most recently from 7a81c3f to b78da19 Compare July 23, 2026 22:23
@lerebear
lerebear force-pushed the lerebear/push-krurnqpmzmum branch from b78da19 to 05178ee Compare July 23, 2026 22:48
@lerebear

Copy link
Copy Markdown
Contributor Author

Codex claims that the backtick lexer is overly strict and doesn't correctly recover from invalid backticks
Escaping only escapes the first backtick, not all of them

Those findings are both accurate, but I don't think they're worth fixing at the moment:

  1. In my brief explorations, they seem to require a significantly more complex design.
  2. Those patterns appear to occur in real docstrings infrequently.
  3. When they do occur, we generally just risk displaying a few extra backticks literally, which is incorrect but harmless.

Of course we can readjust in the future if we receive complaints.

@lerebear
lerebear merged commit 77b9cc5 into main Jul 24, 2026
54 checks passed
@lerebear
lerebear deleted the lerebear/push-krurnqpmzmum branch July 24, 2026 03:12
@lerebear lerebear added the internal An internal refactor or improvement label Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal An internal refactor or improvement server Related to the LSP server ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants