Skip to content

Support quoted literal search terms for CLI flags (e.g. --dry-run) (#14041) - #14665

Open
chishxd wants to merge 2 commits into
sphinx-doc:masterfrom
chishxd:feat/quoted-search
Open

Support quoted literal search terms for CLI flags (e.g. --dry-run) (#14041)#14665
chishxd wants to merge 2 commits into
sphinx-doc:masterfrom
chishxd:feat/quoted-search

Conversation

@chishxd

@chishxd chishxd commented Sep 1, 2026

Copy link
Copy Markdown

Purpose

Fixes search for CLI-flag-style terms (e.g. --dry-run, -v) which are currently
unsearchable due to two separate issues in the search pipeline:

  1. Indexer : _word_re in sphinx/search/__init__.py only matched \w+,
    so hyphens were treated as word separators at index-build time. A page mentioning
    --dry-run would index the term as dry and run — the flag itself was never
    stored as a searchable token.

  2. Query parser : searchtools.js's splitQuery/_parseQuery treats any
    term beginning with - as an exclusion operator. Even if the index preserved the
    token, searching --dry-run (or -v) would silently exclude results rather than
    search for them.

The simple fix:

  • _word_re is updated to (?<!\w)-{1,2}\w[\w-]*|\w+ which preserves leading
    single/double hyphens as part of a token only at a word boundary

  • splitQuery now extracts "quoted phrases" from the query string before running
    the normal tokenizer on the remainder, returning {quotedTerms, plainTerms}
    instead of a flat array. _parseQuery adds quoted terms directly to searchTerms,
    bypassing the - exclusion check entirely

Scope / known limitations:

  • This does not change behavior for unquoted CLI-flag searches... typing --dry-run
    without quotes still goes through the normal splitter/exclusion path. Quoting is
    the documented workaround, matching the direction discussed in the linked issue.
  • SearchLanguage.js_splitter_code, the existing hook that lets custom language
    search implementations override splitQuery, is not yet updated to the new
    {quotedTerms, plainTerms} return shape. A custom override returning a flat array
    would silently break with this change. I haven't touched other-language search
    implementations, flagging this for maintainer input on whether it needs handling
    in this PR or a follow-up.

Tests added:

  • Python: unit tests for _word_re covering CLI flags, short flags, and the
    compound-word regression guard.
  • JS: unit tests for splitQuery (quoted extraction, unchanged plain-term behavior)
    and _parseQuery (quoted terms bypass exclusion).
  • Integration: a sphinx-build fixture asserting --dry-run appears as an indexed
    term in the built searchindex.js, and that a quoted search for it returns results.

CHANGES.rst entry added.

References

AI Disclosure

AI tools were used during preparation of this PR:

  • I used Claude for design discussion and planning.. mostly like a rubber duck, working through the
    distinction between the indexer-side and query-parser-side problems, reviewing the
    regex design for edge cases (compound words, word-boundary detection), and
    reviewing/debugging the JS diff stuff
  • Also used GLM with opencode for test suite...

All code in this PR was reviewed and understood by me before submission, and all
interaction with maintainers/reviewers on this PR will be done by me directly, per
the project's AI policy.

…ling

- Enhance word regex pattern in sphinx.search to better handle hyphenated words
- Improve search query parsing to support quoted terms separately
- Refactor searchtools.js with better organized Search object structure
- Enhance search result processing and term handling
- Update _word_re regex to preserve CLI flags like --dry-run and -v as single tokens
- Modify splitQuery to return {quotedTerms, plainTerms} for better term handling
- Update _parseQuery to process quoted terms separately, bypassing exclusion logic
- Add Python tests for CLI flag tokenization and regression guards
- Add JavaScript tests for quoted term extraction and parsing behavior
- Add integration test verifying CLI flags are properly indexed
- Update CHANGES.rst with entry for sphinx-doc#14041

Fixes sphinx-doc#14041
@chishxd chishxd changed the title Feat/quoted search Support quoted literal search terms for CLI flags (e.g. --dry-run) (#14041) Sep 1, 2026
@chishxd

chishxd commented Sep 2, 2026

Copy link
Copy Markdown
Author

woahh that's some crazy amount of Red crosses... But i also tested running test on clean master branch, and got these same errors, the Pygments update seems to have broken test assertions... And Maybe I messed up some linting error, for which I will push some update soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant