Support quoted literal search terms for CLI flags (e.g. --dry-run) (#14041) - #14665
Open
chishxd wants to merge 2 commits into
Open
Support quoted literal search terms for CLI flags (e.g. --dry-run) (#14041)#14665chishxd wants to merge 2 commits into
chishxd wants to merge 2 commits into
Conversation
…ling - Enhance word regex pattern in sphinx.search to better handle hyphenated words - Improve search query parsing to support quoted terms separately - Refactor searchtools.js with better organized Search object structure - Enhance search result processing and term handling
- Update _word_re regex to preserve CLI flags like --dry-run and -v as single tokens
- Modify splitQuery to return {quotedTerms, plainTerms} for better term handling
- Update _parseQuery to process quoted terms separately, bypassing exclusion logic
- Add Python tests for CLI flag tokenization and regression guards
- Add JavaScript tests for quoted term extraction and parsing behavior
- Add integration test verifying CLI flags are properly indexed
- Update CHANGES.rst with entry for sphinx-doc#14041
Fixes sphinx-doc#14041
Author
|
woahh that's some crazy amount of Red crosses... But i also tested running test on clean master branch, and got these same errors, the Pygments update seems to have broken test assertions... And Maybe I messed up some linting error, for which I will push some update soon |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes search for CLI-flag-style terms (e.g. --dry-run, -v) which are currently
unsearchable due to two separate issues in the search pipeline:
Indexer :
_word_reinsphinx/search/__init__.pyonly matched\w+,so hyphens were treated as word separators at index-build time. A page mentioning
--dry-runwould index the term asdryandrun— the flag itself was neverstored as a searchable token.
Query parser :
searchtools.js'ssplitQuery/_parseQuerytreats anyterm beginning with
-as an exclusion operator. Even if the index preserved thetoken, searching
--dry-run(or-v) would silently exclude results rather thansearch for them.
The simple fix:
_word_reis updated to (?<!\w)-{1,2}\w[\w-]*|\w+ which preserves leadingsingle/double hyphens as part of a token only at a word boundary
splitQuerynow extracts "quoted phrases" from the query string before runningthe normal tokenizer on the remainder, returning {quotedTerms, plainTerms}
instead of a flat array. _parseQuery adds quoted terms directly to searchTerms,
bypassing the - exclusion check entirely
Scope / known limitations:
--dry-runwithout quotes still goes through the normal splitter/exclusion path. Quoting is
the documented workaround, matching the direction discussed in the linked issue.
SearchLanguage.js_splitter_code, the existing hook that lets custom languagesearch implementations override
splitQuery, is not yet updated to the new{quotedTerms, plainTerms}return shape. A custom override returning a flat arraywould silently break with this change. I haven't touched other-language search
implementations, flagging this for maintainer input on whether it needs handling
in this PR or a follow-up.
Tests added:
_word_recovering CLI flags, short flags, and thecompound-word regression guard.
splitQuery(quoted extraction, unchanged plain-term behavior)and
_parseQuery(quoted terms bypass exclusion).sphinx-buildfixture asserting--dry-runappears as an indexedterm in the built
searchindex.js, and that a quoted search for it returns results.CHANGES.rstentry added.References
AI Disclosure
AI tools were used during preparation of this PR:
distinction between the indexer-side and query-parser-side problems, reviewing the
regex design for edge cases (compound words, word-boundary detection), and
reviewing/debugging the JS diff stuff
All code in this PR was reviewed and understood by me before submission, and all
interaction with maintainers/reviewers on this PR will be done by me directly, per
the project's AI policy.