fix: handle wrapped Gmail quote headers across all languages - #31
Open
jimmy927 wants to merge 2 commits into
Open
fix: handle wrapped Gmail quote headers across all languages#31jimmy927 wants to merge 2 commits into
jimmy927 wants to merge 2 commits into
Conversation
Gmail wraps long quote headers onto two lines, leaving the wrote-keyword
alone with an optional sender + colon — same shape across locales
("On ...\nwrote:", "Den ...\nskrev:", "Am ...\nschrieb:" etc.). The
per-language `wrote_header` patterns are line-anchored and require
content between keyword and ":", so the wrap was missed in several
languages (SV, DE, FR, ES). Other languages (EN, NL, IT) only worked
because their patterns happened to use \s? where the wrap newline fits.
This is a Gmail-layout quirk, not a language one. Add two per-language
tokens — `gmail_wrap_start` (the date-line prefix, e.g. "On"/"Den"/
"Am") and `gmail_wrap_keyword` (the verb on the wrap line, e.g.
"wrote"/"skrev"/"schrieb") — alongside the existing `wrote_header` /
`from_header`. `EmailReplyParser.HEADER_REGEX` assembles a single
cross-language wrap matcher from the tokens of the *configured*
languages, preserving the per-language opt-in. A language that doesn't
set the tokens (e.g. ja, whose Gmail format is fundamentally different)
simply doesn't contribute.
Adding a new language now needs only two extra lines on its dict; no
edits to global constants or the parser.
The SV language pattern keeps a small relaxation for the non-wrap
"Den ... skrev:" case (sender before keyword, no content between skrev
and ":") — that's distinct from the wrap issue and was the original
trigger for this PR.
All eight wrap-test fixtures are fully synthetic — no real names,
emails, or content. 43/43 tests pass; no regressions in any language.
Gmail wraps long quote headers at ~76 chars. The existing cross-language matcher handles wrap-BEFORE-keyword (the verb falls onto its own line). Verb-in-middle languages (sv/da/de) also see a second shape where the wrap lands inside the recipient's <...> bracket, putting the trailing colon alone on the next line — e.g. Den ... skrev Anna Andersson < anna@example.com>: Add a sibling regex built from the same gmail_wrap_start/gmail_wrap_keyword tokens for this second shape, plus sv/de fixtures + tests covering it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gmail wraps long quote headers onto two lines, leaving the wrote-keyword alone with an optional sender + colon — same shape across locales ("On ...\nwrote:", "Den ...\nskrev:", "Am ...\nschrieb:" etc.). The per-language
wrote_headerpatterns are line-anchored and require content between keyword and ":", so the wrap was missed in several languages (SV, DE, FR, ES). Other languages (EN, NL, IT) only worked because their patterns happened to use \s? where the wrap newline fits.This is a Gmail-layout quirk, not a language one. Add two per-language tokens —
gmail_wrap_start(the date-line prefix, e.g. "On"/"Den"/ "Am") andgmail_wrap_keyword(the verb on the wrap line, e.g. "wrote"/"skrev"/"schrieb") — alongside the existingwrote_header/from_header.EmailReplyParser.HEADER_REGEXassembles a single cross-language wrap matcher from the tokens of the configured languages, preserving the per-language opt-in. A language that doesn't set the tokens (e.g. ja, whose Gmail format is fundamentally different) simply doesn't contribute.Adding a new language now needs only two extra lines on its dict; no edits to global constants or the parser.
The SV language pattern keeps a small relaxation for the non-wrap "Den ... skrev:" case (sender before keyword, no content between skrev and ":") — that's distinct from the wrap issue and was the original trigger for this PR.
All eight wrap-test fixtures are fully synthetic — no real names, emails, or content. 43/43 tests pass; no regressions in any language.