terminal: change cell width when wider grapheme detected

[jacobsandlund]

This PR updates the logic in Terminal print to include more cases of changing a cell to be wide due to a grapheme cluster that needs to be wide but starts off narrow. The existing case of this is a text-presentation code point followed by VS16 to make it emoji presentation. This PR handles more cases that are found in scripts such as Devanagari where the correct grapheme width calculation sums up multiple code points of non-zero widths. An example, as seen from uucode's issue #1 is क्‍ष, which now with #9680 merged is one grapheme cluster instead of two, but the U+0915 (first code point) is width one and U+0937 (final code point) is also width one, and the whole cluster should be width 1 + 1 = 2. This is important to address with the grapheme break change otherwise these scripts would show with narrow cells, incorrectly.

Before:

After:

---

Note that the logic here just takes width_zero_in_grapheme and if it's not zero width, makes the cell wide. This is actually wrong for graphemes with prepend (usually/always? zero width) followed by a character that should be narrow width, but that's affecting a much smaller number of graphemes. To address that, we would need to run the full wcwidth from uucode on the grapheme, and compare the width output with the current cell's Wide. I figured it'd be better to incrementally just handle the bulk of the cases with the width_zero_in_grapheme check.

This also adds tests to make sure moving the cell is handled correctly, which was not the case for the existing VS16 logic.

There's a lot of code here to handle transferring the graphemes when the narrow cell should wrap to the next line to become wide. I'd like feedback on the approach here before attempting to clean anything up, if desired (pull it out into a separate method?).

AI was used in some of the uucode changes in #9678 (Amp--primarily for tests), but everything was carefully vetted and much of it done by hand. This PR was made without AI.

[mitchellh]

The unit tests are good, but we're going to definitely have to test throughput of this one since print is the number one hot path. I didn't look closely at the implementation yet to have concrete suggestions (or, to see if it's just perfect).

I think testing cat speeds of large files, vet-bench, and and the terminal-stream Ghostty benchmark (which handles print) will do great. Even just one to start would help.

[jacobsandlund]

sounds good i'll look at running these

[jacobsandlund]

vtebench shows the same results:

running terminal-stream on 200 MB of random utf8 shows no difference:

also the same on 100 MB of Thai wikipedia dump:

[jquast]

I've been integrating with this branch it is looking good and agreement with interpretation of unicode standards and my next update with python wcwidth.

The margin wrap stuff is something else, it does get complex, only Kovid has written a specification and it makes clear about "VS-16 (and similar) can wrap, but VS-15 does not unwrap" and the like.

[mitchellh]

Thanks for the follow ups. Due to what this is touching I really need to do a deep line by line review on this and haven't had the chance yet.

[jacobsandlund]

Just compared ucs-detect output (with latest ucs-detect from #10965) on this branch vs main and here's what I see:

main

Language score 74.6%

grapheme-width-changes

Language score 95.8%

Drop from 100% on unicode 16 (this branch)

Prior to the upgrade to unicode 17, this branch was scoring 100%. With Unicode 17, more code points got added to Indic_Conjunct_Break=Linker, and this made for wider graphemes (width 3 and 4) that Ghostty currently doesn't support. https://ampcode.com/threads/T-019c8b04-f521-7703-b99f-dc99f7b8e302

[jquast]

Just to chime in that I researched more today, the source of foot and windows terminal, they also cap graphemes to a width of 2.

I don't think this is technically correct, but a common limitation of terminals not to support cells that are 3, 4 or 5 cells wide. kitty does terrible with these, making them narrow even, but recovers excellently when using text sizing protocol to "fill out" the desired width (3,4,5) measured by python wcwidth for complex script like javanaese.

Previously I allowed complex graphemes to grow up to width 5 in python wcwidth, e.g. https://github.com/jquast/ucs-detect/blob/e6532f0dd7d93dcb8fcfe9e0bc582ed915875c5a/docs/ucs_example_files/ucs_graphemes_5.txt depicted above

And although that may better compliment the font engine's result, like in ghostty this sequence seems to "spill out" and over to 4 cells even though it is capped at 2 for cursor advance:

But anyway these 3 terminals (ghostty, foot, terminal.exe) are the only that appear to support complex graphemes, they all cap to 2 cells, and so the next release of wcwidth + spec will have a correction to match this and change ucs-detect accordingly.

Screenshot from foot, also allows only 2 for "cursor advance" but spills out the same as ghostty,