Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .pylintrc
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ disable=
too-many-boolean-expressions,
redundant-u-string-prefix,
consider-using-f-string,
consider-using-max-builtin,
consider-using-min-builtin,

[FORMAT]
max-line-length: 100
Expand Down
8 changes: 6 additions & 2 deletions docs/intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -539,10 +539,12 @@ History
=======

0.8.0 *(unreleased)*
* **Improved** memory usage and import time for Python 3.15 using lazy imports `PR #221`_.
* **Improved** performance on Python 3.15 using standard library iter_graphemes() `PR #206`_.
* **Improved** memory usage and import time for Python 3.15 using lazy imports `PR #221`_.
* **Bugfix** Invisible_Stacker viramas now form conjuncts (Burmese, Khmer, etc.) and
change some Virama width calculations to match `jacobsandlund/uucode`_ (ghostty) `PR #XXX`_.
change some Virama width calculations to match `jacobsandlund/uucode`_ (ghostty) `PR #223`_.
* **Updated** graphemes width maximum now 2, matching Ghostty, foot, and Windows Terminal `PR
#224`_.

0.7.0 *2026-05-02*
* **New** support for `kitty text sizing protocol`_ (OSC 66) in `width()`_ and `clip()`_.
Expand Down Expand Up @@ -776,6 +778,8 @@ https://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c::
.. _`PR #204`: https://github.com/jquast/wcwidth/pull/204
.. _`PR #206`: https://github.com/jquast/wcwidth/pull/206
.. _`PR #221`: https://github.com/jquast/wcwidth/pull/221
.. _`PR #223`: https://github.com/jquast/wcwidth/pull/223
.. _`PR #224`: https://github.com/jquast/wcwidth/pull/224
.. _`Issue #101`: https://github.com/jquast/wcwidth/issues/101
.. _`Issue #155`: https://github.com/jquast/wcwidth/issues/155
.. _`Issue #190`: https://github.com/jquast/wcwidth/issues/190
Expand Down
19 changes: 8 additions & 11 deletions docs/specs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,12 +105,14 @@ it directly follows, making the pair width 2. Wide characters are unchanged.

Any character of non-zero width followed by an ``Mc`` (`Spacing Combining Mark`_)
character when measured in sequence by :func:`wcwidth.wcswidth` or
:func:`wcwidth.width`. The ``Mc`` character adds +1 to the total width,
:func:`wcwidth.width`. The ``Mc`` character adds +1 to the cluster width,
reflecting its *positive advance width* as defined in `General Category`_
(Table 4-4). Zero-width combining marks (``Mn``) between the base character
and the ``Mc`` do not break the association. For example, a consonant followed
by a Nukta (``Mn``) and then a vowel sign (``Mc``) is measured as base + 1.

Any grapheme cluster width is limisted to 2 cells since 0.8.0, `PR #224`_.

Virama Conjunct Formation
-------------------------

Expand All @@ -121,16 +123,10 @@ as a Pure_Killer or Invisible_Stacker depending on context") and
or consonant stacking", the "only as consonant stackers" category
described in the Virama section header).

- A ``Consonant`` immediately following a ``Virama`` contributes 0 width.
- The conjunct still occupies cells and the next visible advance settles it:

- A following ``Mc`` (`Spacing Combining Mark`_, e.g. a vowel sign) counts as
1 cell and closes the conjunct.
- A following character with positive width (or end of string) adds 1 cell
for the conjunct before counting its own width.

- Chains work the same way: C + virama + C + virama + C collapses each
virama+consonant pair.
- A ``Virama`` contributes 0 width (category ``Mn``).
- A ``Consonant`` immediately following a ``Virama`` adds its width to the
current grapheme cluster.
- The cluster total is capped at 2 cells since 0.8.0, `PR #224`_.
- ``Mn`` marks do not break conjunct context within the same `aksara`_.
- ZWJ (`U+200D`_) after a virama is consumed without breaking conjunct state,
supporting explicit half-form requests (virama + ZWJ + consonant).
Expand Down Expand Up @@ -187,3 +183,4 @@ See also: `L2/2023/23107`_ "Proper Complex Script Support in Text Terminals".
.. _`L2/2023/23107`: https://www.unicode.org/L2/L2023/23107-terminal-suppt.pdf
.. _`Unicode Standard Annex #29`: https://www.unicode.org/reports/tr29/
.. _`uncodedata.iter_graphemes()`: https://docs.python.org/3.15/library/unicodedata.html#unicodedata.iter_graphemes
.. _`PR #224`: https://github.com/jquast/wcwidth/pull/224
28 changes: 14 additions & 14 deletions tests/test_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -308,8 +308,8 @@ def test_devanagari_script():
"\u093F") # MatraL, Category 'Mc', East Asian Width property 'N' -- DEVANAGARI VOWEL SIGN I
# 23107-terminal-suppt.pdf suggests wcwidth.wcwidth should return (2, 0, 0, 1)
expect_length_each = (1, 0, 1, 0)
# virama conjunct collapses KA+virama+SSA into one cell, Mc adds +1
expect_length_phrase = 3
# grapheme cluster capped at 2 cells (ghostty, foot, Windows Terminal)
expect_length_phrase = 2

# exercise,
length_each = tuple(map(wcwidth.wcwidth, phrase))
Expand All @@ -330,8 +330,8 @@ def test_tamil_script():
# 23107-terminal-suppt.pdf suggests wcwidth.wcwidth should return (3, 0, 0, 4)
expect_length_each = (1, 0, 1, 0)

# virama conjunct collapses KA+virama+SSA into one cell, Mc adds +1
expect_length_phrase = 3
# grapheme cluster capped at 2 cells (ghostty, foot, Windows Terminal)
expect_length_phrase = 2

# exercise,
length_each = tuple(map(wcwidth.wcwidth, phrase))
Expand All @@ -353,8 +353,8 @@ def test_kannada_script():
"\u0cc8") # MatraUR, Category 'Mc', East Asian Width property 'N' -- KANNADA VOWEL SIGN AI
# 23107-terminal-suppt.pdf suggests should be (2, 0, 3, 1)
expect_length_each = (1, 0, 1, 0)
# virama conjunct collapses RA+virama+JHA into one cell, Mc adds +1
expect_length_phrase = 3
# grapheme cluster capped at 2 cells (ghostty, foot, Windows Terminal)
expect_length_phrase = 2

# exercise,
length_each = tuple(map(wcwidth.wcwidth, phrase))
Expand Down Expand Up @@ -437,16 +437,16 @@ def test_mc_width_consistency(repeat):


@pytest.mark.parametrize("phrase,expected", [
("\u0999\u09CD\u0997\u09C7", 3),
("\u0915\u094D\u0924\u093F", 3),
("\u0915\u094D\u0930\u093F", 3),
("\u0A95\u0ACD\u0A95\u0ACB", 3),
("\u0938\u094D\u0924\u094D\u0930", 3),
("\u0999\u09CD\u0997\u09C7", 2),
("\u0915\u094D\u0924\u093F", 2),
("\u0915\u094D\u0930\u093F", 2),
("\u0A95\u0ACD\u0A95\u0ACB", 2),
("\u0938\u094D\u0924\u094D\u0930", 2),
("\u0938\u094D\u0924", 2),
("\u0915\u094D\u0020", 2),
("\u09A4\u09CD\u200D\u09AA", 2),
("\u0915\u094D\u200D\u0924", 2),
("\u0D15\u0D4D\u0D15\u0D41\u0D02", 3),
("\u0D15\u0D4D\u0D15\u0D41\u0D02", 2),
("\u0915\u094D\u0924\u0941\u0902", 2),
])
def test_virama_conjunct(phrase, expected):
Expand All @@ -456,9 +456,9 @@ def test_virama_conjunct(phrase, expected):

@pytest.mark.parametrize("phrase,expected", [
("\u1000\u1039\u1000", 2), # Burmese KA+VIRAMA+KA
("\u1000\u1039\u1000\u1039\u1002", 3), # Burmese KA+V+KA+V+GA
("\u1000\u1039\u1000\u1039\u1002", 2), # Burmese KA+V+KA+V+GA (capped)
("\u1000\u1039\u200D\u1000", 2), # Burmese KA+V+ZWJ+KA
("\u1782\u17D2\u1782\u17C1", 3), # Khmer KO+COENG+KO+VOWEL_E (Mc closes)
("\u1782\u17D2\u1782\u17C1", 2), # Khmer KO+COENG+KO+VOWEL_E (capped)
("\u1780\u17D2\u1780", 2), # Khmer KA+COENG+KA
])
def test_virama_conjunct_invisible_stacker(phrase, expected):
Expand Down
1 change: 1 addition & 0 deletions tests/test_text_sizing.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ def test_text_sizing_sequence(given_sequence, expected_text, expected_params, ex
('\x1b]66;w=2;A\x07\x1b]66;w=3;B\x07', 5),
('\x1b]66;s=2:w=3;text\x1b\\', 6),
('\x1b[31m\x1b]66;w=2;AB\x07\x1b[0m', 2),
('\x1b]66;;\x01\x07', 0),
])
def test_strings_with_text_sizing(text, expected):
"""Verify measured width strings containing OSC66."""
Expand Down
6 changes: 6 additions & 0 deletions tests/test_width.py
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,12 @@ def test_width_escape_sequences(text, expected, name):
('\x1b', 0, 'lone_ESC'),
('\x1b!', 1, 'ESC_unrecognized'),
('*\x1b*', 2, 'lone_ESC_between_text'),
('\tA\u0CBE\rB\u0CBE\b', 10, 'backspace_flush_below_max_extent'),
('\tA\u0CBE\rB\u0CBE\x1b', 10, 'esc_flush_below_max_extent'),
('\tA\u0CBE\rB\u0CBE\x00', 10, 'zero_width_flush_below_max_extent'),
('\tA\u0CBE\rB\u0CBE\x01', 10, 'illegal_ctrl_flush_below_max_extent'),
('\tA\u0CBE\rB\u0CBE\n', 10, 'vertical_ctrl_flush_below_max_extent'),
('\u4e2d\u0bcd\u4e2d\u0bcd\u4e2d\u0bcd\u4e2d', 2, 'glitch_virama_chain_capped'),
]


Expand Down
2 changes: 1 addition & 1 deletion tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ warn_unused_ignores = true
[testenv:codespell]
basepython = python3.14
deps = codespell
commands = codespell --skip="*.pyc,htmlcov,_build,build,*.egg-info,.tox,data,./tests/*.txt,*.csv,*.ods,table_*.py,docs/specs.rst,*.isorted" \
commands = codespell --skip="*.pyc,htmlcov,_build,build,*.egg-info,.tox,data,./tests/*.txt,*.csv,*.ods,table_*.py,docs/specs.rst,*.isorted,ucs-detect" \
--ignore-words-list="thirdparty,claus,oclock,womens,aprox" \
--uri-ignore-words-list '*' \
--summary --count
Expand Down
4 changes: 3 additions & 1 deletion wcwidth/_clip.py
Original file line number Diff line number Diff line change
Expand Up @@ -605,7 +605,9 @@ def _write_cells(s: str, w: int, write_col: int,
f"{n_backward} cells left from column {col}, "
f"exceeding string start"
)
col = max(0, col - n_backward)
col -= n_backward
if col < 0:
col = 0
idx = m.end()
continue

Expand Down
27 changes: 18 additions & 9 deletions wcwidth/_wcswidth.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,13 @@ def wcswidth(
total_width = 0
idx = 0

# grapheme-clustering state
# grapheme-clustering state and local re-binding for performance
last_measured_idx = -2
last_measured_ucs = -1
prev_was_virama = False
cluster_width = 0
vs16_nw_table = VS16_NARROW_TO_WIDE['9.0.0']
_bisearch = bisearch

while idx < end:
char = pwcs[idx]
Expand All @@ -96,10 +99,8 @@ def wcswidth(

# 6. VS16 (U+FE0F): converts preceding narrow character to wide.
if ucs == 0xFE0F and last_measured_idx >= 0:
total_width += bisearch(
ord(pwcs[last_measured_idx]),
VS16_NARROW_TO_WIDE['9.0.0'],
)
if _bisearch(last_measured_ucs, vs16_nw_table):
cluster_width = 2
last_measured_idx = -2 # prevent double application
idx += 1
continue
Expand Down Expand Up @@ -127,17 +128,25 @@ def wcswidth(
# C0/C1 control character
return -1
if w > 0:
total_width += w
# virama+consonant extends current cluster; otherwise start new
if prev_was_virama:
cluster_width = 2
else:
if cluster_width:
total_width += cluster_width
cluster_width = w
last_measured_idx = idx
last_measured_ucs = ucs
prev_was_virama = False
elif last_measured_idx >= 0 and bisearch(ucs, _CATEGORY_MC_TABLE):
# Spacing Combining Mark (Mc) following a base character adds 1
total_width += 1
elif last_measured_idx >= 0 and _bisearch(ucs, _CATEGORY_MC_TABLE):
# Spacing Combining Mark (Mc) following a base character
cluster_width = 2
last_measured_idx = -2
prev_was_virama = False
else:
prev_was_virama = ucs in _ISC_VIRAMA_SET
idx += 1

if cluster_width:
total_width += cluster_width
return total_width
68 changes: 55 additions & 13 deletions wcwidth/_width.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,16 +166,25 @@ def width(
# - ambiguous_width=2: full positional args needed (results differ, separate cache is correct)
_wcwidth = wcwidth if ambiguous_width == 1 else lambda c: wcwidth(c, 'auto', ambiguous_width)

# grapheme-clustering state
# grapheme-clustering state and local re-bindings for performance
last_measured_idx = -2
last_measured_ucs = -1
prev_was_virama = False
cluster_width = 0
vs16_nw_table = VS16_NARROW_TO_WIDE['9.0.0']
_bisearch = bisearch

while idx < text_len:
char = text[idx]

# 1. ESC sequences
if char == '\x1b':
# Flush pending cluster before processing escape sequence
if cluster_width:
current_col += cluster_width
if current_col > max_extent:
max_extent = current_col
cluster_width = 0
m = _SEQUENCE_CLASSIFY.match(text, idx)
if not m:
# 1a. Errant ESC or unknown sequence: only the first character is zero-width
Expand Down Expand Up @@ -206,7 +215,9 @@ def width(
f"{n_backward} cells left from column {current_col}, "
f"exceeding string start"
)
current_col = max(0, current_col - n_backward)
current_col -= n_backward
if current_col < 0:
current_col = 0
# 2d. OSC 66 Text Sizing — has positive display width
elif (ts_meta := m.group('ts_meta')) is not None:
ts_text = m.group('ts_text')
Expand All @@ -221,13 +232,19 @@ def width(
# Escape sequences break VS16 adjacency: reset last-measured state
last_measured_idx = -2
last_measured_ucs = -1
max_extent = max(max_extent, current_col)
if current_col > max_extent:
max_extent = current_col
continue

# 2. Vertical or Illegal control characters zero width or error when 'strict'
if char in ILLEGAL_CTRL:
if strict:
raise ValueError(f"Illegal control character {ord(char):#x} at position {idx}")
if cluster_width:
current_col += cluster_width
if current_col > max_extent:
max_extent = current_col
cluster_width = 0
idx += 1
last_measured_idx = -2
last_measured_ucs = -1
Expand All @@ -236,13 +253,23 @@ def width(
if char in VERTICAL_CTRL:
if strict:
raise ValueError(f"Vertical movement character {ord(char):#x} at position {idx}")
if cluster_width:
current_col += cluster_width
if current_col > max_extent:
max_extent = current_col
cluster_width = 0
idx += 1
last_measured_idx = -2
last_measured_ucs = -1
continue

# 3. Horizontal movement characters
if char in HORIZONTAL_CTRL:
if cluster_width:
current_col += cluster_width
if current_col > max_extent:
max_extent = current_col
cluster_width = 0
if char == '\t' and tabsize > 0:
current_col += tabsize - (current_col % tabsize)
elif char == '\b':
Expand All @@ -255,14 +282,20 @@ def width(
"indeterminate starting column"
)
current_col = 0
max_extent = max(max_extent, current_col)
if current_col > max_extent:
max_extent = current_col
idx += 1
last_measured_idx = -2
last_measured_ucs = -1
continue

# 4. Zero-width control characters
if char in ZERO_WIDTH_CTRL:
if cluster_width:
current_col += cluster_width
if current_col > max_extent:
max_extent = current_col
cluster_width = 0
idx += 1
last_measured_idx = -2
last_measured_ucs = -1
Expand All @@ -286,9 +319,8 @@ def width(

# 6. VS16 (U+FE0F): converts preceding narrow character to wide.
if ucs == 0xFE0F and last_measured_idx >= 0:
if bisearch(ord(text[last_measured_idx]), VS16_NARROW_TO_WIDE['9.0.0']):
current_col += 1
max_extent = max(max_extent, current_col)
if _bisearch(last_measured_ucs, vs16_nw_table):
cluster_width = 2
last_measured_idx = -2 # prevent double application
idx += 1
continue
Expand All @@ -313,19 +345,29 @@ def width(
# 8. Normal character: measure with wcwidth
w = _wcwidth(char)
if w > 0:
current_col += w
max_extent = max(max_extent, current_col)
# virama+consonant extends current cluster; otherwise start new
if prev_was_virama:
cluster_width = 2
else:
if cluster_width:
current_col += cluster_width
if current_col > max_extent:
max_extent = current_col
cluster_width = w
last_measured_idx = idx
last_measured_ucs = ucs
prev_was_virama = False
elif last_measured_idx >= 0 and bisearch(ucs, _CATEGORY_MC_TABLE):
# Spacing Combining Mark (Mc) following a base character adds 1
current_col += 1
max_extent = max(max_extent, current_col)
elif last_measured_idx >= 0 and _bisearch(ucs, _CATEGORY_MC_TABLE):
# Spacing Combining Mark (Mc) following a base character
cluster_width = 2
last_measured_idx = -2
prev_was_virama = False
else:
prev_was_virama = ucs in _ISC_VIRAMA_SET
idx += 1

if cluster_width:
current_col += cluster_width
if current_col > max_extent:
max_extent = current_col
return max_extent
Loading
Loading