gh-148991: Add colour to tokenize CLI output

[hugovk]

For example, given this script which has a bit of everything:

Details

# A comment

x = 42
y = 3.14
z = 2 + 3j
name = "hello"
raw = b"bytes"
f_str = f"value: {x}"
t_str = t"template: {x}"

match x:
    case 1:
        pass

for i in range(10):
    if i % 2 == 0:
        print(i)

def greet(name: str = "world") -> None:
    ...

class Foo:
    @staticmethod
    def bar(a, b=True, *args, **kwargs):
        return a + b

lambda x: x * 2

result = [i**2 for i in range(5)]
mapping = {"key": "value", 1: None}
s = {1, 2, 3}
t = (1,)

try:
    1 / 0
except ZeroDivisionError as e:
    raise ValueError("oops") from e

async def main():
    await something()
    async for item in aiter():
        yield item

a = x if x > 0 else -x
b = x | y & z ^ ~x
c = x << 1 >> 2
d: int
e += 1  # type: ignore

• Issue: Add colour to tokenize CLI output #148991

---

📚 Documentation preview 📚: https://cpython-previews--148992.org.readthedocs.build/