[ty] Avoid treating augmented assignments as attribute definitions - #27633
Conversation
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 96.98%. The percentage of expected errors that received a diagnostic held steady at 93.62%. The number of fully passing files held steady at 106/133. |
Memory usage reportSummary
Significant changesClick to expand detailed breakdownprefect
sphinx
trio
flake8
|
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
unresolved-attribute |
34 | 0 | 1 |
unsupported-operator |
0 | 14 | 2 |
invalid-argument-type |
0 | 1 | 2 |
unsound-return-statement |
0 | 2 | 0 |
not-iterable |
0 | 0 | 1 |
| Total | 34 | 17 | 6 |
Flaky changes detected. This PR summary excludes flaky changes; see the HTML report for details.
Raw diff (57 changes)
aiortc (https://github.com/aiortc/aiortc)
- src/aiortc/rtcsctptransport.py:547:15 error[unsupported-operator] Operator `<` is not supported between objects of type `int | Unknown | None` and `int`
- src/aiortc/rtcsctptransport.py:548:21 error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> DataChunk, (s: slice[SupportsIndex | None, SupportsIndex | None, SupportsIndex | None], /) -> list[DataChunk]]` cannot be called with key of type `None` on object of type `list[DataChunk]`
- src/aiortc/rtcsctptransport.py:555:25 error[unsupported-operator] Operator `+=` is not supported between objects of type `None` and `Literal[1]`
- src/aiortc/rtcsctptransport.py:566:21 error[unsupported-operator] Operator `+=` is not supported between objects of type `None` and `Literal[1]`
- src/aiortc/rtcsctptransport.py:571:71 error[unsupported-operator] Operator `+` is not supported between objects of type `int | Unknown | None` and `Literal[1]`
- src/aiortc/rtcsctptransport.py:574:67 error[unsupported-operator] Operator `+` is not supported between objects of type `int | Unknown | None` and `Literal[1]`
- src/aiortc/rtcsctptransport.py:581:17 error[unsupported-operator] Operator `+=` is not supported between objects of type `None` and `Literal[1]`
apprise (https://github.com/caronc/apprise)
- apprise/conversion.py:1250:23 error[unresolved-attribute] Attribute `rstrip` is not defined on `None` in union `None | Unknown | str`
+ apprise/conversion.py:1250:23 error[unresolved-attribute] Attribute `rstrip` is not defined on `None` in union `None | str | Unknown`
core (https://github.com/home-assistant/core)
- homeassistant/components/hunterdouglas_powerview/cover.py:113:13 error[unsupported-operator] Operator `|=` is not supported between objects of type `None` and `Literal[CoverEntityFeature.STOP]`
- homeassistant/components/hunterdouglas_powerview/cover.py:373:9 error[unsupported-operator] Operator `|=` is not supported between objects of type `None` and `Literal[CoverEntityFeature.OPEN_TILT]`
- homeassistant/components/lcn/cover.py:199:13 error[unsupported-operator] Operator `|=` is not supported between objects of type `None` and `Literal[CoverEntityFeature.SET_POSITION]`
- homeassistant/components/overkiz/climate/atlantic_electrical_towel_dryer.py:69:13 error[unsupported-operator] Operator `+=` is not supported between objects of type `None` and `list[str]`
- homeassistant/components/overkiz/climate/atlantic_electrical_towel_dryer.py:72:20 error[unsupported-operator] Operator `not in` is not supported between objects of type `str` and `list[str] | None`
- homeassistant/components/overkiz/climate/atlantic_electrical_towel_dryer.py:148:12 error[unsupported-operator] Operator `in` is not supported between objects of type `Literal["drying"]` and `list[str] | None`
- homeassistant/components/overkiz/climate/atlantic_electrical_towel_dryer.py:190:14 error[unsupported-operator] Operator `in` is not supported between objects of type `Literal["drying"]` and `list[str] | None`
- homeassistant/components/velux/cover.py:248:9 error[unsupported-operator] Operator `|=` is not supported between objects of type `None` and `Literal[CoverEntityFeature.OPEN_TILT]`
dulwich (https://github.com/dulwich/dulwich)
- dulwich/object_store.py:4082:20 warning[unsound-return-statement] Unsound return statement: `tuple[(EWAHBitmap & ~AlwaysFalsy) | (Unknown & ~AlwaysFalsy), Pack & ~AlwaysFalsy]` is not a subtype of `tuple[EWAHBitmap, Pack] | None`
- dulwich/objects.py:1040:16 warning[unsound-return-statement] Unsound return statement: `list[bytes | Unknown]` is not a subtype of `list[bytes]`
manticore (https://github.com/trailofbits/manticore)
+ manticore/platforms/evm.py:2982:9 error[unresolved-attribute] Object of type `Self@add_refund` has no attribute `_refund`
+ manticore/platforms/evm.py:2985:9 error[unresolved-attribute] Object of type `Self@sub_refund` has no attribute `_refund`
pip (https://github.com/pypa/pip)
+ src/pip/_vendor/pygments/filters/__init__.py:833:12 error[unresolved-attribute] Object of type `Self@__init__` has no attribute `tabs`
+ src/pip/_vendor/pygments/filters/__init__.py:834:13 error[unresolved-attribute] Object of type `Self@__init__` has no attribute `tabs`
+ src/pip/_vendor/pygments/filters/__init__.py:835:12 error[unresolved-attribute] Object of type `Self@__init__` has no attribute `newlines`
+ src/pip/_vendor/pygments/filters/__init__.py:836:13 error[unresolved-attribute] Object of type `Self@__init__` has no attribute `newlines`
+ src/pip/_vendor/pygments/filters/__init__.py:842:20 error[unresolved-attribute] Object of type `Self@filter` has no attribute `tabs`
+ src/pip/_vendor/pygments/filters/__init__.py:843:24 error[unresolved-attribute] Object of type `Self@filter` has no attribute `newlines`
+ src/pip/_vendor/pygments/filters/__init__.py:859:51 error[unresolved-attribute] Object of type `Self@filter` has no attribute `tabs`
+ src/pip/_vendor/pygments/filters/__init__.py:859:62 error[unresolved-attribute] Object of type `Self@filter` has no attribute `newlines`
pywin32 (https://github.com/mhammond/pywin32)
+ pythonwin/pywin/scintilla/formatter.py:670:24 error[unresolved-attribute] Object of type `Self@RegisterStyle` has no attribute `nextstylenum`
+ pythonwin/pywin/scintilla/formatter.py:671:13 error[unresolved-attribute] Object of type `Self@RegisterStyle` has no attribute `nextstylenum`
scipy (https://github.com/scipy/scipy)
+ scipy/special/_orthogonal.py:187:9 error[unresolved-attribute] Object of type `Self@_scale` has no attribute `_coeffs`
spack (https://github.com/spack/spack)
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:35:9 error[unresolved-attribute] Unresolved attribute `_count` on type `Self@__new__`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:41:29 error[unresolved-attribute] Object of type `Self@__new__` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:45:16 error[unresolved-attribute] Object of type `Self@__len__` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:59:22 error[unresolved-attribute] Object of type `Self@__getitem__` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:81:64 error[unresolved-attribute] Object of type `Self@__eq__` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:274:18 error[unresolved-attribute] Object of type `Self@set` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:276:21 error[unresolved-attribute] Object of type `Self@set` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:280:38 error[unresolved-attribute] Object of type `Self@set` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:282:34 error[unresolved-attribute] Object of type `Self@set` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:284:17 error[unresolved-attribute] Object of type `Self@set` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:317:13 error[unresolved-attribute] Object of type `Self@_create_new_root` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:329:34 error[unresolved-attribute] Object of type `Self@append` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:333:30 error[unresolved-attribute] Object of type `Self@append` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:350:9 error[unresolved-attribute] Object of type `Self@_mutating_fill_tail` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:361:29 error[unresolved-attribute] Object of type `Self@_mutating_extend` has no attribute `_count`
+ lib/spack/spack/vendor/pyrsistent/_pvector.py:389:23 error[unresolved-attribute] Object of type `Self@_push_tail` has no attribute `_count`
spark (https://github.com/apache/spark)
- python/pyspark/pandas/frame.py:12066:35 error[unsupported-operator] Operator `|` is not supported between objects of type `Column | (Unknown & ~None)` and `None | Column | Unknown`
+ python/pyspark/pandas/frame.py:12066:35 error[unsupported-operator] Operator `|` is not supported between objects of type `Column | (Unknown & ~None)` and `None | Column`
- python/pyspark/pandas/frame.py:12081:61 error[invalid-argument-type] Argument to bound method `InternalFrame.with_filter` is incorrect: Expected `Column | Series[Unknown]`, found `None | Column | Unknown`
+ python/pyspark/pandas/frame.py:12081:61 error[invalid-argument-type] Argument to bound method `InternalFrame.with_filter` is incorrect: Expected `Column | Series[Unknown]`, found `None | Column`
- python/pyspark/pandas/frame.py:12094:61 error[invalid-argument-type] Argument to bound method `InternalFrame.with_filter` is incorrect: Expected `Column | Series[Unknown]`, found `None | Column | Unknown`
+ python/pyspark/pandas/frame.py:12094:61 error[invalid-argument-type] Argument to bound method `InternalFrame.with_filter` is incorrect: Expected `Column | Series[Unknown]`, found `None | Column`
sympy (https://github.com/sympy/sympy)
- sympy/core/add.py:1238:28 error[unsupported-operator] Operator `not in` is not supported between objects of type `Unknown` and `None | set[Unknown] | (Unknown & ~AlwaysFalsy)`
+ sympy/core/add.py:1238:28 error[unsupported-operator] Operator `not in` is not supported between objects of type `Unknown` and `None | set[Unknown]`
- sympy/core/add.py:1244:26 error[not-iterable] Object of type `None | set[Unknown] | (Unknown & ~AlwaysFalsy)` may not be iterable
+ sympy/core/add.py:1244:26 error[not-iterable] Object of type `None | set[Unknown]` may not be iterable
+ sympy/geometry/polygon.py:1695:16 error[unresolved-attribute] Object of type `Self@rotation` has no attribute `_rot`
+ sympy/geometry/polygon.py:1943:9 error[unresolved-attribute] Object of type `Self@spin` has no attribute `_rot`
+ sympy/geometry/polygon.py:1965:9 error[unresolved-attribute] Object of type `Self@rotate` has no attribute `_rot`
+ sympy/geometry/polygon.py:2048:15 error[unresolved-attribute] Object of type `Self@vertices` has no attribute `_rot`
+ sympy/geometry/polygon.py:1533:53 error[unresolved-attribute] Object of type `Self@args` has no attribute `_rot`
Merging this PR will not alter performance
Comparing Footnotes
|
a867375 to
892f4c7
Compare
|
Per the report, the false positives in the ecosystem report are generally consistent with other type checkers. |
|
(I find the title of this PR confusing -- it sounds like the opposite of what the PR does, which is to stop inferring (the existence of) instance attributes from augmented assignments.) |
|
Thanks, sorry, the PR went through a few iterations. I tightened it up. |
| // Expression inference can temporarily substitute a cycle placeholder for every | ||
| // subexpression. `None` has a stable type regardless of that cycle, so preserve it when | ||
| // deriving comparison constraints for recursively inferred attributes. | ||
| let expression_type = |expr: &ast::Expr, env: &ProgramEnvironment<'db>| { | ||
| if expr.is_none_literal_expr() { | ||
| Type::none(db, env) | ||
| } else { | ||
| inference.expression_type(expr) | ||
| } | ||
| }; |
There was a problem hiding this comment.
I'm still looking into this (instance attributes are hard) but one thing that caught my eye which might be worth looking into:
This seems like a narrow fix specifically targeting None literal as using anything other than None results in the same issue that this block is trying to prevent:
from typing import Literal
class Counter:
def __init__(self, value: str | Literal[False]) -> None:
self.value = value
def update(self) -> None:
if self.value is False:
return
self.value += "x" # A
self.value += "x" # B
# revealed: str | Literal[False] | Unknown
reveal_type(Counter("").value)It's interesting that this issue doesn't occur when there's single augmented assignment, it required having two augmented assignment as in the above example. I asked Codex to figure out why this is and it gave me the following (adding it as verbatim):
With only assignment A, the dependency chain is:
Infer attribute → Infer assignment A → Infer guard: self.value is False → Look up self.value → Infer assignment A again → Assignment A is already running: recover with Divergent → Guard finishes normally and still knows False is False → Narrowing succeedsThe cycle occurs in assignment A’s query. The guard query itself never becomes cyclic, so its literal operand retains the correct type.
With assignments A and B:
Infer attribute → Infer assignment A → Infer guard: self.value is False → Look up self.value → Infer assignment A again → Recover A with Divergent → Infer assignment B → Infer the same guard again → Guard is already running: recover the entire guard with Divergent → False is incorrectly treated as Divergent → Narrowing fails → Assignment B contributes UnknownThe crucial difference is that after recovering A, attribute lookup continues to B because it must consider every assignment. B then requests the same guard expression, creating a cycle in the expression-inference query, whose recovery replaces every subexpression—including False—with Divergent.
There was a problem hiding this comment.
I think it would be useful to include this context in the comment (i.e. that this is a narrow fix for a more general issue, but for now we've intentionally decided to only apply the narrow version.)
There was a problem hiding this comment.
Sounds good, added.
dhruvmanila
left a comment
There was a problem hiding this comment.
I wouldn't consider myself to be an expert in instance attributes but after spending some time trying to understand attribute lookup machinery between instances, classes and the MRO-based lookup, I don't see any issue with the change here. It looks good from my end. I do feel like it might be useful for @sharkdp (after he's back) to give it a quick look as well if possible.
Related to the new diagnostics in the ecosystem result, I see that those have been deliberately left out given that other type checkers also don't support it. Can you open an issue to keep track of this missing cases?
As an aside, I had to spend a lot of time trying to understand the how does the MRO-based lookup fit into the wider attribute lookup process even after David had spent a ty sharing session explaining it and I still find it confusing 🙈
| reveal_type(UsesGeneratedDescriptor().generated_descriptor) # revealed: Literal["descriptor"] | ||
| ``` | ||
|
|
||
| An augmented assignment to a data descriptor on a metaclass calls the descriptor's `__set__` method. |
|
It looks like 21 of the false-positives are covered by astral-sh/ty#1268. Eight of the diagnostics (from pip) use literals with This case doesn't seem to have an issue but I'm also not sure we actually want to support it: class Example:
def __new__(cls):
instance = super().__new__(cls)
instance.value = 0
return instance
def current(self) -> int:
return self.value |
…stral-sh#27633) ## Summary Previously, we treated augmented assignments as ordinary bindings when inferring implicit attributes. However, `self.value += 1` first reads `self.value`, so it cannot establish an otherwise missing attribute: ```python class Counter: def __init__(self) -> None: self.value = 0 def increment(self) -> None: self.value += 1 class UninitializedCounter: def increment(self) -> None: self.value += 1 # error: [unresolved-attribute] ``` We now distinguish assignments that establish an attribute from augmented assignments that require an existing value. Class and instance member lookup collects augmented assignments while traversing the MRO and incorporates their inferred results only after resolving an independent attribute binding. This preserves missing-attribute diagnostics and allows augmented assignments to contribute to inferred attribute types.
Summary
Previously, we treated augmented assignments as ordinary bindings when inferring implicit attributes. However,
self.value += 1first readsself.value, so it cannot establish an otherwise missing attribute:We now distinguish assignments that establish an attribute from augmented assignments that require an existing value. Class and instance member lookup collects augmented assignments while traversing the MRO and incorporates their inferred results only after resolving an independent attribute binding.
This preserves missing-attribute diagnostics and allows augmented assignments to contribute to inferred attribute types.