Skip to content

Commit b444300

Browse files
igerberclaude
andcommitted
Address PR #366 CI review round 4 (2 P1 + 1 P2): WooldridgeDiD Poisson estimand wording + is_count_like non-negativity guard
P1 #1 (Wooldridge Poisson estimand wording): The guide §4.11 and §5.3 worked example described `WooldridgeDiD(method="poisson")`'s `overall_att` as a "multiplicative effect" / "log-link effect" / "proportional change" to be reported. Verified against `wooldridge.py:1225` (`att = _avg(mu_1 - mu_0, cell_mask)`) and `_reporting_helpers.py:262-281` (registered estimand: "ASF-based average from Wooldridge ETWFE ... average-structural-function (ASF) contrast between treated and counterfactual untreated outcomes ... on the natural outcome scale"): the actual quantity is `E[exp(η_1)] - E[exp(η_0)]`, an outcome-scale DIFFERENCE, not a multiplicative ratio. An agent following the previous wording would misreport the headline scalar. Rewrote both surfaces to: - Describe the estimand as an ASF-based outcome-scale difference, citing `wooldridge.py:1225` and Wooldridge (2023) + REGISTRY.md §WooldridgeDiD nonlinear / ASF path. - Explicitly note the headline `overall_att` is a difference on the natural outcome scale, NOT a multiplicative ratio. - Mention that a proportional / percent-change interpretation can be derived post-hoc as `overall_att / E[Y_0]` but is not the estimator's reported scalar. Added `test_autonomous_count_outcome_uses_asf_outcome_scale_estimand` in `tests/test_guides.py`: extracts §4.11 and §5.3 blocks, asserts forbidden phrases ("multiplicative effect under qmle", "estimates the multiplicative effect", "multiplicative (log-link) effect", "report the multiplicative effect", "report the multiplicative") do NOT appear, and asserts §5.3 explicitly contains "ASF" and "outcome scale" so future edits cannot silently weaken the description. P1 #2 (`is_count_like` non-negativity guard): The `is_count_like` heuristic gated on integer-valued + has-zeros + right-skewed + > 2 distinct values, but did NOT exclude negative support. Verified against `wooldridge.py:1105-1109`: Poisson method hard-rejects `y < 0` with `ValueError`. Without a value_min >= 0 guard, a right-skewed integer outcome with zeros and some negatives would set `is_count_like=True` and steer an agent toward an estimator that then refuses to fit. Added `value_min >= 0.0` to the heuristic and explained the non-negativity gate in the docstring + autonomous guide §2 field reference (now reads "is_integer_valued AND pct_zeros > 0 AND skewness > 0.5 AND n_distinct_values > 2 AND value_min >= 0"). The guide also notes that the gate exists specifically to align the routing signal with WooldridgeDiD Poisson's hard non-negativity requirement. Added `test_outcome_shape_count_like_excludes_negative_support` in `tests/test_profile_panel.py` covering a Poisson-distributed outcome with a small share of negative integers spliced in: asserts `is_count_like=False` despite the other four conditions firing. P2 (test coverage for both P1s): Both regressions above guard the new contracts. The guide test guards the wording surface; the profile test guards the heuristic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 22ff5dc commit b444300

4 files changed

Lines changed: 153 additions & 25 deletions

File tree

diff_diff/guides/llms-autonomous.txt

Lines changed: 45 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -187,10 +187,15 @@ view. Every field below appears as a top-level key in that dict.
187187
(covers integer dtype and floats that happen to be integer-valued).
188188
- `is_count_like: bool` - heuristic for count-shaped outcomes:
189189
`is_integer_valued AND pct_zeros > 0 AND skewness > 0.5 AND
190-
n_distinct_values > 2`. When `True`, an OLS DiD on the raw
191-
outcome may have well-behaved point estimates but questionable
192-
asymptotic SEs; consider `WooldridgeDiD` (QMLE Poisson family)
193-
or document the choice. See §5.3 for a worked example.
190+
n_distinct_values > 2 AND value_min >= 0`. When `True`, an OLS
191+
DiD on the raw outcome may have well-behaved point estimates
192+
but questionable asymptotic SEs; consider
193+
`WooldridgeDiD(method="poisson")` (QMLE), which estimates the
194+
overall ATT as an ASF-based outcome-scale difference (NOT a
195+
multiplicative ratio) and rejects negative outcomes at fit
196+
time. The non-negativity gate aligns the routing signal with
197+
that hard requirement so the estimator will accept the panel.
198+
See §5.3 for a worked example.
194199
- `is_bounded_unit: bool` - all non-NaN values lie in `[0, 1]`.
195200
When `True` and the linear-DiD point estimate is near the
196201
boundary of feasible support, interpret with care (the linear
@@ -618,15 +623,22 @@ count-shaped outcome (e.g., number of incidents per unit-period) for
618623
which linear DiD point estimates are unbiased but asymptotic SEs may
619624
mislead. `PanelProfile.outcome_shape` exposes the relevant facts:
620625

621-
- `outcome_shape.is_count_like == True` (integer-valued, has zeros,
622-
right-skewed, more than two distinct values) - linear-OLS DiD on the
623-
raw count works for the point estimate but inference may be off.
624-
Consider `WooldridgeDiD` (QMLE Poisson) which estimates the
625-
multiplicative effect directly and gives correctly-calibrated SEs
626-
under quasi-MLE. Linear-OLS DiD with cluster-robust SEs remains a
627-
defensible choice when the count is large enough that the normal
628-
approximation holds; the shape field flags the consideration, not a
629-
rule. §5.3 walks through this pattern with a concrete profile.
626+
- `outcome_shape.is_count_like == True` (integer-valued, non-negative,
627+
has zeros, right-skewed, more than two distinct values) - linear-OLS
628+
DiD on the raw count works for the point estimate but asymptotic SEs
629+
assume normal-shaped errors that a right-skewed distribution can
630+
violate. Consider `WooldridgeDiD(method="poisson")`, which estimates
631+
the overall ATT as an ASF-based outcome-scale difference (per-cell
632+
average of `E[exp(η_1)] - E[exp(η_0)]`, Wooldridge 2023; see
633+
REGISTRY.md §WooldridgeDiD nonlinear / ASF path) under QMLE, with
634+
correctly-calibrated SEs. The headline `overall_att` is a difference
635+
on the outcome scale, NOT a multiplicative ratio; a proportional
636+
interpretation can be derived post-hoc as `overall_att / E[Y_0]` if
637+
desired but is not the estimator's reported scalar. Linear-OLS DiD
638+
with cluster-robust SEs remains a defensible choice when the count
639+
is large enough that the normal approximation holds; the shape field
640+
flags the consideration, not a rule. §5.3 walks through this pattern
641+
with a concrete profile.
630642
- `outcome_shape.is_bounded_unit == True` (values in `[0, 1]`,
631643
e.g. a proportion) - linear DiD can produce predictions outside
632644
`[0, 1]` and inference at the boundary is questionable. No
@@ -798,18 +810,26 @@ Reasoning chain:
798810

799811
1. Same staggered-binary narrowing as §5.1 (CS/SA/dCDH/Imputation/
800812
TwoStage/Stacked/ETWFE/EfficientDiD applicable).
801-
2. `outcome_shape.is_count_like == True` -> linear-OLS DiD on the
802-
raw count gives unbiased point estimates of the additive effect
803-
but its asymptotic SEs assume normal-shaped errors, which a
804-
right-skewed count distribution violates. `WooldridgeDiD`
805-
(`method="poisson"`) estimates the multiplicative
806-
(log-link) effect under QMLE with correct asymptotic SEs, and
807-
maps onto the staggered design natively.
808-
3. Decision: fit `WooldridgeDiD(method="poisson")` for the
809-
primary estimate; report the multiplicative effect (proportional
810-
change) rather than the additive effect. Optionally fit a linear
811-
DiD as a robustness check and document which scale the headline
812-
uses.
813+
2. `outcome_shape.is_count_like == True` AND
814+
`outcome_shape.value_min >= 0` -> linear-OLS DiD on the raw count
815+
gives unbiased point estimates but its asymptotic SEs assume
816+
normal-shaped errors that a right-skewed count distribution
817+
violates. `WooldridgeDiD(method="poisson")` (QMLE) estimates the
818+
overall ATT as an ASF-based outcome-scale difference: the per-cell
819+
average of `E[exp(η_1)] - E[exp(η_0)]` (Wooldridge 2023; see
820+
REGISTRY.md §WooldridgeDiD nonlinear / ASF path), with
821+
correctly-calibrated SEs under quasi-MLE. The Poisson fitter
822+
hard-rejects negative outcomes (`y < 0` raises `ValueError` at
823+
line ~1105 of `wooldridge.py`), which is why `is_count_like` gates
824+
on `value_min >= 0`.
825+
3. Decision: fit `WooldridgeDiD(method="poisson")` for the primary
826+
estimate; report the headline `overall_att` as a difference on the
827+
outcome scale (the same units as the count), not as a
828+
multiplicative ratio. A proportional / percent-change reading can
829+
be derived post-hoc by dividing the ATT by an estimate of
830+
`E[Y_0]`, but the estimator's reported scalar is the difference.
831+
Optionally fit a linear DiD as a robustness check and document
832+
that both estimands are on the same scale.
813833
4. Caveat: when the outcome includes structural zeros that violate
814834
Poisson conditional moments (overdispersion), consider negative
815835
binomial QMLE or a hurdle model; the current suite does not

diff_diff/profile.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -637,12 +637,20 @@ def _compute_outcome_shape(valid: pd.Series, outcome_dtype_kind: str) -> Optiona
637637
skewness = float(m3 / (std**3))
638638
excess_kurtosis = float(m4 / (m2**2) - 3.0)
639639

640+
# Non-negativity is part of the contract: `is_count_like == True`
641+
# is the routing signal toward `WooldridgeDiD(method="poisson")`,
642+
# which hard-rejects negative outcomes at fit time
643+
# (`wooldridge.py:1105` raises `ValueError` on `y < 0`). Without the
644+
# `value_min >= 0` guard, a right-skewed integer outcome with zeros
645+
# and some negatives could set `is_count_like=True` and steer an
646+
# agent toward an estimator that will then refuse to fit.
640647
is_count_like = bool(
641648
is_integer_valued
642649
and pct_zeros > 0.0
643650
and skewness is not None
644651
and skewness > 0.5
645652
and n_distinct > 2
653+
and value_min >= 0.0
646654
)
647655

648656
return OutcomeShape(

tests/test_guides.py

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,62 @@ def test_autonomous_contains_worked_examples_section():
8383
)
8484

8585

86+
def test_autonomous_count_outcome_uses_asf_outcome_scale_estimand():
87+
"""§4.11 and §5.3 must describe `WooldridgeDiD(method="poisson")`'s
88+
`overall_att` as an ASF-based outcome-scale difference (matching the
89+
estimator at `wooldridge.py:1225` and the reporting helper at
90+
`_reporting_helpers.py:262-281`), NOT as a multiplicative /
91+
proportional / log-link effect. An agent following an example that
92+
described the headline as "multiplicative" would misreport the
93+
scalar - the library's reported `overall_att` is `E[exp(η_1)] -
94+
E[exp(η_0)]`, a difference on the natural outcome scale.
95+
96+
Guards against regressing the wording back to "multiplicative
97+
effect" / "proportional change" framing. Multiplicative
98+
interpretations may appear in the guide as a clearly-marked
99+
derived post-hoc reading, but never as the description of the
100+
estimator's reported `overall_att`."""
101+
text = get_llm_guide("autonomous")
102+
# Locate §4.11 and §5.3 blocks; check that within them the Poisson
103+
# path is described with ASF / outcome-scale wording, NOT as the
104+
# estimator's reported scalar being multiplicative or proportional.
105+
sec_4_11_start = text.index("### §4.11 Outcome-shape considerations")
106+
sec_4_11_end = text.index("## §5. Worked examples")
107+
sec_4_11 = text[sec_4_11_start:sec_4_11_end]
108+
109+
sec_5_3_start = text.index("### §5.3 Count-shaped outcome")
110+
sec_5_3_end = text.index("## §6. Post-fit validation utilities")
111+
sec_5_3 = text[sec_5_3_start:sec_5_3_end]
112+
113+
forbidden_phrases = (
114+
"multiplicative effect under qmle",
115+
"estimates the multiplicative effect",
116+
"multiplicative (log-link) effect",
117+
"report the multiplicative effect",
118+
"report the multiplicative",
119+
)
120+
for section_name, body in (("§4.11", sec_4_11), ("§5.3", sec_5_3)):
121+
lowered = body.lower()
122+
for phrase in forbidden_phrases:
123+
assert phrase not in lowered, (
124+
f"{section_name} of the autonomous guide describes the "
125+
f"WooldridgeDiD Poisson `overall_att` with the phrase "
126+
f"{phrase!r}; the estimator returns an ASF-based "
127+
f"outcome-scale difference (`E[exp(η_1)] - E[exp(η_0)]`), "
128+
f"not a multiplicative ratio. See `wooldridge.py:1225` "
129+
f"and `_reporting_helpers.py:262-281`."
130+
)
131+
132+
# Positive: each block must explicitly anchor the estimand to the
133+
# ASF / outcome-scale framing so future edits can't silently weaken
134+
# the description.
135+
assert "ASF" in sec_5_3, "§5.3 must reference the ASF interpretation"
136+
assert "outcome scale" in sec_5_3.lower(), (
137+
"§5.3 must label the WooldridgeDiD `overall_att` as an "
138+
"outcome-scale quantity to prevent multiplicative-ratio drift."
139+
)
140+
141+
86142
def test_autonomous_worked_examples_avoid_recommender_language():
87143
"""Worked examples must mirror the rest of the guide's discipline:
88144
no prescriptive language in the example reasoning. Multiple paths

tests/test_profile_panel.py

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -927,6 +927,50 @@ def test_outcome_shape_binary_outcome_not_count_like():
927927
assert shape.n_distinct_values == 2
928928

929929

930+
def test_outcome_shape_count_like_excludes_negative_support():
931+
"""`is_count_like` is the routing signal toward
932+
`WooldridgeDiD(method="poisson")`, which raises `ValueError` on
933+
negative outcomes (`wooldridge.py:1105`). The heuristic must
934+
therefore gate on `value_min >= 0`: a right-skewed integer outcome
935+
with zeros AND some negative values must NOT set `is_count_like`,
936+
even though the other four conditions (integer-valued, has zeros,
937+
skewness > 0.5, > 2 distinct values) are satisfied. Otherwise the
938+
autonomous-guide §5.3 reasoning chain would steer agents toward an
939+
estimator that will then refuse to fit the panel."""
940+
rng = np.random.default_rng(37)
941+
first_treat = {u: 2 for u in range(101, 201)}
942+
943+
# Mix Poisson with a small share of negative integers - integer-valued,
944+
# has zeros (Poisson(0.5) yields ~60% zeros), and right-skewed; but
945+
# value_min < 0 should kill is_count_like.
946+
def _outcome_fn(u, t, tr, _rng):
947+
base = int(rng.poisson(0.5 + 0.2 * tr))
948+
# 5% of cells become a small negative integer, e.g. -1 or -2.
949+
if rng.random() < 0.05:
950+
return -int(rng.integers(1, 3))
951+
return base
952+
953+
df = _make_panel(
954+
n_units=200,
955+
periods=range(0, 4),
956+
first_treat=first_treat,
957+
outcome_fn=_outcome_fn,
958+
)
959+
profile = profile_panel(df, unit="u", time="t", treatment="tr", outcome="y")
960+
shape = profile.outcome_shape
961+
assert shape is not None
962+
assert shape.value_min < 0, "fixture must include some negative outcomes"
963+
assert shape.is_integer_valued is True
964+
assert shape.pct_zeros > 0.1
965+
assert shape.skewness is not None
966+
assert shape.is_count_like is False, (
967+
"is_count_like must be False when value_min < 0; otherwise an "
968+
"agent following the §5.3 worked example would route the panel "
969+
"to WooldridgeDiD(method='poisson'), which raises ValueError on "
970+
"negative outcomes."
971+
)
972+
973+
930974
def test_outcome_shape_continuous_normal():
931975
"""Normally-distributed float outcome must have is_integer_valued=False,
932976
is_count_like=False, is_bounded_unit=False (signed values exit [0,1])."""

0 commit comments

Comments
 (0)