igerber
diff --git a/‎CHANGELOG.md‎
Lines changed: 1 addition & 1 deletion b/‎CHANGELOG.md‎
Lines changed: 1 addition & 1 deletion
@@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
-- **Tutorial 22: Survey-Weighted HAD** (`docs/tutorials/22_had_survey_design.ipynb`) — end-to-end walkthrough of `HeterogeneousAdoptionDiD` + `did_had_pretest_workflow` on a BRFSS-shape stratified household-survey panel (5 strata × 6 PSUs/stratum × 2 states/PSU = 60 states; post-stratification raking weights with CV ~ 0.30; FPC = 30 PSUs/stratum; PSU × period interaction shocks injected so cluster correlation survives DiD first-differencing). Demonstrates the `SurveyDesign(strata=...)` path through the Stute pretest family that the previous `[Unreleased]` entry unblocked. Eight numbered sections: motivation; panel + in-notebook helper for attaching survey columns to a HAD panel; naive vs survey-aware headline fit with a side-by-side ATT / SE / CI table (~10% SE inflation, sign-only direction asserted); a dedicated section explaining why the SE inflation is modest for HAD specifically (WAS-d_lower IF concentration at the boundary vs full-panel regression coefficients); event-study fit with sup-t cband under the survey design (per-horizon table + matplotlib gated plot); pretest workflow on both `aggregate="overall"` and `aggregate="event_study"` paths walking the Phase 4.5 C0 QUG-deferred verdict suffix and the now-supported stratified-clustered Stute multiplier bootstrap; "Communicating to Leadership" two-paragraph template (executive + methodologist); Extensions + Summary Checklist surfacing the still-deferred `lonely_psu='adjust'` + singleton-strata, replicate-weight designs, and the permanent QUG-under-survey C0 deferral. Companion drift-test file `tests/test_t22_had_survey_design_drift.py` (25 tests across 5 groups: panel + survey composition with deterministic exact pins; naive-vs-survey headline with sign-only SE-inflation anchor; event-study cband-vs-pointwise ordering and post/pre coverage; pretest overall path with `_QUG_DEFERRED_SUFFIX` lock and Yatchew `sigma2_*` deterministic pins; pretest event-study path with the SAME `_QUG_DEFERRED_SUFFIX` lock plus a SEPARATE substring lock on `report.summary()` for the L736 QUG-skip note). Bootstrap p-value pins use abs tolerance bands `>= 0.25` per `feedback_strata_bootstrap_path_divergence` (stratified Mammen multiplier paths reduce effective dofs vs non-strata; PR #432 commit `aef07020` already had to relax bit-equality bands on this code path). T20 and T21 "Extensions" bullets updated with forward-pointers to T22; `docs/practitioner_decision_tree.rst` HAD universal-rollout and survey sections each gain a `.. tip::` cross-link to T22 (adjacent to T20 / T17, NOT displacing); `docs/api/had.rst` gains a "Survey-aware fit" cross-reference; `docs/survey-roadmap.md` gains a "Phase 4.5 C ✅ Shipped" entry; bundled `diff_diff/guides/llms.txt`, `llms-full.txt`, and `llms-practitioner.txt` carry T22 inventory entries; `docs/doc-deps.yaml` wires T22 as a dependent of both `had.py` and `had_pretests.py`. Closes the Phase 5 (wave 2 second slice) tutorial gap; the realistic survey-weighted HAD workflow on BRFSS / CPS / NHANES / ACS-shaped designs is now end-to-end documented for practitioners.
+- **Tutorial 22: Survey-Weighted HAD** (`docs/tutorials/22_had_survey_design.ipynb`) — end-to-end walkthrough of `HeterogeneousAdoptionDiD` + `did_had_pretest_workflow` on a BRFSS-shape stratified household-survey panel (5 strata × 6 PSUs/stratum × 2 states/PSU = 60 states; post-stratification raking weights with CV ~ 0.30; FPC = 30 PSUs/stratum; PSU × period interaction shocks injected so cluster correlation survives DiD first-differencing). Demonstrates the `SurveyDesign(strata=...)` path through the Stute pretest family that the previous `[Unreleased]` entry unblocked. Eight numbered sections: motivation; panel + in-notebook helper for attaching survey columns to a HAD panel; naive vs survey-aware headline fit with a side-by-side ATT / SE / CI table (~10% SE inflation, sign-only direction asserted); a dedicated section explaining why the SE inflation is modest for HAD specifically (WAS-d_lower IF concentration at the boundary vs full-panel regression coefficients); event-study fit with sup-t cband under the survey design (per-horizon table + matplotlib gated plot); pretest workflow on both `aggregate="overall"` and `aggregate="event_study"` paths walking the Phase 4.5 C0 QUG-deferred verdict suffix and the now-supported stratified-clustered Stute multiplier bootstrap; "Communicating to Leadership" two-paragraph template (executive + methodologist); Extensions + Summary Checklist surfacing the still-deferred `lonely_psu='adjust'` + singleton-strata, replicate-weight designs, and the permanent QUG-under-survey C0 deferral. Companion drift-test file `tests/test_t22_had_survey_design_drift.py` (25 tests across 5 groups: panel + survey composition with deterministic exact pins; naive-vs-survey headline with sign-only SE-inflation anchor; event-study cband-vs-pointwise ordering and post/pre coverage; pretest overall path with `_QUG_DEFERRED_SUFFIX` lock and Yatchew `sigma2_*` deterministic pins; pretest event-study path with the SAME `_QUG_DEFERRED_SUFFIX` lock plus a SEPARATE substring lock on `report.summary()` for the L736 QUG-skip note). Bootstrap p-value pins use abs tolerance bands `>= 0.25` per `feedback_strata_bootstrap_path_divergence` (stratified Mammen multiplier paths reduce effective dofs vs non-strata; PR #432 commit `aef07020` already had to relax bit-equality bands on this code path). T20 and T21 "Extensions" bullets updated with forward-pointers to T22; `docs/practitioner_decision_tree.rst` HAD universal-rollout and survey sections each gain a `.. tip::` cross-link to T22 (adjacent to T20 / T17, NOT displacing); `docs/api/had.rst` gains a "Survey-aware fit" cross-reference; `docs/survey-roadmap.md` gains a "Phase 4.5 C ✅ Shipped" entry; bundled `diff_diff/guides/llms.txt` and `llms-practitioner.txt` carry T22 inventory entries (the `llms-full.txt` reference guide is left as a follow-up to keep T22 PR scope tight); `docs/doc-deps.yaml` wires T22 as a dependent of both `had.py` and `had_pretests.py`. Closes the Phase 5 (wave 2 second slice) tutorial gap; the realistic survey-weighted HAD workflow on BRFSS / CPS / NHANES / ACS-shaped designs is now end-to-end documented for practitioners.
 - **HAD pretest workflow: stratified survey-design support (Phase 4.5 C continuation).** Lifts the `NotImplementedError` gate on `SurveyDesign(strata=...)` in `stute_test` (`had_pretests.py:1927-1940` pre-PR) and `stute_joint_pretest` (`:3259-3271` pre-PR), and by inheritance in `joint_pretrends_test`, `joint_homogeneity_test`, and `did_had_pretest_workflow` (the wrappers delegate to the joint Stute helper). Implements a documented synthesis of clustered-wild-bootstrap ingredients (Cameron-Gelbach-Miller 2008 cluster-level multipliers; Davidson-Flachaire 2008 wild-bootstrap centering; Djogbenou-MacKinnon-Nielsen 2019 cluster-wild consistency for nonlinear functionals; Kreiss-Lahiri 2012 within-block centering analogy; Wu 1986 / Liu 1988 Bessel small-sample correction) — no single paper covers the exact composition for the stratified Stute CvM functional. The recipe: within-stratum demean + `sqrt(n_h/(n_h-1))` Bessel rescale applied to the PSU multipliers `psu_mults` BEFORE the per-obs broadcast `eta_obs = psu_mults[b, psu_col_idx]` in the wild-residual loop. Bootstrap CvM variance matches the analytical Binder-TSL stratified target `V_S = sum_h (1 - f_h) (n_h / (n_h - 1)) sum_j (psi_hj - psi_h_bar)²` exactly (the `(1 - f_h)` FPC factor was already baked in by `generate_survey_multiplier_weights_batch`; this PR bakes the remaining `(n_h / (n_h - 1))` factor and enforces within-stratum-mean-zero centering). New shared helper `bootstrap_utils.apply_stratum_centering(psu_mults, resolved_survey, psu_ids, psu_axis=...)` is called from both the new Stute path (psu_axis=1 on the multiplier matrix) AND the existing HAD sup-t event-study cband bootstrap (psu_axis=0 on the PSU-aggregated influence tensor; refactored bit-exactly from the inline block previously at `had.py:2172-2204`). Locks the algebraic identity architecturally instead of leaving parallel code blocks to drift. MC oracle consistency validated under a 4-stratum × 6-PSU/stratum stratified null DGP with weights+strata+PSU (200 seeded draws, empirical Type I at α=0.05 in `[0, 0.10]` — 3σ band; the FPC bake-in is covered separately by the helper-unit test `test_fpc_baked_in_helper_is_fpc_agnostic`); MC power validated under a known-alternative stratified DGP (rejection > 0.50). HAD sup-t event-study cband bit-parity preserved (`atol=1e-14, rtol=1e-14` on the refactored helper output + 29 existing cband tests passing post-refactor; that helper-level bit-parity test locks the axis-0 algebra). A separate wired-in regression at `tests/test_had_pretests.py::TestStuteStratifiedSurveyBootstrap::test_stute_call_sites_invoke_apply_stratum_centering` monkey-patches the helper and asserts both Stute call sites (`stute_test` at `had_pretests.py:1985` and `stute_joint_pretest` at `:3312`) invoke it with `psu_axis=1` — that test fails if either call site is disconnected (the axis-0 helper-parity test alone does not catch that case). See `docs/methodology/REGISTRY.md` § HeterogeneousAdoptionDiD — "Note (Stute stratified survey-bootstrap calibration)" for the full derivation. Remaining deferrals: `lonely_psu='adjust'` + singleton-strata (same pseudo-stratum centering gap as the HAD sup-t deviation at REGISTRY:2382) and replicate-weight designs (BRR/Fay/JK1/JKn/SDR — separate Rao-Wu / JKn bootstrap composition). Unblocks the realistic survey-weighted HAD workflow on BRFSS/CPS/NHANES/ACS-shaped designs.
 - **Conley (1999) Wave A mechanical extensions** on top of the Phase 1+2 sandwich (`diff_diff/conley.py`, `diff_diff/linalg.py`, `diff_diff/estimators.py`, `diff_diff/twfe.py`). **(1) DiD support (#118):** `DifferenceInDifferences(vcov_type="conley").fit(..., unit="<col>")` is now supported. `unit` is a fit-time kwarg (NOT on `__init__`; unused unless Conley is set; not part of `get_params()` / `set_params()`) mirroring `MultiPeriodDiD.fit(unit=...)` / `TwoWayFixedEffects.fit(unit=...)`. DiD inherits the same panel block-decomposed sandwich as MPD/TWFE; on a 2-period panel it matches `MultiPeriodDiD(...).fit(..., post_periods=[1], reference_period=0)` bit-exactly. Missing `unit=`/`conley_lag_cutoff`/`conley_coords`/`conley_cutoff_km` raise `ValueError`; `survey_design=` + Conley raises `NotImplementedError` (Bertanha-Imbens 2014 follow-up); `inference="wild_bootstrap"` + Conley raises `NotImplementedError`. **(2) Combined spatial + cluster product kernel (#119):** `compute_robust_vcov(vcov_type="conley", cluster_ids=...)` / `LinearRegression(vcov_type="conley", cluster_ids=...)` / `TwoWayFixedEffects(vcov_type="conley", cluster="<col>")` / `MultiPeriodDiD(vcov_type="conley", cluster="<col>")` / `DifferenceInDifferences(vcov_type="conley", cluster="<col>")` apply `K_total[i, j] = K_space(d_ij/h) · 1{c_i = c_j}`. On the panel block-decomposed path the cluster indicator multiplies BOTH the spatial sandwich AND the serial sandwich; the validator enforces that `cluster_ids` is constant within each unit across periods (the within-unit serial mask is then trivially all-ones; cross-sectional path has no such constraint). TWFE's default auto-cluster on the Conley path remains silently dropped (combining with unit-level clusters would zero out all between-unit pairs and defeat the spatial pooling); users must pass an explicit above-unit cluster (e.g. region) to opt in. DiD has no auto-cluster — the choice is fully explicit. Two limit fixtures anchor correctness (no R parity — R `conleyreg` does not support combined kernels): all-unique-clusters reduces to HC0; huge-cutoff reduces to pure within-cluster CR1. The huge-cutoff reduction is EXACT only for `conley_kernel="uniform"` (`K(u) = 1` for `|u| ≤ 1`); for `conley_kernel="bartlett"` the identity is asymptotic since `K_bartlett(u) = 1 - |u| < 1` for `u > 0`. The fixture anchor uses uniform for an exact identity check. Per-slice mask construction (NOT full n×n) preserves memory on panel paths. **(3) Sparse k-d-tree fast path (#120):** auto-activates for the spatial Bartlett meat when `n > 5_000` AND metric is `"haversine"` or `"euclidean"` AND kernel is `"bartlett"`. Builds a CSR sparse kernel matrix via `scipy.spatial.cKDTree.query_ball_tree` instead of materializing the full n×n distance matrix; haversine projects to a 3-D unit-sphere chord representation with the exact great-circle recomputed for in-range neighbors only. Bit-identity parity vs the dense path at `atol=1e-10`; R parity at `atol=1e-6` is preserved on the existing 3 panel R fixtures with the sparse path force-enabled. The bartlett-only gate is for boundary correctness — bartlett at `u=1` is exactly 0, so the sparse path safely drops at-cutoff pairs; uniform at `u=1` is 1 and would require a closed-interval query semantic that haversine chord projection cannot reliably preserve. Constants: `_CONLEY_SPARSE_N_THRESHOLD = 5_000` (auto-toggle); `_CONLEY_DENSE_WARN_N` renamed `_CONLEY_DENSE_OOM_WARN_N = 20_000` (memory exhaustion threshold for the dense fallback — independent of the sparse threshold). Private `_conley_sparse: Optional[bool]` kwarg on `_compute_conley_vcov` controls the toggle (`None` = auto, `True` = force, `False` = force dense; `True` with an unsupported kernel/metric raises). The serial component (within-unit Bartlett over time) remains dense regardless — per-unit slices are small. **(4) Callable `conley_metric` validation (#123):** result must satisfy shape `(n, n)`, finite, non-negative, symmetric to `atol=1e-10`, AND zero on the diagonal (`|d(i, i)| ≤ 1e-10`); each failure raises a targeted `ValueError` naming the violated invariant. The zero-diagonal contract is load-bearing for the Conley sandwich: the `i = j` term must reduce to the HC0 diagonal `X_i ε_i² X_i'` via `K(0) = 1`; positive self-distance would silently attenuate the HC0 contribution by `K(d_ii / h) < 1`. Built-in metrics (`"haversine"`, `"euclidean"`) satisfy this by construction. Previously, malformed callables produced opaque BLAS errors deep in the pipeline. **Tests:** `tests/test_conley_vcov.py::TestConleySparse` (12), `::TestConleySparseRParityForced` (3), `::TestConleyCluster` (10), `::TestConleyDistanceMetrics` extended (7 new); existing rejection tests flipped to behavioral; `test_did_conley_matches_mpd_post_periods_1` locks the DiD-vs-MPD bit-exact agreement. **Docs:** REGISTRY `## ConleySpatialHAC` updates: new "Combined spatial + cluster product kernel" + "Performance / scale" subsections, DiD-vs-TWFE cluster asymmetry paragraph, updated panel-API restrictions table. TODO rows 118 / 119 / 120 / 123 removed; rows 121 (Conley + survey_design / weights, Bertanha-Imbens 2014) and 122 (`SyntheticDiD(vcov_type="conley")`, spatial-block bootstrap per Politis-Romano 1994) retained for future waves.
 - **Conley (1999) spatial-HAC standard errors via `vcov_type="conley"`** on cross-sectional `LinearRegression` / `compute_robust_vcov` plus panel `MultiPeriodDiD` / `TwoWayFixedEffects` (Phases 1 and 2 of the spillover-conley initiative). **Cross-sectional contract:** `conley_coords` (n × 2 array of lat/lon or projected coords), `conley_cutoff_km=<float>` (positive finite bandwidth in km for haversine, or coord units for euclidean — REQUIRED, no default per the no-silent-failures contract), `conley_metric="haversine"|"euclidean"|callable` (default `"haversine"`; great-circle uses Earth's mean radius 6371.01 km matching R `conleyreg`), `conley_kernel="bartlett"|"uniform"` (default `"bartlett"`; both kernels emit a `UserWarning` if the resulting meat has a materially negative eigenvalue — neither the radial 1-D Bartlett nor the uniform kernel is formally PSD-guaranteed; Conley 1999's explicit PSD formula is the 2-D separable lattice product window at Eq 3.14). Cross-sectional variance estimator `Var̂(β) = (X'X)^{-1} · ( Σ_{i,j} K(d_ij/h) · X_i ε_i ε_j X_j' ) · (X'X)^{-1}` (Conley 1999 Eq 4.2). **Panel contract (Phase 2, new):** Three new co-required kwargs `conley_time` (n-length array), `conley_unit` (n-length array), and `conley_lag_cutoff=<int>` (non-negative; 0 means within-period spatial only, no serial component) switch into the **block-decomposed panel sandwich** that matches R `conleyreg` with `lag_cutoff > 0`: `XeeX_total = Σ_t (within-period spatial sandwich) + Σ_u (within-unit Bartlett temporal sandwich, lag ∈ {1..L}, same-time excluded)`. This is NOT a multiplicative product kernel — verified empirically against `conleyreg::time_dist` and `XeeXhC` at ~1e-14 on the panel parity fixtures. The temporal kernel is hardcoded Bartlett `(1 - |lag|/(L+1))` regardless of `conley_kernel`, mirroring `conleyreg::time_dist.cpp`; documented as a `Note (deviation from R-symmetric API)` in REGISTRY. **Panel estimator wire-up (Phase 2):** `MultiPeriodDiD(vcov_type="conley", conley_lag_cutoff=...).fit(..., unit=...)` and `TwoWayFixedEffects(vcov_type="conley", conley_lag_cutoff=...).fit(..., unit=...)` lift the Phase 1 fit-time rejection; the `conley_time` and `conley_unit` arrays are auto-derived from the existing `time` and `unit` column-name arguments at fit-time. `DifferenceInDifferences(vcov_type="conley")` is also supported (Wave A #118 in this release; see the Wave A entry above) — pass `unit=<col>` as a fit-time kwarg to `DiD.fit(...)`. **Other constraints (Phase 1, unchanged):** `SyntheticDiD(vcov_type="conley")` raises `TypeError` (uses bootstrap variance, not analytical sandwich); `set_params` mirrors the constructor rejection. `vcov_type="conley"` + `weights=` / `survey_design=` raises `NotImplementedError` (Bertanha-Imbens 2014 weighted-Conley deferred to a follow-up PR). `vcov_type="conley"` + explicit `cluster_ids=` is supported via the combined spatial + cluster product kernel (Wave A #119; see the Wave A entry above). TWFE's default auto-cluster on the Conley path is silently dropped (combining with unit-level clusters would defeat the spatial pooling); users opt into the combined kernel by passing an explicit above-unit cluster. `inference="wild_bootstrap"` + Conley raises (incompatible inference modes). A sparse k-d-tree fast path auto-activates for the spatial Bartlett meat when `n > 5_000` with bartlett kernel and haversine/euclidean metric (Wave A #120); the dense fallback still emits an OOM `UserWarning` at `n > 20_000`. **Implementation:** Helpers live in `diff_diff/conley.py` (`_haversine_km`, `_pairwise_distance_matrix`, `_bartlett_kernel`, `_uniform_kernel`, `_validate_conley_kwargs`, `_compute_conley_vcov` — the validator and sandwich helper now accept keyword-only `time` / `unit` / `lag_cutoff` for the panel path); `compute_robust_vcov` in `diff_diff/linalg.py` threads the new kwargs through. **R `conleyreg` parity (Düsterhöft 2021, CRAN v0.1.9)** on **six** benchmark fixtures (`benchmarks/data/r_conleyreg_conley_golden.json`, regenerable via `benchmarks/R/generate_conley_golden.R`): 3 cross-sectional (Phase 1) + 3 new panel fixtures (`panel_haversine_lag1`, `panel_haversine_lag2`, `panel_lat_lon_realistic_lag1`; n_units × T = 60×3, 80×5, 100×4 at lag={1,2,1}); observed max abs diff ~5.7e-16. Earth radius 6371.01 km matches `conleyreg::haversine_dist`. Test file `tests/test_conley_vcov.py` skips parity cleanly when the JSON is absent. New REGISTRY section `## ConleySpatialHAC`. Subsequent phases of the spillover-conley initiative (ring-indicator spillover-aware DiD per Butts 2021; survey-design / replicate-weight support; `SyntheticDiD` Conley path) are tracked in `TODO.md` under "Tech Debt from Code Reviews" → spillover-conley rows.