Explore Apache Beam for defining pipelines (POC + evaluation)

[ekim1394]

Summary

Explores whether Apache Beam is a good fit for defining this project's pipelines. The existing Reader → Transform → Writer model maps onto Beam almost one-to-one, so this adds a small, isolated proof-of-concept that proves it on the in-process direct runner, plus a written evaluation with a recommendation.

This is an exploration, not a migration — nothing in the production ETL path changes, and the default install/CI are untouched.

What's here:

• Optional beam extra (pyproject.toml) — kept out of core/dev on purpose; apache-beam pins heavy transitive deps (pyarrow/numpy/protobuf/dill) that can clash with the polars/duckdb/pyarrow stack. Adds a run-beam-example console script.
• src/spicy_regs/beam/adapters.py — the reusable bridge: ReadWith (Reader → beam.Create), ApplyTransform (Transform → ParDo/FlatMap), DedupBy (the bulk merge → GroupByKey + pick-latest), WriteWith (Writer → sink DoFn).
• src/spicy_regs/beam/example_pipeline.py — a runnable POC that reuses the real Pipeline base and ExtractRecords transform, adds a whole-dataset dedup, and runs on the in-process runner.
• src/spicy_regs/beam/README.md — the evaluation: how the model maps onto Beam, what Beam buys, what it costs, a recommendation (don't adopt in production at current scale), and concrete "revisit when…" triggers. Linked from CONTRIBUTING.md.
• tests/test_beam_pipeline.py — hermetic test guarded by importorskip so the default suite (no beam extra) skips it and stays fast.
• Excludes the experimental module from ty (apache-beam ships no type stubs).

Recommendation in short: the model fits and the spike proves it, but the per-agency workload is already embarrassingly parallel (ThreadPoolExecutor + cron) and runs free on GitHub Actions, so adopting Beam isn't justified yet. Keep it as an optional, experimental path.

Test plan

Verified locally:

• uv run pytest — 230 passed, 1 skipped (the Beam test, since the beam extra isn't installed by default), 3 deselected.
• uv run ruff check . — all checks passed.
• uv run ty check — all checks passed (beam module excluded).
• uv sync --extra beam — apache-beam 2.72.0 resolves cleanly; no downgrade of numpy/pandas/pyarrow at the runtime Python (3.12).
• uv run pytest tests/test_beam_pipeline.py -v — passes; asserts the transform shapes + uppercases records and DedupBy collapses the duplicate key keeping the latest modify_date.
• Manually exercised: uv run run-beam-example prints the transformed, deduplicated records.

$ uv run run-beam-example
{'post_id': '1', 'user_id': '1', 'title': 'FIRST POST UPDATED', 'body': 'hello again', 'modify_date': '2024-06-01'}
{'post_id': '2', 'user_id': '2', 'title': 'SECOND POST', 'body': 'world', 'modify_date': '2024-01-01'}

Checklist

• My change is scoped to one concern
• I added or updated tests for new behavior
• I updated docs (README / CONTRIBUTING / module READMEs) if relevant
• CI is green (pending)

🤖 Generated with Claude Code

---

Generated by Claude Code