Skip to content

.agent-plan.md

Latest commit

 

History

History
225 lines (187 loc) · 13.8 KB

.agent-plan.md

File metadata and controls

225 lines (187 loc) · 13.8 KB
Raw

.agent-plan.md — leadforge

Dynamic tracker. Update this file as phases complete. Keep it current.

Current System State

v0.5.0 in progress — Milestones 7–11 complete, v4 dataset shipped. Full simulation engine + render/bundle + exposure filtering + CLI commands + validation harness implemented. v4 engine changes + build pipeline merged (PR #21). v4 dataset generated and validated. PR-agent refresh fallback wiring fixed for bot-authored reviews. 609 tests passing.

Next Up — v4 Lead Scoring Dataset

The primary focus is producing a v4 lead scoring dataset that fixes the issues found in v1–v3 datasets. This requires targeted engine changes + a build pipeline, followed by dataset release.

See docs/v4/design.md for full details.

v4-M0: Planning + spike ✓ (PR #19, #20)

  • docs/v4/design.md — consolidated requirements, contract, engine changes, implementation plan
  • docs/v4/validation_spec.md — automated validation checks
  • docs/v4/planning_pr_review.md — self-review and treatment plan
  • scripts/spike_category_signal.py — spike experiment validating category signal approach
  • Updated CLAUDE.md, AGENTS.md, .agent-plan.md

v4-M1: Engine + build pipeline ✓ (PR #21, merged)

Engine changes:

  • Add category_latent_correlations to difficulty_profiles.yaml (intro profile, scale 1.8)
  • Apply correlations in simulation/population.py via _apply_category_latent_correlations()
  • Wire correlations through api/generator.py → build_population()
  • Add snapshot_day parameter to render/snapshots.py with windowed aggregation
  • Add new features: touches_week_1, days_since_first_touch, expected_acv, total_touches_all
  • Add opportunity_created feature (tracks ANY opp, not just open ones)
  • Add FeatureSpec entries to schema/features.py
  • 19 new tests (16 windowed snapshot + 3 category-latent correlations)

Build pipeline:

  • scripts/build_v4_snapshot.py — day-10 snapshot + leakage trap + structured missingness + subsampling
  • scripts/validate_v4_dataset.py — full validation per docs/v4/validation_spec.md
  • End-to-end: generate bundle → build CSV → validate → all 7 mandatory checks pass
  • LR AUC 0.652 (without trap); 0.034 boost with trap

v4-M2: Documentation + release ✓ (PR #22)

  • Generate lead_scoring_intro_v4.csv (in datasets-private repo)
  • Write RELEASE_v4.md with column dictionary, validation results, pedagogical notes
  • Update dataset repo README (v4 now recommended, v3 superseded)
  • Tune snapshot day (14 → 10) for leakage trap boost ≥ 0.03 after stacking fix
  • Add StandardScaler to validation script LR for convergence
  • Update .agent-plan.md to reflect completion

Deferred Items

From existing roadmap (M12–M15)

Item Status Rationale
M12: CLI --json flag Deferred No consumer needs it yet; add post-v4
M12: CLI --strict flag Deferred Per-check control is better than global flag
M12: CLI help text polish Deferred Low priority vs dataset
M14: Sample bundle commit Absorbed into v4-M2 v4 dataset IS the sample
M14: Notebook 1 (inspecting world) Deferred Do after v4 ships
M14: Notebook 2 (lead scoring baseline) Deferred v4 validation script covers this
M14: Notebook 3 (public vs instructor) Discarded No current audience
M14: Notebook 4 (recipe customization) Discarded Premature
M15: Docs polish + v1.0 RC Deferred Do after v4 ships

From post-v1 list

  • Second vertical
  • LTV labels as first-class task outputs
  • Continuous-time / richer event engine
  • Plugin architecture
  • External-API enrichment
  • Web UI or dashboard
  • Engine fix: is_sql=False → never converts (deterministic invariant)

Context Pointers

  • v4 design (requirements, contract, engine changes, plan): docs/v4/design.md
  • v4 validation spec: docs/v4/validation_spec.md
  • v4 self-review: docs/v4/planning_pr_review.md
  • Spike experiment: scripts/spike_category_signal.py
  • Existing roadmap: docs/leadforge_implementation_plan.md
  • CLI commands: leadforge/cli/commands/
  • Validation modules: leadforge/validation/
  • Snapshot builder: leadforge/render/snapshots.py
  • Mechanism policy: leadforge/mechanisms/policies.py
  • Difficulty profiles: leadforge/recipes/b2b_saas_procurement_v1/difficulty_profiles.yaml

Completed Phases

Milestone 11 — Validation Harness ✓ (v0.5.0)

  • validation/bundle_checks.py: orchestrator — artifact, FK, leakage checks + wires realism/difficulty
  • validation/invariants.py: determinism (same seed → identical hashes), exposure monotonicity (student ⊂ instructor)
  • validation/realism.py: conversion rate bounds, non-empty core tables, feature value ranges (non-negative counts, valid booleans), stage distribution diversity
  • validation/difficulty.py: known-difficulty validation, ordering check (no-op until engine modulates by difficulty)
  • validation/drift.py: cross-seed stability — conversion rate spread, degenerate seed detection
  • All wired into validate_bundle() via include_realism flag
  • 18 new validation tests; total 581 passing

Milestone 10 — CLI Commands ✓ (v0.4.0)

  • cli/commands/generate.py: fully wired — parses all flags, calls Generator.from_recipe().generate(), writes bundle via .save()
  • cli/commands/inspect.py: reads manifest.json and prints summary (recipe, seed, mode, tables with row counts, task splits, metadata presence)
  • cli/commands/validate.py: checks manifest presence, required files, table row counts, SHA-256 hashes, task split integrity, FK constraints, leakage (unexpected columns)
  • 22 CLI tests (smoke, generate integration, inspect output, validate pass/fail/corrupt/missing); total 562 passing

Milestone 9 — Exposure Filtering ✓ (v0.4.0)

  • exposure/filters.py: BundleFilter frozen dataclass; FILTERS dict keyed by ExposureMode; get_filter() accepts str | ExposureMode
  • exposure/metadata.py: write_metadata_dir() — writes metadata/ with graph.json, graph.graphml, world_spec.json, latent_registry.json, mechanism_summary.json
  • exposure/modes.py: apply_exposure(bundle, root, mode) — dispatch; removes stale metadata/ for student_public
  • Wired into api/bundle.py between dataset card and manifest steps
  • 22 exposure tests; total 547 passing

Milestone 8 — Render / Bundle Layer ✓ (v0.4.0 in PR)

  • render/relational.py: to_dataframes() — 9-table dict of typed DataFrames from SimulationResult + PopulationResult
  • render/snapshots.py: build_snapshot() — 30-column leakage-free lead snapshot with touch/session/activity aggregates, account/contact field joins
  • render/tasks.py: write_task_splits() — deterministic 70/15/15 train/valid/test Parquet split + task_manifest.json
  • render/manifests.py: build_manifest() / write_manifest() — manifest.json with provenance, row counts, SHA-256 hashes
  • api/bundle.py: write_bundle() — orchestrates all render steps; writes full bundle to disk
  • core/models.py: WorldBundle.save(path) — delegates to write_bundle() via lazy import
  • api/generator.py: Generator.generate() — fully implemented end-to-end flow
  • 31 new render tests; total 521 passing

Milestone 7 — Simulation Engine ✓ (v0.4.0 in PR)

  • simulation/state.py: LeadSimState — per-lead mutable state (stage, dwell, converted, churned, sql_day)
  • simulation/engine.py: simulate_world() — 90-day daily-step loop; SimulationResult output type
    • Churn: daily 0.4% probability → closed_lost
    • Stage advance: HazardTransition drives mql → … → negotiation
    • Conversion: ConversionHazard fires from negotiation → closed_won
    • Touch emission: RecencyDecayIntensity per day; session (30% of touch-days) and sales-activity (20% at sql+) emission
    • Post-sim: OpportunityRow for sql+ leads; CustomerRow + SubscriptionRow for closed_won leads
  • 45 new tests; total 490 passing

Milestone 6 — Mechanism Layer v1 ✓ (v0.3.0 merged)

  • mechanisms/base.py: Mechanism ABC, MechanismContext, MechanismSummary, MechanismAssignment
  • mechanisms/static.py: CategoricalDraw, BoundedNumericDraw, MixtureDraw
  • mechanisms/influence.py: AdditiveInfluence, LogisticInfluence, SaturatingInfluence, ThresholdInfluence, InteractionTerm
  • mechanisms/scores.py: LatentScore — logistic score from weighted latent combination
  • mechanisms/hazards.py: ConversionHazard — daily conversion probability from latent score
  • mechanisms/transitions.py: StageSequence, HazardTransition — funnel stage advancement
  • mechanisms/counts.py: PoissonIntensity, RecencyDecayIntensity — touch/session counts
  • mechanisms/categorical.py: CategoricalInfluence, CHANNEL_QUALITY_SCORES
  • mechanisms/measurement.py: NoisyProxy, NoisyCategorization, ProxyCompression
  • mechanisms/policies.py: assign_mechanisms() — motif-family-aware MechanismAssignment factory
  • 74 tests; total 437 passing

Milestone 5 — Population Generation ✓ (v0.3.0 in PR)

  • leadforge/simulation/population.py: build_population() — accounts (3 latent traits), contacts (4 latent traits, conditional on account), leads (1 latent trait, FK-consistent), LatentState and PopulationResult output types
  • Motif-family latent biases (_MOTIF_LATENT_BIAS) linking world structure to population
  • tests/simulation/test_population.py: 26 tests covering counts, determinism, FK integrity, latent value ranges, trait completeness, motif bias properties, and observable field validity
  • Total: 358 tests passing

Milestone 4 — World Structure Layer ✓ (v0.3.0 in PR)

  • leadforge/structure/node_types.py: NodeType enum (9 categories); ROOT_ELIGIBLE, REQUIRES_PARENT, LEAF_ONLY constraint sets
  • leadforge/structure/graph.py: WorldGraph wrapping networkx.DiGraph with DAG validation, node-type legality, nondegeneracy, and outcome-reachability checks; JSON + GraphML export
  • leadforge/structure/motifs.py: MotifFamily frozen dataclass; all 5 v1 families (fit_dominant, intent_dominant, sales_execution_sensitive, demo_trial_mediated, buying_committee_friction); get_motif_family() lookup
  • leadforge/structure/rewiring.py: rewire() — optional-node dropping, edge-weight jitter, optional latent-confounder injection; fully deterministic given seed
  • leadforge/structure/sampler.py: sample_hidden_graph(seed, motif_family_name=None) — selects motif, applies rewiring, validates, retries up to 20 times
  • pyproject.toml: added networkx>=3.2 + numpy>=1.26; mypy override for networkx
  • 132 new tests; total 327 passing

Milestone 3 — Schema Layer ✓ (v0.2.0 in PR)

  • leadforge/core/ids.py: make_id(prefix, n) + ID_PREFIXES registry
  • leadforge/schema/entities.py: typed row dataclasses for all 9 tables (accounts, contacts, leads, touches, sessions, sales_activities, opportunities, customers, subscriptions) with DTYPE_MAP, to_dict(), empty_dataframe(), Parquet round-trip via schema/tables.py
  • leadforge/schema/relationships.py: FKConstraint, ALL_CONSTRAINTS (10 FK edges), validate_fk() helper raising FKViolationError
  • leadforge/schema/features.py: FeatureSpec frozen dataclass + LEAD_SNAPSHOT_FEATURES (29 features, one target)
  • leadforge/schema/dictionaries.py: feature_dictionary_df() + write_feature_dictionary()
  • leadforge/schema/tasks.py: SplitSpec, TaskManifest, CONVERTED_WITHIN_90_DAYS constant
  • pyproject.toml: added pandas≥2.0 + pyarrow≥14.0 as core deps; mypy overrides for both
  • 82 new tests; total 192 passing

Milestone 2 — Narrative Layer ✓ (v0.2.0 merged)

  • leadforge/narrative/spec.py: frozen dataclasses NarrativeSpec, CompanySpec, ProductSpec, MarketSpec, GtmMotionSpec, PersonaSpec, FunnelStageSpec — all with validated from_dict()
  • leadforge/narrative/dataset_card.py: render_dataset_card(world_spec) — Markdown card
  • leadforge/core/models.py: WorldSpec gets narrative: NarrativeSpec | None field
  • leadforge/api/generator.py: world_spec property; from_recipe() resolves narrative into WorldSpec
  • 51 new tests (spec validation, dataset card, Generator integration); total 110 passing

Milestone 1 — Canonical Config, Recipe & Model Objects ✓ (v0.2.0 merged)

  • leadforge/core/rng.py: RNGRoot with SHA-256-derived named substreams
  • leadforge/core/hashing.py: hash_config() — stable SHA-256 digest of GenerationConfig
  • leadforge/core/serialization.py: load_yaml, load_json, dump_json
  • leadforge/core/models.py: GenerationConfig with __post_init__ validation + package_version
  • leadforge/api/recipes.py: typed Recipe dataclass, from_dict, resolve_config (full precedence: explicit kwargs > override dict > recipe defaults > package defaults)
  • leadforge/api/generator.py: Generator.from_recipe() fully implemented (skeleton — no simulation); generate() stubs to v0.3.0
  • leadforge/recipes/b2b_saas_procurement_v1/narrative.yaml: company, product, market, GTM, personas, funnel stages
  • leadforge/recipes/b2b_saas_procurement_v1/difficulty_profiles.yaml: intro / intermediate / advanced signal-noise profiles
  • 39 new tests (rng, hashing, recipes, generator); total 59 passing

Milestone 0 — Project Foundation ✓ (v0.1.0)

  • pyproject.toml, README.md, LICENSE, .pre-commit-config.yaml
  • Full package skeleton with __init__.py stubs for all submodules
  • leadforge/core/: enums.py, exceptions.py, models.py, rng.py (stub), ids.py (stub)
  • leadforge/cli/: main.py + four commands (list-recipes implemented, others stubbed)
  • leadforge/recipes/: registry + b2b_saas_procurement_v1/recipe.yaml
  • .github/workflows/ci.yml: lint, typecheck, test matrix (3.11 + 3.12) with coverage upload
  • 20 tests passing; ruff + mypy clean