[Epic] Discovery Engine — modular, deterministic, multi-source route/endpoint discovery for hackbrowser

[badchars]

Vision

A modular, deterministic, multi-source discovery engine inside the hackbrowser worker that finds routes/pages and endpoints far beyond UI clicking — while staying model-agnostic: discovery is deterministic detector code, so swapping the LLM never changes what is discovered (only how findings are tested).

Everything runs inside the hackbrowser browser context (authenticated session + corporate-proxy-aware), so fetches of specs/bundles/sitemaps use the real logged-in, proxy-configured browser — not a separate HTTP client that would hit 401s / TLS walls.

Principles

• Deterministic detectors, not model initiative. Common→advanced structures are recognized by pluggable code, fast and stable. The LLM does judgment/testing, not discovery.
• Modular registry. New framework/spec = add one detector; core (context/registry/emitters/BFS/ingest) never changes.
• Two emitters. Pages/routes → BFS crawl queue (browser visits → capture → ingest). API endpoints → ingest directly (proxy-agents http_replay + test).
• Additive + guarded. Never regress existing crawls; reuse the executor (keeps ingest trigger-correlation); guard destructive nav.

Detector contract (core)

interface Detector {
  name: string
  kind: "nav" | "spec" | "js-route" | "api-call"
  applies(ctx): Promise<boolean>      // cheap framework/relevance check
  detect(ctx): Promise<{ pages: string[]; endpoints: Endpoint[]; confidence: number }>
}

Registry = Detector[]; applicable ones run; results routed by the emitters.

Phases (sub-issues)

• Phase 1 — core: detector contract + registry + emitters + sitemap/robots detectors.
• Phase 2 — nav-sweep: imperative nav traversal (deterministic checklist + existing LLM judgment). Addresses hackbrowser: shallow crawl on button/router-nav SPAs — collectDOMLinks only harvests <a href> #120.
• Phase 3 — spec: OpenAPI/Swagger + GraphQL spec discovery (browser-fetch + parse → endpoints).
• Phase 4 — js-route/api-call: framework fingerprint + per-framework route-table + fetch/axios URL extraction from loaded bundles.

Delivery

Each detector is its own module (discovery/detectors/*.ts), independently testable (input DOM/bundle → expected pages/endpoints). Implemented in small, logical, granular commits — one behavior per commit.

Honest limits

• js-route/api-call extraction is fragile under minification (source-map-aware; confidence-scored; LLM fallback).
• Detectors need maintenance as frameworks change.
• Closed shadow DOM / canvas / cross-origin iframes remain out of scope.

[badchars]

Sub-issues (phased)

• Phase 1 — detector core + sitemap/robots: Discovery Engine — Phase 1: detector core (contract + registry + emitters) + sitemap/robots #126
• Phase 2 — nav-sweep (addresses hackbrowser: shallow crawl on button/router-nav SPAs — collectDOMLinks only harvests <a href> #120): Discovery Engine — Phase 2: nav-sweep (deterministic imperative-nav traversal) #127
• Phase 3 — OpenAPI/Swagger + GraphQL spec: Discovery Engine — Phase 3: OpenAPI/Swagger + GraphQL spec discovery #128
• Phase 4 — JS-bundle route + API-call extraction: Discovery Engine — Phase 4: JS-bundle route + API-call extraction (framework fingerprint) #129

Order: 1 → 2 → 3 → 4 (foundation first; each phase ships independently). Phase 2 is the concrete fix for the observed shallow button/router-nav crawl; Phase 3 is the highest endpoint-coverage win.