hackbrowser worker crashes (exit 1) mid-crawl when a target closes — unguarded request.allHeaders()

[badchars]

Summary

The hackbrowser crawl worker can crash with exit code 1 in the middle of a crawl. The request-capture handler awaits request.allHeaders() without a guard; when the page/context/browser closes while requests are still in flight, Playwright rejects with Target page, context or browser has been closed. Because this happens inside a fire-and-forget page.on("request", ...) handler, the rejection is unhandled and takes down the entire worker process — so the crawl ends after exploring only a fraction of the target.

Environment

• Version: 1.1.16
• Mode: full-auto-headed
• Runtime: Bun 1.3.14 (macOS arm64)

Symptom

The TUI shows the crawl as Failed:

hackbrowser worker exited unexpectedly (code 1): ...
      const headers = await request.allHeaders();
                            ^
error: allHeaders: Target page, context or browser has been closed
      at .../cyberstrike/bin/hackbrowser-worker.js
Bun v1.3.14 (macOS arm64)

Several endpoints were captured before the crash — i.e. the crawl was still in progress (not finished) when the worker died.

Root cause

packages/hackbrowser/src/agent.ts, the request interceptor:

page.on("request", async (request) => {
  ...
  const method = request.method()
  const headers = await request.allHeaders()   // <-- unguarded; rejects if target closed
  const postData = request.postData() ?? null
  ...
})

In headed mode a page/context can close mid-crawl (a popup/tab closing, or a navigation) while requests are still in flight. allHeaders() then rejects, and because the handler is async and fire-and-forget, the rejection is unhandled and crashes the worker (exit 1). The response branch further down is already wrapped in try/catch; this call was not. The sibling page.on("response", ...) login detector is sync-only and unaffected.

Proposed fix

Guard the allHeaders() call and drop the capture on a closed target instead of crashing:

const method = request.method()
let headers: Record<string, string>
try {
  headers = await request.allHeaders()
} catch {
  // Target/page/browser closed mid-flight — drop this capture rather than
  // letting an unhandled rejection crash the worker (exit 1).
  return
}
const postData = request.postData() ?? null

Optional hardening: add a process-level unhandledRejection handler in the worker as a safety net, so any other stray closed-target rejection degrades gracefully instead of killing the crawl.

Impact

Crawl terminates early — only part of the application is explored/captured. Most likely in headed mode and on apps that open/close popups or navigate during the crawl.

[Serhatcck]

Fixed on `dev` — commit `3f7e08258`.

Guarded the `request.allHeaders()` call in the `page.on("request", async …)` capture handler with a try/catch; on a closed target the capture is dropped (a request whose headers never arrived isn't a usable capture) instead of letting the unhandled rejection crash the worker. A debug line records the in-flight URL that was dropped, for diagnosis.

Scope-checked: `allHeaders()` was the SOLE unguarded await in the handler — the UI-context and response branches below are already try/caught. The early return is consistent with the handler's existing skip returns (out-of-scope / unparseable URL), so `pendingUIContext` handling is unchanged.

Verified end-to-end: a real crawl against a page firing continuous background requests now completes cleanly (pages explored, endpoint captured) while the guard caught 143 real "target closed" rejections mid-crawl that would each previously have crashed the worker (exit 1). Adversarial review found no new failure mode — the success path is byte-for-byte unchanged; only the previously-crashing closed-target path now degrades gracefully. 99/99 tests pass.