Optionally use papis publisher downloaders for paywalled DOI PDFs

[chenpeizhi]

Background

When a DOI's PDF is paywalled, download-ref today tries Semantic Scholar's open-access URL, then the arXiv preprint, then hands off to the manual browser-based Sci-Hub helper. papis maintains 25 per-publisher PDF downloaders (ACS, ScienceDirect, Springer, Taylor & Francis, IEEE, IOP, APS, …) behind a match(url) interface — community-maintained knowledge of how each publisher serves PDFs, which works when the user has institutional access. sci-brain already has a pattern for optional external tools with graceful absence: latexpand (preferred TeX flattener, Python fallback) and Playwright (Sci-Hub). This issue applies that pattern to papis. Motivation and full papis analysis: #30. Directly relevant to #19: chemistry papers live on exactly these publisher sites and rarely have arXiv preprints.

Objective

If papis is importable, try its publisher downloaders for a paywalled DOI after the arXiv-preprint fallback and before recommending Sci-Hub — with zero change in behavior when papis is absent.

Interface (Input → Output)

In: the existing DOI PDF-fallback loop in fetch_metadata.py, for DOIs still missing a PDF; papis optionally installed (import papis.downloaders succeeds).
Out: on success, .raw/doi/<safe>.pdf (same %PDF + %%EOF validation as fetch_pdf) and status ok doi:<doi> (papis:<downloader-name>). On papis absent or no matching downloader or download failure: the current miss line, unchanged. Preflight note in SKILL.md: "optional: pipx install papis enables publisher-PDF downloads".

Technical recommendations (non-binding)

papis.downloaders.get_matching_downloaders(url) on the https://doi.org/<doi> resolution target is one of the few papis APIs that doesn't assume a papis library. Guard the entire hook with try: import papis.downloaders except ImportError and a broad exception fallthrough — papis API churn must degrade to miss, never crash the batch. Cap this at ~30 lines.

Verification

All checks run offline via monkeypatching, so they work in CI without institutional access:

1. Behavioural positive: a unit test monkeypatches the papis hook's downloader-resolution seam to return a fake downloader yielding valid PDF bytes → the loop writes .raw/doi/<safe>.pdf passing the %%EOF check and prints the (papis:…) status — proving wiring, validation, and status format.
2. Negative control (absence): the same test module simulates ImportError for papis → the loop's output for the same manifest is byte-identical to today's behavior (assert the exact miss line and no new files) — proving zero-cost absence.
3. Negative control (bad payload): the fake downloader returns HTML bytes → the PDF validation rejects it, status miss, no file left behind.
4. One manual, recorded check in the PR (not CI): with real papis installed and institutional access, one supported-publisher DOI fetched end-to-end; paste the status line. If no access is available, state so — the monkeypatched tests carry the verification.

Dependencies

Part of #30. Partially addresses #19 (PDF side; the metadata side is the importer-chain issue). Independent of other children.

Out of scope

Making papis a required dependency; publishers papis lacks (Wiley, RSC, Nature — #19 stays open for those); replacing the Sci-Hub fallback.

🤖 Generated with Claude Code