🐛 fix(storage): reclaim ingress intents no upload can finalize

[gaborbernat]

A hosted PyPI upload stages a durable ingress intent before it stores anything, and every exit from admit_and_store that never reached the advance to admitted left that intent Pending for good. Nothing settled it afterwards: no production code wrote IntentPhase::Expired, prune_ingress_intents deleted only Admitted and Expired rows, and the crash-recovery sweep returned early on the missing upload row that a failed store leaves behind.

The cost decides the severity. A leaked intent holds its record and its bytes against the authority's 65,536 record and 64 GiB ceilings, so a publisher CI retrying on a bad link turns leaked intents into 503 ingress admission retention is full for every further upload of that project, with the disk space long since freed. A retry of the same upload still goes through, since it deduplicates onto the pending record and republishes. The leak accumulates instead, and until now the operator-run authority drain was the only thing that cleared it. A head of leaked intents also refilled the sweep's 256-row batch on every pass and starved the recoverable intents behind it, against the module's own claim that one unfinalizable intent never blocks the backlog.

The synchronous failures release their intent as they return. A replayed operation, an unreadable claim, an unavailable authority home, a quota block, and a store fault all leave nothing durable, so store_admitted returns them as Err and the caller reclaims the record and its capacity before the client reads the error. Only the request that staged the intent may release it. An admission that deduplicated onto an intent a concurrent identical resend staged carries fresh: false, so the record stays with the upload still storing its bytes.

A client that hangs up mid-store drops the request future outright, which leaves no code to run that release, so only a reaper reaches it. Expiring on elapsed time alone would also drop intents a slow home datacenter can still finalize, so the reaper acts on the sweep's evidence instead. finalize_one records a refusal on the one arm that proves no upload can ever finalize the intent here, locate finding no stored rows, and leaves the transient skips uncounted: a fence rejection and an ACL with no live write token both clear on their own. After MAX_INTENT_REFUSALS the sweep stops offering the intent, and expire_stale_intents may then advance it to Expired past its staging deadline, where pruning returns the record and the bytes. One counter serves both ends, keeping an unfinalizable head out of the batch and supplying the evidence expiry requires.

flowchart LR
    Stage[stage intent Pending] --> Store[store_upload]
    Store -- ok --> Admit[advance to Admitted]
    Store -- fault or quota block --> Release[release_intent now]
    Store -- client hangs up --> Sweep{sweep finds stored rows?}
    Sweep -- yes --> Admit
    Sweep -- no, 3 passes --> Expire[Expired past the deadline]
    Expire --> Prune[pruned, capacity returned]
    Release --> Prune
    classDef accent fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a;
    classDef good fill:#c9e7d4,stroke:#2f855a,color:#0f2f1e;
    classDef warn fill:#ffe3a3,stroke:#d29200,color:#3a2c00;
    class Stage,Store accent;
    class Admit,Prune good;
    class Sweep,Release,Expire warn;

Loading

list_pending_intents now takes the refusal ceiling its caller wants, so the operator drain walks every pending intent while the sweep skips the ones it has given up on. peryx-storage stays ecosystem-neutral: it counts refusals and enforces the deadline without interpreting either, and peryx-driver owns the two policy constants the sweep and the reaper share. This change owns only the Pending lifetime and leaves the pending-order index rewrite alone.

Closes #1526

[codspeed-hq]

Merging this PR will not alter performance

✅ 29 untouched benchmarks
⏩ 133 skipped benchmarks¹

---

_{Comparing fix/expire-unfinalizable-ingress-intents-1526 (e357574) with main (5f765fe)}

Footnotes

1. 133 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[read-the-docs-community]

Documentation build overview

📚 peryx | 🛠️ Build #34317273 | 📁 Comparing e357574 against latest (5f765fe)

  🔍 Preview build  

2 files changed

± core/availability/finalization/index.html
± ecosystems/pypi/reference/uploads/index.html