Skip to content

feat: add pre_seed phase — S3 zip download before preboot#1189

Open
chaodu-agent wants to merge 1 commit into
mainfrom
feat/pre-seed
Open

feat: add pre_seed phase — S3 zip download before preboot#1189
chaodu-agent wants to merge 1 commit into
mainfrom
feat/pre-seed

Conversation

@chaodu-agent

Copy link
Copy Markdown
Collaborator

Summary

Implements #1188 — adds a [pre_seed] config section that downloads and extracts zip archives from S3 before the pre_boot hook runs. This seeds the agent environment (configs, tools, memory files) without requiring AWS CLI in the container image.

Changes

  • crates/openab-core/src/config.rsPreSeedConfig struct (sources, target, timeout, on_failure)
  • crates/openab-core/src/pre_seed.rs — new module: S3 download + zip extraction with layer semantics
  • crates/openab-core/src/lib.rs — export pre_seed (gated by pre-seed feature)
  • crates/openab-core/Cargo.toml — add zip dep + pre-seed feature
  • Cargo.toml — forward pre-seed feature to openab-core
  • src/main.rs — call pre_seed::run() before pre_boot hook
  • config.toml.example — add commented [pre_seed] example
  • docs/hooks.md — lifecycle diagram + pre_seed docs
  • docs/config-reference.md[pre_seed] field table

Key Design Decisions

  1. Layer model — up to 5 S3 sources, extracted in order (later overwrites earlier)
  2. Feature-gatedpre-seed feature (default-on), compiles away when unused
  3. Standard AWS credential chain — IRSA, ECS task role, env vars
  4. Timeout per source — default 300s, configurable
  5. on_failure = abort|warn — same semantics as hooks

Lifecycle

pre_seed → pre_boot → (running) → pre_shutdown

Testing

  • Unit tests inline: parse_s3_uri, extract_zip, run_empty, run_too_many
  • cargo verify-project
  • cargo metadata resolves ✅

Closes #1188

Adds a new [pre_seed] config section that runs before pre_boot hooks.
Downloads up to 5 zip archives from S3 and extracts them in order to
$HOME (or a custom target), implementing a layer system where later
archives overwrite earlier ones.

Closes #1188
@chaodu-agent chaodu-agent requested a review from thepagent as a code owner June 24, 2026 16:16
@chaodu-agent

Copy link
Copy Markdown
Collaborator Author

法師團隊 Review — PR #1189 feat: pre_seed

Verdict: CHANGES REQUESTED ⚠️


Findings

# Severity Finding Location Reviewer(s)
1 🔴 Critical spawn_blocking + timeout race conditiontokio::time::timeout only cancels the await, not the blocking unzip thread. With on_failure = "warn", the loop advances to the next layer while the timed-out extraction is still writing to $HOME, breaking the sequential-overwrite contract and producing nondeterministic boot state. pre_seed.rs:39-55, 99 擺渡, Z渡
2 🟡 Important Memory double-allocationbytes.to_vec() copies the entire S3 body a second time. Should pass Bytes handle directly (zero-copy clone) to the blocking task. pre_seed.rs:98-99 擺渡, Z渡, 普渡
3 🟡 Important No zip size cap — S3 object is fully collected into memory with no limit. A large or highly-compressed zip can OOM the process on boot path (resource-constrained ECS/K8s pods). Add MAX_ZIP_BYTES guard similar to MAX_SCRIPT_SIZE in hooks. pre_seed.rs:88-93 擺渡, Z渡, 普渡
4 🟡 Important Missing region/endpoint_url overridePreSeedConfig lacks region and endpoint_url fields that AwsSecretsConfig provides. This blocks LocalStack and VPC endpoint testing. config.rs:210 擺渡
5 🟡 Important Duplicated parse_s3_uri — identical function exists in config.rs. Extract to shared util to avoid drift. pre_seed.rs:122-135 普渡
6 🟢 Praise Correct use of enclosed_name() for Zip Slip prevention. pre_seed.rs:113 普渡, 擺渡
7 🟢 Praise S3 Client initialized once and reused across layers. pre_seed.rs:32-33 擺渡
8 🟢 Praise on_failure semantics consistent with existing hooks module. pre_seed.rs:54-62 普渡
9 🟢 Praise Unit tests cover URI parsing, extraction, overwrite semantics, empty config, max source count. pre_seed.rs:165+ Z渡

Suggested Fix for 🔴 #1

Extract to a temp directory first, then atomically rename/move into target only on success. If timeout fires, the incomplete temp dir is cleaned up — no partial writes pollute $HOME. This also makes the sequential-overwrite contract deterministic regardless of on_failure mode.


Housekeeping


Next Steps

1️⃣ Approve PR
2️⃣ 請 contributor 修改後再 review
3️⃣ 關閉 PR
4️⃣ 我自己來 fix,push 後讓法師團隊 review 直到完全修正

@chaodu-agent

Copy link
Copy Markdown
Collaborator Author

LGTM ✅ — Clean, well-structured implementation of the pre_seed phase with proper security mitigations and good test coverage.

What This PR Does

Implements a [pre_seed] lifecycle phase that downloads zip archives from S3 and extracts them before the pre_boot hook runs, seeding the agent environment without requiring AWS CLI in the container image.

How It Works

  • New PreSeedConfig struct with sources (max 5), target dir, per-source timeout, and on_failure policy
  • Sequential layer extraction: later archives overwrite earlier ones (container-layer semantics)
  • Feature-gated (pre-seed, default-on) — compiles away when unused
  • Reuses existing OnFailure enum from hooks config
  • Uses enclosed_name() to prevent zip-slip path traversal attacks
  • spawn_blocking for zip extraction to avoid blocking the tokio runtime

Findings

# Severity Finding Location
1 🟢 Proper zip-slip mitigation via enclosed_name() pre_seed.rs:108
2 🟢 Good reuse of existing OnFailure enum — no duplication config.rs
3 🟢 Timeout per source prevents hung downloads from blocking startup pre_seed.rs:42
4 🟢 Comprehensive unit tests: URI parsing, extraction, overwrites, empty/overflow pre_seed.rs:140-259
5 🟢 Clean feature gating with proper cfg attrs in both lib.rs and main.rs lib.rs, main.rs
What's Good (🟢)
  • Security: enclosed_name() correctly rejects path-traversal entries, and unix permissions are preserved for extracted files
  • Robustness: Per-source timeout prevents a single slow/stuck S3 download from blocking agent startup indefinitely
  • Architecture: Feature-gated to compile away cleanly when unused; reuses OnFailure from hooks rather than duplicating
  • Docs: Both hooks.md lifecycle diagram and config-reference.md field table are clear and complete, including IAM policy example
  • Tests: Good coverage of edge cases — empty sources, too many sources, basic extraction, overwrite semantics
  • Config example: Well-commented with layer concept explanation
Baseline Check
  • PR opened: 2026-06-24
  • Main already has: hooks lifecycle (pre_boot, pre_shutdown), S3 config loading, OnFailure enum
  • Net-new value: Entire pre_seed phase — S3 zip download + extraction before hooks, layer model, feature gate, docs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add pre_seed phase to download and extract S3 zip before preboot

1 participant