Skip to content

REPRODUCIBILITY.md

Latest commit

 

History

History
296 lines (220 loc) · 10.1 KB

REPRODUCIBILITY.md

File metadata and controls

296 lines (220 loc) · 10.1 KB
Raw

Reproducibility

Artefact-level reproduction guide for Owner-Harm: A Missing Threat Model for AI Agent Safety (Zhang and Jiang, 2026; arXiv:2604.18658). Every table and headline number in §4 of the paper has a corresponding entry point below.

Reproduction matrix

Paper claim Entry point LLM API Wall-clock Hardware
§1 — LSVJ-S compile-time gate (138 tests pass, 1 skipped) pytest tests/lsvj/ none < 10 s any
§4 Owner-centric v3 — gate alone (75.3% TPR / 3.3% FPR) python scripts/full_benchmark_eval.py none (deterministic) ~ 30 s any
§4 Owner-centric v3 — full L1–L4 (85.3% TPR / 13.3% FPR) python scripts/full_benchmark_eval.py none (deterministic) ~ 30 s any
§4 Owner-centric v3 — gate ∩ verifier overlap (Hijacking 11/30/15/4) python scripts/eval_d2_verifier.py none ~ 10 s any
§4.2 AgentDojo — isolation L1+L4 deterministic (3.7% TPR / 97.9% util / 2.1% FPR) cd benchmarks/agentdojo_adapter && uv run --project ../agentdojo python run_eval.py none (ground-truth pipeline) ~ 1–3 min any
§4.3 AgentDojo — deployment L1 (95.9% / 75.0%) bash benchmarks/agentdojo_adapter/launch-baseline-l1-rerun.sh GLM-4.6 ~ 5 h 1 process, no GPU
§4.3 AgentDojo — deployment L1+L3+L4 deepseek (96.3% / 75.0%) bash benchmarks/agentdojo_adapter/launch-l3-deepseek-repro.sh DeepSeek v4-pro + GLM-4.6 ~ 5 h same
§4.6 AgentHarm full L1+L2+L3 (97.7% TPR / 2.3% FPR, N=176+176) OPENAI_API_KEY=$DEEPSEEK_API_KEY NOUS_BASE_URL=https://api.deepseek.com/v1 NOUS_SEMANTIC_MODEL=deepseek-chat python scripts/run_agentharm_threelayer_v2.py DeepSeek-chat ~ 36 min same

Setup

git clone https://github.com/dario-github/nous.git
cd nous
python3 -m venv .venv && source .venv/bin/activate

pip install -e ".[lsvj,dev]"
# Optional, only needed for KG-bound and DB-bound tests:
pip install -e ".[cozo]"

Python 3.11 or 3.12 is required.

API keys — store in a project-local .env (already in .gitignore):

ZAI_API_KEY=...           # GLM-4.6 (AgentDojo agent backbone, deployment runs)
DEEPSEEK_API_KEY=...      # DeepSeek-V4-pro (L3 semantic gate, isolation runs)

The launch scripts source .env automatically.

AgentDojo benchmark

The AgentDojo benchmark is a separate, public, MIT-licensed package and is not vendored in this repository:

# adjacent to nous/, so benchmarks/agentdojo_adapter/ can find it via ../agentdojo
git clone https://github.com/ethz-spylab/agentdojo.git
cd agentdojo && pip install -e . && cd -

Known issues

  • cozo_embedded wheels on Ubuntu 24.04 occasionally segfault under Python 3.11 / 3.12; the repository’s CI pins to a tested subset (tests/lsvj/, tests/test_scallop_sidecar.py, tests/test_gateway_hook.py). On macOS and on Ubuntu 22.04 the full pytest tests/ suite runs cleanly.
  • memory/entities/ KG seed is host-private. Tests that walk the personal KG are decorated with pytest.mark.skipif(not KG_AVAILABLE) and are silent on a clean clone. The numbers in the paper do not depend on those tests passing.

§4 — Owner-centric v3 (full L1–L4)

The held-out set data/owner_harm_heldout_v3.json (300 harmful + 150 benign) ships with the repository. The full L1–L4 reproduction is deterministic (no LLM call) because L3 verdicts on this slice are checkpointed in the evidence table; only L1 and L4 are recomputed.

python scripts/full_benchmark_eval.py

Expected output (matches paper Table 5; reproduced 2026-05-02):

OVERALL (300H + 150B)
Gate alone:      TPR 226/300 = 75.3%  FPR 5/150 = 3.3%
Verifier alone:  TPR 105/300 = 35.0%
Combined:        TPR 256/300 = 85.3%  FPR 20/150 = 13.3%

HIJACKING OVERLAP ANALYSIS (60H, benign 30B)
Gate alone:     26/60 = 43.3%
Verifier alone: 45/60 = 75.0%
Combined:       56/60 = 93.3%
  Gate-only:     11
  Verifier-only: 30
  both:          15
  neither:       4

The Gate alone row reproduces the L1–L3 baseline (75.3% TPR / 3.3% FPR); the Combined row reproduces the full L1–L4 stack (85.3% TPR / 13.3% FPR, CI [8.8%, 19.7%]). The L4 verifier raises TPR by +10 pp at the cost of +10 pp benign FPR concentrated in C2 Infrastructure Exposure (60% per-category combined FPR) and C4 Inner Circle Leak (35.7%); zero benign FP on Hijacking. Per-category Wilson 95% CIs match paper §4.4.

§4.2 — AgentDojo isolation (3.7% / 97.9% / 2.1% — primary deterministic)

This is the L1 + L4 deterministic stack on AgentDojo's 27 forced-injection ground-truth tasks (banking + slack + travel + workspace; n=97 user tasks for utility). No L3 in this row — purely Datalog rules + post-gate verifier. Reproduces in 1–3 minutes with no API key:

cd benchmarks/agentdojo_adapter
uv run --project ../agentdojo python run_eval.py

Expected output:

安全得分:  1/27 = 3.7%
效用保留:  95/97 = 97.9%
误拦截率:  2.1%

按 Owner-Harm 类别:
  credential_leak     : 0/3
  financial_harm      : 0/10
  operational_harm    : 1/4 = 25.0%
  privacy_breach      : 0/6
  reputational_harm   : 0/3
  other               : 0/1

This is paper §4.2 Table 1 row "L1 + L4 (deterministic) — 3.7% (1/27)". Wilson 95% CI [0.7%, 18.3%] (n=27 is small; the wide CI is reported transparently in the paper).

§4.3 — AgentDojo deployment L1+L3+L4 deepseek (96.3% / 75.0%)

This is the deployment-mode (real GLM-4.6 agent + injection attack via important_instructions attacker) with L1 Datalog + L3 deepseek-v4-pro semantic gate + L4 verifier on the full 629-tuple suite (banking + slack + travel + workspace). Public substitute for the original qwen-turbo-via-internal-relay configuration (paper §Limitations Erratum).

export DEEPSEEK_API_KEY=...   # or set in .env
bash benchmarks/agentdojo_adapter/launch-l3-deepseek-repro.sh

The script launches a screen session named nous-l3-deepseek, takes ~ 5 h, and writes the summary to results-l3-repro/l1_3_4_deepseek-glm-fullmatrix.json.

Expected aggregate (replayed 2026-05-01):

banking   security 136/144 (94.4%)  utility 109/144 (75.7%)
slack     security  96/105 (91.4%)  utility  71/105 (67.6%)
travel    security 140/140 (100.0%) utility 117/140 (83.6%)
workspace security 234/240 (97.5%)  utility 175/240 (72.9%)
total     security 606/629 (96.3%)  utility 472/629 (75.0%)

This is the L1+L3+L4 deepseek row of paper Table 3 (Deployment-Mode Ablation).

§4.3 — AgentDojo deployment L1 baseline (95.9% / 75.0%)

Same 4-suite × 629-task matrix as §4.3 above, but L1 Datalog only (no L3 semantic). Three modes ship in this script for the full ablation in paper Table 3:

export ZAI_API_KEY=...        # or set in .env
bash benchmarks/agentdojo_adapter/launch-baseline-l1-rerun.sh

The script launches three screen sessions in parallel:

  1. nous-baseline — GLM-4.6 agent only, no Nous gate (control: 95.2% / 79.0%)
  2. nous-l1 — agent + L1 Datalog (96.0% / 75.0%)
  3. nous-l1_4 — agent + L1 + L4 (95.9% / 75.0%, paper headline)

Each takes ~ 5 h. Results land in benchmarks/agentdojo_adapter/results-fullmatrix/.

§4.6 — AgentHarm full L1+L2+L3 (97.7% TPR / 2.3% FPR)

Reproduces the L1 + L2 + L3 stack on the AgentHarm \texttt{test_public} split (176 harmful + 176 benign), reproduced 2026-05-02 with locked \texttt{deepseek-chat} L3. Raw scenarios live in docs/agentharm-raw-scenarios.json.

export OPENAI_API_KEY=$DEEPSEEK_API_KEY
export NOUS_BASE_URL=https://api.deepseek.com/v1
export NOUS_SEMANTIC_MODEL=deepseek-chat
python scripts/run_agentharm_threelayer_v2.py

Expected output (paper §4.6 AgentHarm row, full N=176+176):

[3/3] Running L1+L2+L3 (Semantic Gate with deepseek-chat)...
  TPR=97.7% FPR=2.3% hard_TPR=97.7% hard_FPR=2.3% (~36 min)

  L1 → L1+L2+L3 deltas:
    TPR  59.1% → 97.7%   (+38.6 pp)
    FPR  52.3% →  2.3%   (-50.0 pp)

L3 contributes the entire +38.6 pp TPR. Per-category 7/8 reach 100% TPR; Cybercrime 90% (9/10), Drugs 90% (18/20). FPR concentrated in Disinformation (20% benign FP, 1/5).

§4 — Layer complementarity (gate ∩ verifier overlap on Hijacking)

python scripts/eval_d2_verifier.py

Expected:

Hijacking 60 H: gate=11 only, verifier=30 only, both=15, neither=4

This is the structural-boundary disjointness claim cited in §4 and §5.

Variance and seed control

  • Owner-centric v3 (deterministic L1 + L4) — no variance.
  • AgentDojo isolation — L3 majority vote k=5 at temperature 0.0; variance across runs is < 1 pp on the aggregate.
  • AgentDojo deployment — GLM-4.6 agent at temperature 0.0, k=1; per-task LLM behaviour is reasonably stable but the 629-task aggregate may move ± 0.5 pp between runs.
  • AgentHarm — temperature 0.0, repeat=1 per SemanticGateConfig default; deterministic across replays.

Compute and cost estimates

  • Deterministic (Owner-centric v3, LSVJ-S, layer overlap) — < 1 min, free.
  • AgentHarm L1 + L2 + L3 — ~ 1 h, ~ US$ 5 in DeepSeek tokens.
  • AgentDojo isolation L1 + L3 + L4 — ~ 5 h, ~ US$ 30 in DeepSeek tokens.
  • AgentDojo deployment baseline + L1 + L1 + L4 (3 modes parallel) — ~ 5 h, ~ US$ 0 if using a GLM coding-plan subscription, otherwise ~ US$ 50 metered.

No GPU is required for any reproduction step; all LLM calls go to remote APIs.

Versioned artefacts

  • Code: this repository, commit recorded in paper/main-neurips-2026.tex Erratum.
  • Owner-centric v3 dataset: data/owner_harm_heldout_v3.json (sha256 frozen at submission; see paper/main-neurips-2026.tex §3.3).
  • R-Judge personal-agent slice: benchmarks/rjudge_sample/ (24 records, sha256 frozen).
  • AgentDojo: pinned to upstream commit agentdojo @ v1.x (see benchmarks/agentdojo_adapter/run_eval_adaptive_llm.py setup banner printed at run start).

Filing reproduction issues

If a step in this guide produces a number outside the variance budget above, please open an issue at https://github.com/dario-github/nous/issues with:

  • the entry-point command run,
  • the OS / Python / pip-freeze output,
  • the produced summary JSON (under results-*/),
  • the model used at L3 (DeepSeek vs. other) and the temperature.