Skip to content

Commit a768c83

Browse files
committed
refactor(examples): rewrite policy demo as Codex-default loop
Re-shape examples/agent-driven-policy-management/ to be a single, clean end-to-end demonstration of the agent-driven policy loop. A Codex agent inside an OpenShell sandbox attempts a GitHub Contents API write, hits a structured 403 from the L7 proxy, reads the policy_advisor skill, drafts a narrow addRule proposal via http://policy.local/v1/proposals, the host auto-approves, the sandbox hot-reloads policy, and the agent's retry succeeds. Whole loop runs in roughly two minutes. Demo cleanup: - Drop .env file ceremony. Defaults resolve from gh: owner via 'gh api user --jq .login', repo defaults to 'openshell-policy-demo', token from gh auth token / GITHUB_TOKEN / GH_TOKEN. With gh auth login and codex login already done, 'bash demo.sh' Just Works. - Codex-specific. Bootstraps ~/.codex/auth.json from credentials injected by the OpenShell provider, runs codex exec --sandbox danger-full-access (OpenShell is the actual security boundary; bwrap nesting cannot create user namespaces inside the sandbox container). - Tighter narrative output: a single 'Preflight' step, a run summary banner before launch, an inline narration of what's happening inside the sandbox while we poll for the proposal (including the literal structured 403 body the agent acts on), and an OCSF trace at the end filtered to the three events that tell the story (DENY, RELOAD, ALLOW). - Replace Python heredoc templating with sed; uploads use the single-flag pattern (--upload "${PAYLOAD_DIR}:/sandbox") with files referenced at the basename-prefixed path that #952 / #1028 established. - README documents the trust model honestly: structured rule is the contract, agent rationale is a hint, prover validation badge in progress per RFC 0001. Move the deterministic no-LLM regression harness out of examples/ into e2e/policy-advisor/ -- it was a parallel demo, not an example. Same loop without the LLM, useful for iterating on the proxy and policy.local API.
1 parent 61e95c6 commit a768c83

10 files changed

Lines changed: 473 additions & 301 deletions

File tree

e2e/policy-advisor/README.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
2+
<!-- SPDX-License-Identifier: Apache-2.0 -->
3+
4+
# Policy Advisor end-to-end test
5+
6+
Deterministic, no-LLM exercise of the agent-driven policy loop:
7+
8+
1. Start a sandbox with a read-only GitHub L7 policy.
9+
2. From inside the sandbox, attempt a GitHub contents PUT and assert OpenShell
10+
returns a structured `policy_denied` 403.
11+
3. Submit a narrow `addRule` proposal through `http://policy.local/v1/proposals`.
12+
4. Approve the draft from the host and retry until the write succeeds.
13+
14+
This proves the proxy, the structured deny body, the `policy.local` HTTP API,
15+
the gateway proposal path, and the hot-reload of approved rules — without
16+
involving an LLM. The user-facing demo (`examples/agent-driven-policy-management/`)
17+
runs the same loop with Codex driving from inside the sandbox.
18+
19+
## Run it
20+
21+
```bash
22+
DEMO_GITHUB_OWNER=<your-handle> \
23+
DEMO_GITHUB_REPO=openshell-policy-demo \
24+
bash e2e/policy-advisor/test.sh
25+
```
26+
27+
Requires an active OpenShell gateway (`openshell gateway start`) and a GitHub
28+
token with contents write on the repository (auto-resolved from `gh auth token`,
29+
`GITHUB_TOKEN`, or `GH_TOKEN`).

examples/agent-driven-policy-management/validation/policy.template.yaml renamed to e2e/policy-advisor/policy.template.yaml

File renamed without changes.

examples/agent-driven-policy-management/validation/runner.sh renamed to e2e/policy-advisor/sandbox-runner.sh

File renamed without changes.

examples/agent-driven-policy-management/validation/validation.sh renamed to e2e/policy-advisor/test.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66
set -euo pipefail
77

88
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
9-
REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
9+
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
1010
POLICY_TEMPLATE="${SCRIPT_DIR}/policy.template.yaml"
11-
RUNNER_SOURCE="${SCRIPT_DIR}/runner.sh"
11+
RUNNER_SOURCE="${SCRIPT_DIR}/sandbox-runner.sh"
1212

1313
if [[ -z "${OPENSHELL_BIN:-}" ]]; then
1414
if [[ -x "${REPO_ROOT}/target/debug/openshell" ]]; then

examples/agent-driven-policy-management/.env.example

Lines changed: 0 additions & 29 deletions
This file was deleted.

examples/agent-driven-policy-management/README.md

Lines changed: 71 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -3,90 +3,85 @@
33

44
# Agent-Driven Policy Management Demo
55

6-
Run the first policy-advisor MVP loop with a real agent:
6+
Run the full agent-driven policy loop end-to-end:
77

8-
1. Use the active OpenShell gateway.
9-
2. Create a GitHub provider from a host token.
10-
3. Start a sandbox with your agent command and an uploaded task file.
11-
4. Let the agent hit an OpenShell `policy_denied` response.
12-
5. Let the agent read `/etc/openshell/skills/policy_advisor.md` and submit a
13-
narrow proposal through `http://policy.local/v1/proposals`.
14-
6. Approve the draft rule from outside the sandbox.
15-
7. Let the agent retry and confirm the GitHub write succeeds.
8+
1. A Codex agent inside an OpenShell sandbox tries to write a markdown file to
9+
GitHub via the Contents API.
10+
2. OpenShell denies the request with a structured `policy_denied` 403 because
11+
the initial policy only allows read-only access to `api.github.com`.
12+
3. The agent reads `/etc/openshell/skills/policy_advisor.md`, drafts the
13+
narrowest rule needed, and submits it to `http://policy.local/v1/proposals`.
14+
4. You approve the proposal from the host with one keystroke.
15+
5. The sandbox hot-reloads the merged policy and the agent's retry succeeds.
1616

17-
The shell script is agent-agnostic. It does not know how to sign in to a
18-
specific coding agent. Pass the provider names and sandbox command for the
19-
agent you want to run.
17+
The whole loop usually finishes in under two minutes.
2018

2119
## Prerequisites
2220

23-
- An active OpenShell gateway that includes the current sandbox supervisor
24-
build.
25-
- `curl` and `jq` on the host machine.
26-
- The GitHub CLI (`gh`) if you want to create the scratch repo with the command
27-
below.
28-
- A disposable or demo-safe GitHub repository.
29-
- A GitHub token with contents write permission for that repository.
30-
- An agent provider and policy that let your chosen agent run inside the
31-
sandbox.
32-
33-
## Create A Scratch Repo
34-
35-
Use a private scratch repository with an initial README. The README matters
36-
because GitHub does not create the default branch until the first commit exists.
37-
38-
```bash
39-
gh repo create zredlined/openshell-policy-demo \
40-
--private \
41-
--add-readme \
42-
--description "OpenShell policy advisor demo scratch repo"
43-
```
44-
45-
The demo never creates repositories and refuses to overwrite an existing demo
46-
file. Each default run writes a new timestamped file under
47-
`openshell-policy-advisor-demo/`.
48-
49-
## Quick Start
21+
- An active OpenShell gateway (`openshell gateway start`).
22+
- `gh auth login` (or a `GITHUB_TOKEN` env var with contents-write on a
23+
scratch repo).
24+
- `codex login` on the host.
25+
- A scratch GitHub repository with at least one commit on the default branch.
26+
If you don't have one yet:
5027

51-
The included `policy.template.yaml` only defines the GitHub API target for the
52-
policy-management loop. Use `DEMO_POLICY_FILE` to point at a policy that also
53-
allows your chosen agent to reach its model/provider endpoints.
28+
```shell
29+
gh repo create "$(gh api user --jq .login)/openshell-policy-demo" \
30+
--private --add-readme \
31+
--description "OpenShell policy advisor demo scratch repo"
32+
```
5433

55-
```bash
56-
cp examples/agent-driven-policy-management/.env.example .env
57-
$EDITOR .env
58-
59-
set -a
60-
source .env
61-
set +a
34+
## Run it
6235

36+
```shell
6337
bash examples/agent-driven-policy-management/demo.sh
6438
```
6539

66-
The host script only orchestrates sandbox lifecycle and developer approval. The
67-
policy proposal is authored by the agent inside the sandbox from the installed
68-
skill, structured denial response, and `policy.local` API.
69-
70-
The demo writes one markdown file under:
71-
72-
```text
73-
openshell-policy-advisor-demo/<run-id>.md
74-
```
75-
76-
Use a scratch repository or a demo branch if you do not want this file in a
77-
production repository.
78-
79-
The deterministic non-model validation flow lives in
80-
`examples/agent-driven-policy-management/validation/validation.sh`.
81-
82-
## Options
83-
84-
```bash
85-
export OPENSHELL_BIN=/path/to/openshell
86-
export DEMO_BRANCH=main
87-
export DEMO_RUN_ID="$(date +%Y%m%d-%H%M%S)"
88-
export DEMO_FILE_DIR=openshell-policy-advisor-demo
89-
export DEMO_KEEP_SANDBOX=0
90-
export DEMO_APPROVAL_TIMEOUT_SECS=240
91-
export DEMO_AGENT_PROVIDERS="agent-provider-a agent-provider-b"
92-
```
40+
That's the whole thing. The demo resolves your GitHub handle from `gh`, picks
41+
`openshell-policy-demo` as the repo, and writes one timestamped markdown file
42+
under `openshell-policy-advisor-demo/` per run.
43+
44+
## Overrides (all optional)
45+
46+
| Env var | Default |
47+
|---|---|
48+
| `DEMO_GITHUB_OWNER` | `gh api user --jq .login` |
49+
| `DEMO_GITHUB_REPO` | `openshell-policy-demo` |
50+
| `DEMO_BRANCH` | `main` |
51+
| `DEMO_RUN_ID` | timestamp |
52+
| `DEMO_GITHUB_TOKEN` | falls back to `GITHUB_TOKEN`, `GH_TOKEN`, or `gh auth token` |
53+
| `DEMO_KEEP_SANDBOX` | `0` (set `1` to inspect the sandbox after the demo) |
54+
| `DEMO_APPROVAL_TIMEOUT_SECS` | `240` |
55+
| `OPENSHELL_BIN` | `target/debug/openshell` if present, else `openshell` on `PATH` |
56+
57+
## What the agent sees
58+
59+
`policy.template.yaml` is the initial restrictive policy: a read-only L7 REST
60+
rule for `api.github.com` plus the binary set Codex needs. The agent has to
61+
ask for the additional `PUT /repos/.../contents/...` write itself — that's the
62+
proposal you approve.
63+
64+
## What gets approved (trust model)
65+
66+
Every proposal lands in the gateway as a `PolicyChunk` — a structured object
67+
with three parts, each with a different trust level:
68+
69+
| Field | Source | Trust |
70+
|---|---|---|
71+
| `proposed_rule` (host, port, method, path, binary) | agent, schema-validated by the gateway | structured contract — *this* is what you're approving |
72+
| `rationale` (free-form prose) | agent | hint only — a compromised agent can lie here |
73+
| `validation_result` (prover output) | gateway-side prover | trust signal — but this surface is in progress (see [RFC 0001](../../rfc/0001-agent-driven-policy-management.md)) |
74+
75+
The MVP today shows the structured rule plus the agent's rationale in
76+
`openshell rule get` and the TUI inbox panel. The demo's `openshell rule
77+
approve-all` auto-approves to keep the loop short — in a real session a
78+
developer reviews the structured grant before pressing `a`. Prover-backed
79+
validation badges, computed reachability deltas, and a richer "this is what
80+
the rule actually permits" summary are the next phase. For now, **always
81+
approve based on the structured rule, not the agent's rationale.**
82+
83+
## Going further
84+
85+
`e2e/policy-advisor/test.sh` runs the same loop deterministically without an
86+
LLM (curl + the `policy.local` API directly). Use it to validate the proxy and
87+
proposal pipeline when iterating on the sandbox or gateway code.
Lines changed: 38 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,45 +1,48 @@
11
<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
22
<!-- SPDX-License-Identifier: Apache-2.0 -->
33

4-
# Agent Policy Management Demo Task
4+
You are running inside an OpenShell sandbox. Your job is to write one
5+
markdown file to GitHub via the GitHub Contents API.
56

6-
You are inside an OpenShell sandbox. The user asked you to write one markdown
7-
file to GitHub using the GitHub Contents API.
8-
9-
Target:
7+
## Target
108

119
- Repository: `{{OWNER}}/{{REPO}}`
1210
- Branch: `{{BRANCH}}`
1311
- File path: `{{FILE_PATH}}`
1412
- Run id: `{{RUN_ID}}`
1513

16-
Requirements:
17-
18-
- Use ordinary shell commands and the `GITHUB_TOKEN` environment variable.
19-
- Do not print, echo, or reveal the token.
20-
- First try to write the file with `PUT /repos/{{OWNER}}/{{REPO}}/contents/{{FILE_PATH}}`.
21-
- If OpenShell returns `policy_denied`, read
22-
`/etc/openshell/skills/policy_advisor.md` and follow the local API workflow
23-
there.
24-
- Submit the narrowest proposal that permits only this write.
25-
- Do not include a `tls` field in the proposed endpoint unless you are
26-
explicitly disabling TLS inspection.
27-
- After submitting a proposal, retry the write for up to 120 seconds. The
28-
developer may approve while you are waiting.
29-
- Do not print the full GitHub response body. It can include temporary
30-
`download_url` query tokens. Extract only `content.path`, `content.html_url`,
31-
and `commit.sha`.
32-
- Finish with a short summary that says whether the write succeeded. Include
33-
the GitHub file path and URL if GitHub returns them.
34-
35-
Suggested file content:
36-
37-
```markdown
38-
# OpenShell policy advisor demo
39-
40-
Run id: {{RUN_ID}}
41-
42-
This file was written from inside an OpenShell sandbox after the agent read the
43-
policy advisor skill, submitted a narrow policy proposal, and waited for
44-
developer approval.
45-
```
14+
## What to do
15+
16+
1. PUT the file with `curl` against
17+
`https://api.github.com/repos/{{OWNER}}/{{REPO}}/contents/{{FILE_PATH}}`.
18+
Authenticate with the `GITHUB_TOKEN` environment variable. Do not echo the
19+
token. Body content (base64-encoded into the JSON `content` field):
20+
21+
```
22+
# OpenShell policy advisor demo
23+
24+
Run id: {{RUN_ID}}
25+
26+
Written from inside an OpenShell sandbox after a narrowly-scoped policy
27+
proposal was approved by the developer.
28+
```
29+
30+
2. The first attempt will fail with HTTP 403 and a JSON body containing
31+
`"error": "policy_denied"`. Read the body — it tells you which layer denied
32+
you (`l7`/`rest`), which method/path was missing, and where to learn more.
33+
34+
3. Read `/etc/openshell/skills/policy_advisor.md` and follow it. Submit the
35+
narrowest possible proposal to `http://policy.local/v1/proposals` — exact
36+
host, exact port, exact method, exact path, binary `/usr/bin/curl`. Do not
37+
include query strings. Do not propose wildcard hosts.
38+
39+
4. After submitting, retry the PUT every few seconds for up to 120 seconds.
40+
The developer is approving from outside the sandbox; once approved, the
41+
sandbox hot-reloads policy and the same PUT will succeed.
42+
43+
5. Stop as soon as the PUT returns HTTP 200 or 201. Print a short summary
44+
showing whether it succeeded, plus `content.path` and `content.html_url`
45+
from the GitHub response. Do not print the full response body.
46+
47+
If anything is unclear, prefer making a narrower proposal and asking for
48+
approval again over widening the rule.

0 commit comments

Comments
 (0)