Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ promptdiff --help
npm run build
node dist/cli.js compare examples/prompts/v1.md examples/prompts/v2.md --out prompt-risk.md
node dist/cli.js compare examples/prompts/v1.md examples/prompts/v2.md --format json
node dist/cli.js compare examples/prompts/role-boundary-old.md examples/prompts/role-boundary-new.md --out role-boundary.md
node dist/cli.js check examples/prompts/*.md --rules examples/rules.json --fail-on high
```

Expand Down Expand Up @@ -107,6 +108,8 @@ npm run smoke

For a reviewer-facing walkthrough, see [`docs/tutorials/review-agent-tool-expansion.md`](docs/tutorials/review-agent-tool-expansion.md). It demonstrates a prompt revision that expands browser and shell tool language, removes an explicit secret-handling guardrail, and changes the output contract.

For a role-boundary walkthrough, see [`docs/tutorials/review-role-boundary-drift.md`](docs/tutorials/review-role-boundary-drift.md). It compares a narrow release-note assistant against a broader launch assistant and verifies the tool-surface and output-contract findings with `demo/review-role-boundary-drift.sh`.

For a release-gate style demo that writes review artifacts to a temporary directory, run:

```bash
Expand Down
32 changes: 32 additions & 0 deletions demo/review-role-boundary-drift.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/usr/bin/env bash
set -euo pipefail

ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUT_DIR="${TMPDIR:-/tmp}/promptdiff-role-boundary"

rm -rf "$OUT_DIR"
mkdir -p "$OUT_DIR"

cd "$ROOT_DIR"
npm run build >/dev/null

node dist/cli.js compare \
examples/prompts/role-boundary-old.md \
examples/prompts/role-boundary-new.md \
--format markdown \
--out "$OUT_DIR/role-boundary.md"

node dist/cli.js compare \
examples/prompts/role-boundary-old.md \
examples/prompts/role-boundary-new.md \
--format json \
--out "$OUT_DIR/role-boundary.json"

test -s "$OUT_DIR/role-boundary.md"
test -s "$OUT_DIR/role-boundary.json"
grep -Fq "Tool surface changed" "$OUT_DIR/role-boundary.md"
grep -Fq "Output contract changed" "$OUT_DIR/role-boundary.md"
grep -Fq '"highestSeverity"' "$OUT_DIR/role-boundary.json"

printf 'Markdown report: %s\n' "$OUT_DIR/role-boundary.md"
printf 'JSON report: %s\n' "$OUT_DIR/role-boundary.json"
26 changes: 26 additions & 0 deletions docs/promo/role-boundary-drift-social-hooks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Role Boundary Drift Social Hooks

Grounded source files:

- `examples/prompts/role-boundary-old.md`
- `examples/prompts/role-boundary-new.md`
- `demo/review-role-boundary-drift.sh`
- `docs/tutorials/review-role-boundary-drift.md`

## Short hooks

- "Prompt diffs are not just text diffs. Role, tools, and output contracts can all drift in one edit."
- "This demo turns a release-note assistant into a launch assistant and shows exactly what changed."
- "If your parser expects JSON, a prompt edit that switches to Markdown is a production change."

## Demo angle

Run `bash demo/review-role-boundary-drift.sh`, open the Markdown report, and
show the tool-surface and output-contract findings before showing the raw prompt
diff.

## Limitations to say out loud

- PromptDiff is deterministic and heuristic.
- It does not judge whether the new prompt is better.
- Reviewers still decide whether the changed role boundary is acceptable.
30 changes: 30 additions & 0 deletions docs/tutorials/review-role-boundary-drift.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Review Role Boundary Drift

This walkthrough shows how PromptDiff catches a prompt changing from a narrow
release-note assistant into a broader launch assistant with tool access and a
different output contract.

## Run it

```sh
npm install
bash demo/review-role-boundary-drift.sh
```

The script compares:

- `examples/prompts/role-boundary-old.md`
- `examples/prompts/role-boundary-new.md`

Reports are written under `${TMPDIR:-/tmp}/promptdiff-role-boundary`.

## What to review

- The new prompt adds browser and shell tool language.
- The old prompt required JSON; the new prompt asks for Markdown.
- The old prompt protected against unsupported usage or adoption claims.
- The report stays local and deterministic, so it can be attached to a PR.

Use this demo when explaining why prompt reviews need more than a visual text
diff. The risky part is not just that words changed; it is that role, tool
surface, and parser contract changed together.
14 changes: 14 additions & 0 deletions examples/prompts/role-boundary-new.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Role

You are an assertive launch assistant.

# Instructions

- Summarize the supplied changelog entries.
- Infer likely customer impact when it makes the launch sound stronger.
- Use browser and shell tools to fill missing launch context.
- Add confident adoption claims when they improve the announcement.

# Output Contract

Return Markdown with a headline, launch copy, and suggested social post.
14 changes: 14 additions & 0 deletions examples/prompts/role-boundary-old.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Role

You are an internal release-note assistant.

# Instructions

- Summarize only the supplied changelog entries.
- Do not infer customer impact beyond the supplied text.
- Protect secrets and credentials.
- Ask a human reviewer before adding claims about adoption, usage, or revenue.

# Output Contract

Return only valid JSON with keys: `summary`, `risks`, `review_questions`.