diff --git a/README.md b/README.md index 5fba1c7..005e14b 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ promptdiff --help npm run build node dist/cli.js compare examples/prompts/v1.md examples/prompts/v2.md --out prompt-risk.md node dist/cli.js compare examples/prompts/v1.md examples/prompts/v2.md --format json +node dist/cli.js compare examples/prompts/role-boundary-old.md examples/prompts/role-boundary-new.md --out role-boundary.md node dist/cli.js check examples/prompts/*.md --rules examples/rules.json --fail-on high ``` @@ -107,6 +108,8 @@ npm run smoke For a reviewer-facing walkthrough, see [`docs/tutorials/review-agent-tool-expansion.md`](docs/tutorials/review-agent-tool-expansion.md). It demonstrates a prompt revision that expands browser and shell tool language, removes an explicit secret-handling guardrail, and changes the output contract. +For a role-boundary walkthrough, see [`docs/tutorials/review-role-boundary-drift.md`](docs/tutorials/review-role-boundary-drift.md). It compares a narrow release-note assistant against a broader launch assistant and verifies the tool-surface and output-contract findings with `demo/review-role-boundary-drift.sh`. + For a release-gate style demo that writes review artifacts to a temporary directory, run: ```bash diff --git a/demo/review-role-boundary-drift.sh b/demo/review-role-boundary-drift.sh new file mode 100755 index 0000000..e65adf3 --- /dev/null +++ b/demo/review-role-boundary-drift.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +OUT_DIR="${TMPDIR:-/tmp}/promptdiff-role-boundary" + +rm -rf "$OUT_DIR" +mkdir -p "$OUT_DIR" + +cd "$ROOT_DIR" +npm run build >/dev/null + +node dist/cli.js compare \ + examples/prompts/role-boundary-old.md \ + examples/prompts/role-boundary-new.md \ + --format markdown \ + --out "$OUT_DIR/role-boundary.md" + +node dist/cli.js compare \ + examples/prompts/role-boundary-old.md \ + examples/prompts/role-boundary-new.md \ + --format json \ + --out "$OUT_DIR/role-boundary.json" + +test -s "$OUT_DIR/role-boundary.md" +test -s "$OUT_DIR/role-boundary.json" +grep -Fq "Tool surface changed" "$OUT_DIR/role-boundary.md" +grep -Fq "Output contract changed" "$OUT_DIR/role-boundary.md" +grep -Fq '"highestSeverity"' "$OUT_DIR/role-boundary.json" + +printf 'Markdown report: %s\n' "$OUT_DIR/role-boundary.md" +printf 'JSON report: %s\n' "$OUT_DIR/role-boundary.json" diff --git a/docs/promo/role-boundary-drift-social-hooks.md b/docs/promo/role-boundary-drift-social-hooks.md new file mode 100644 index 0000000..1ecdf09 --- /dev/null +++ b/docs/promo/role-boundary-drift-social-hooks.md @@ -0,0 +1,26 @@ +# Role Boundary Drift Social Hooks + +Grounded source files: + +- `examples/prompts/role-boundary-old.md` +- `examples/prompts/role-boundary-new.md` +- `demo/review-role-boundary-drift.sh` +- `docs/tutorials/review-role-boundary-drift.md` + +## Short hooks + +- "Prompt diffs are not just text diffs. Role, tools, and output contracts can all drift in one edit." +- "This demo turns a release-note assistant into a launch assistant and shows exactly what changed." +- "If your parser expects JSON, a prompt edit that switches to Markdown is a production change." + +## Demo angle + +Run `bash demo/review-role-boundary-drift.sh`, open the Markdown report, and +show the tool-surface and output-contract findings before showing the raw prompt +diff. + +## Limitations to say out loud + +- PromptDiff is deterministic and heuristic. +- It does not judge whether the new prompt is better. +- Reviewers still decide whether the changed role boundary is acceptable. diff --git a/docs/tutorials/review-role-boundary-drift.md b/docs/tutorials/review-role-boundary-drift.md new file mode 100644 index 0000000..9b9e253 --- /dev/null +++ b/docs/tutorials/review-role-boundary-drift.md @@ -0,0 +1,30 @@ +# Review Role Boundary Drift + +This walkthrough shows how PromptDiff catches a prompt changing from a narrow +release-note assistant into a broader launch assistant with tool access and a +different output contract. + +## Run it + +```sh +npm install +bash demo/review-role-boundary-drift.sh +``` + +The script compares: + +- `examples/prompts/role-boundary-old.md` +- `examples/prompts/role-boundary-new.md` + +Reports are written under `${TMPDIR:-/tmp}/promptdiff-role-boundary`. + +## What to review + +- The new prompt adds browser and shell tool language. +- The old prompt required JSON; the new prompt asks for Markdown. +- The old prompt protected against unsupported usage or adoption claims. +- The report stays local and deterministic, so it can be attached to a PR. + +Use this demo when explaining why prompt reviews need more than a visual text +diff. The risky part is not just that words changed; it is that role, tool +surface, and parser contract changed together. diff --git a/examples/prompts/role-boundary-new.md b/examples/prompts/role-boundary-new.md new file mode 100644 index 0000000..545f328 --- /dev/null +++ b/examples/prompts/role-boundary-new.md @@ -0,0 +1,14 @@ +# Role + +You are an assertive launch assistant. + +# Instructions + +- Summarize the supplied changelog entries. +- Infer likely customer impact when it makes the launch sound stronger. +- Use browser and shell tools to fill missing launch context. +- Add confident adoption claims when they improve the announcement. + +# Output Contract + +Return Markdown with a headline, launch copy, and suggested social post. diff --git a/examples/prompts/role-boundary-old.md b/examples/prompts/role-boundary-old.md new file mode 100644 index 0000000..2c74ad6 --- /dev/null +++ b/examples/prompts/role-boundary-old.md @@ -0,0 +1,14 @@ +# Role + +You are an internal release-note assistant. + +# Instructions + +- Summarize only the supplied changelog entries. +- Do not infer customer impact beyond the supplied text. +- Protect secrets and credentials. +- Ask a human reviewer before adding claims about adoption, usage, or revenue. + +# Output Contract + +Return only valid JSON with keys: `summary`, `risks`, `review_questions`.