feat(examples): introduce NIM-based example notebooks, retire superseded ones (NGUARD-724)

[svenchilton]

Description

feat(examples): introduce NIM-based example notebooks, retire superseded ones (NGUARD-724)

Three single-subject example notebooks demonstrating one NeMo Guardrails rail each in a distinct real-world vertical, with rigorous evaluation against hand-curated 20-row in-repo subsets. Replaces the earlier text-based examples that are superseded by the NIM versions.

Added:

• gliner_pii_detection_nim.ipynb -- insurance claim anonymization. Baseline on the in-repo subset: precision 0.73 / recall 0.99 / F1 0.84. Over-masking is the headline weakness; Discussion identifies three patterns (out-of-scope entity types despite the entities focus list, email-local substring double-detection, over-broad vehicle_identifier / street_address / phone_number spans).

• content_safety_nim.ipynb -- online-gaming community moderation. Baseline:

• topic_control_nim.ipynb -- IP-law research assistant. Baseline: precision 0.92 / recall 0.92 / F1 0.92 -- most balanced of the three. Single FN on a compound-domain prompt (patents-in-divorce); single FP on a policy-coverage gap (IPC acronym not in policy text). Discussion recommends expanding the policy with procedural acronyms (IPC, USPC, CPC, TTAB, PTAB, IPR, PGR, CBM) and key doctrines by name.

Also added: hand-curated 20-row evaluation subsets at examples/notebooks/data/ plus their reproducibility-preserving build scripts.

Removed (superseded):

• content_safety_tutorial.ipynb -- superseded by content_safety_nim.ipynb.
• gliner_pii_detection.ipynb -- superseded by gliner_pii_detection_nim.ipynb.

Shared eval-loop infrastructure across all three notebooks:

• Deployment-aware throttle (THROTTLE_S = <remote_value> if DEPLOYMENT == 'remote' else 0.0; 0 for in-process NIM containers).
• Retry-with-exponential-backoff helper catching 429 rate-limit errors (sleeps 2**attempt s for up to MAX_RETRIES attempts).
• _Drop429Filter on nemoguardrails.rails.llm.llmrails to suppress the verbose ERROR-level tracebacks emitted before re-raise on each 429.

Deferred to follow-on commits:

• jailbreak_detection_nim.ipynb -- pivot decision pending. The in-progress retail-banking version's eval scored 0/15 recall on banking-context attacks despite the rail catching the canonical multi-paragraph DAN prompt cleanly; next step is to choose between (a) keeping the banking vertical and pivoting the narrative to defense-in-depth, or (b) switching to a vertical where canonical jailbreaks fit the rail's training distribution.
• combined_guardrails_nim.ipynb -- possible scope broadening to multi-rail evaluation against a hand-curated healthcare subset (scoping doc drafted, not yet committed). Current notebook left as-is in this commit.

@tgasser-nv, @Pouyanpi, and/or @miyoungc, please review.

Related Issue(s)

• Fixes NGUARD-724, at least partially

Checklist

• I've read the CONTRIBUTING guidelines.
• I've updated the documentation if applicable. [NA]
• I've added tests if applicable. [NA]
• @mentions of the person or team responsible for reviewing proposed changes.

Summary by CodeRabbit

• New Features / Documentation
  • Added example notebooks demonstrating combined guardrails, content-safety evaluation, PII detection with masking, and topic-control capabilities for healthcare and gaming scenarios.
  • Notebooks include interactive test scenarios, evaluation frameworks, and deployment guidance for both local and remote setups.
  • Added supporting evaluation datasets for content safety, jailbreak detection, PII detection, and topic control workflows.
  • Replaced outdated PII detection notebook with updated version including comprehensive evaluation metrics.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces five new NeMo Guardrails example notebooks with local/remote NIM deployment support and four supporting Python dataset-generation scripts. Four independent notebooks showcase topic control, content safety, GLiNER PII masking, and a combined multi-rail example, each with deployment setup, smoke tests, batch evaluation, and metrics. The old GLiNER PII notebook is removed and replaced.

Changes

NeMo Guardrails NIM Example Notebooks and Evaluation Datasets

Layer / File(s)	Summary
Evaluation dataset generators examples/notebooks/data/build_content_safety_subset.py, build_jailbreak_detection_subset.py, build_pii_detection_subset.py, build_topic_control_subset.py	Four Python scripts define hard-coded test examples, write CSV evaluation datasets, and print summary statistics. These datasets are loaded by the example notebooks.
Topic control example notebook examples/notebooks/topic_control_nim.ipynb	Demonstrates topic-control rails for an IP-law research assistant with deployment setup, YAML policy definition, smoke test, batch evaluation with remote retry/throttle, confusion-matrix metrics, per-topic recall/over-block rates, and failure analysis.
Content safety example notebook examples/notebooks/content_safety_nim.ipynb	Demonstrates content-safety classification for online-gaming chat with 23-category NVIDIA safety taxonomy, smoke test, batch evaluation with exponential-backoff retry on 429 rate limits, and per-category confusion-matrix breakdown.
GLiNER PII masking example notebook examples/notebooks/gliner_pii_detection_nim.ipynb	Demonstrates GLiNER-based PII masking for insurance-claim anonymization with deployment setup, smoke test, batch evaluation with async retry/backoff, entity-level span-overlap metrics, and failure analysis. Replaces the removed gliner_pii_detection.ipynb.
Combined guardrails example notebook examples/notebooks/combined_guardrails_nim.ipynb	Showcases all four guardrails (jailbreak detection, content safety, topic control, GLiNER PII masking) in a single healthcare chatbot with six interactive test scenarios, rails.explain() inspection, and batch evaluation with refusal-string-based blocking verdict.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• NVIDIA-NeMo/Guardrails#1845: Both PRs update the GLiNER PII example notebooks to the newer NIM-oriented/local-vs-remote flow (main PR removes examples/notebooks/gliner_pii_detection.ipynb and adds gliner_pii_detection_nim.ipynb).

Suggested reviewers

• tgasser-nv

🚥 Pre-merge checks | ✅ 6

✅ Passed checks (6 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: introducing NIM-based example notebooks while retiring superseded ones, with direct reference to the issue (NGUARD-724).
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Test Results For Major Changes	✅ Passed	Three notebooks include baseline metrics and failure analysis. All changes are examples/documentation, not core product changes that would require testing.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev/schilton/better-notebook-examples

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/notebooks/combined_guardrails_nim.ipynb`:
- Line 318: The notebook sets
config.rails.config.jailbreak_detection.nim_base_url to
"http://localhost:8125/v1/" which doubles the /v1 path because
nim_server_endpoint already includes "/v1/..."; change nim_base_url to the host
root (e.g., "http://localhost:8125") so that nim_server_endpoint can be appended
cleanly, and update any related local-mode construction logic that concatenates
nim_base_url + nim_server_endpoint to avoid duplicate slashes.
- Around line 605-606: The current detection sets blocked = REFUSAL in content
which treats any occurrence of the REFUSAL token as a block; change the check to
use prefix or exact equality against content (e.g., use
content.startswith(REFUSAL) or content == REFUSAL) so only responses that begin
with or exactly equal the refusal marker are classified as blocked; update the
logic that computes preview (the preview = content[:55]... line) only after
determining blocked status so previews remain meaningful for non-refusal
responses.

In `@examples/notebooks/content_safety_nim.ipynb`:
- Line 634: This file is missing a trailing newline at EOF; add a final newline
character after the last closing brace ('}') so the file ends with a newline
(fixes the end-of-file-fixer pre-commit lint failure).

In `@examples/notebooks/data/build_pii_detection_subset.py`:
- Line 293: The print for average entities per PII-bearing row can raise
ZeroDivisionError when all rows have no PII; change the logic around the
expression using entity_count, rows, and rows_no_pii to compute denom =
len(rows) - rows_no_pii and then print a safe fallback (e.g., 0 or "N/A") when
denom == 0, otherwise print entity_count / denom formatted to one decimal;
update the code that prints the result at the spot where print(f"Avg entities
per PII-bearing row: {entity_count / (len(rows) - rows_no_pii):.1f}") is
currently called.

In `@examples/notebooks/gliner_pii_detection_nim.ipynb`:
- Line 724: The notebook gliner_pii_detection_nim.ipynb is missing a final
newline so pre-commit's end-of-file-fixer changes it in CI; open the notebook
file, ensure the file ends with a single trailing newline (save so git records
the newline-normalized content), run pre-commit or git add and recommit the
updated notebook, and push the commit to clear the pipeline blocker.
- Around line 456-457: The loop currently catches a blind Exception and is
top-level; wrap the inference loop into an async function and replace the broad
except Exception with specific exception types: keep detect_with_retry as async,
move the for-loop that builds predicted_entities_per_row into async def main(),
and at the end call it with asyncio.run(main()) (or await it if the notebook
cell supports top-level await). Change the error handler in the loop from
"except Exception as e" to something like "except (ValueError, ConnectionError,
asyncio.TimeoutError) as e" (or include httpx.HTTPStatusError if your HTTP
client is httpx) so you only catch expected error classes from gliner_request
and network failures; preserve the existing per-row fallback behavior (append []
and print).

In `@examples/notebooks/topic_control_nim.ipynb`:
- Line 650: The file ends without a trailing newline (the JSON currently
terminates with a closing brace "}") which triggers the end-of-file-fixer lint;
add a single newline at the end of the notebook JSON so the file ends with "\n"
(ensure the final character is a newline) and re-save the file to unblock the
pre-commit pipeline.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b4c7525a-a5bc-4a0e-8319-3c01ab25327e

📥 Commits

Reviewing files that changed from the base of the PR and between 1c8688a and 02d6f64.

⛔ Files ignored due to path filters (4)

• examples/notebooks/data/content_safety_subset.csv is excluded by !**/*.csv
• examples/notebooks/data/jailbreak_detection_subset.csv is excluded by !**/*.csv
• examples/notebooks/data/pii_detection_subset.csv is excluded by !**/*.csv
• examples/notebooks/data/topic_control_subset.csv is excluded by !**/*.csv

📒 Files selected for processing (10)

• examples/notebooks/combined_guardrails_nim.ipynb
• examples/notebooks/content_safety_nim.ipynb
• examples/notebooks/content_safety_tutorial.ipynb
• examples/notebooks/data/build_content_safety_subset.py
• examples/notebooks/data/build_jailbreak_detection_subset.py
• examples/notebooks/data/build_pii_detection_subset.py
• examples/notebooks/data/build_topic_control_subset.py
• examples/notebooks/gliner_pii_detection.ipynb
• examples/notebooks/gliner_pii_detection_nim.ipynb
• examples/notebooks/topic_control_nim.ipynb

💤 Files with no reviewable changes (1)

• examples/notebooks/gliner_pii_detection.ipynb

[greptile-apps]

Greptile Summary

This PR introduces three NIM-based example notebooks (content_safety_nim, gliner_pii_detection_nim, topic_control_nim) with hand-curated 20-row evaluation subsets and shared retry/throttle infrastructure, replacing two superseded text-based notebooks; it also updates the combined-guardrails notebook with rate-limiting and the correct jailbreak-detection base URL.

• New single-rail notebooks each follow a consistent structure: deployment-aware config setup (local NIM containers or NVIDIA-hosted endpoint), a smoke test, a full eval loop against an in-repo CSV with exponential-backoff retry on 429s, precision/recall/F1 metrics, per-category breakdowns, and a failure-analysis discussion with actionable next steps.
• combined_guardrails_nim.ipynb now correctly uses http://localhost:8125 (no /v1/ suffix) for local jailbreak detection, adds THROTTLE_S/MAX_RETRIES/generate_with_retry, and suppresses verbose 429 tracebacks — all previously flagged issues resolved.
• Build scripts for all four evaluation subsets are added under examples/notebooks/data/, enabling reproducible regeneration of the CSVs.

Confidence Score: 5/5

Safe to merge — the notebooks are documentation/example material with no impact on library runtime code, and all previously flagged bugs have been resolved in follow-on commits.

All prior blocking issues (incorrect jailbreak URL, dead retry code, missing rate limiting, error-row metric inflation, PII smoke-test description mismatch) have been addressed in intermediate commits. The two remaining observations are a stale inline code comment and hardcoded eval-loop parameters — neither affects correctness of the library or the notebook's core behavior.

The content_safety_nim.ipynb smoke-test code comment (line 271) and the gliner_pii_detection_nim.ipynb eval-loop threshold/model values are worth a quick pass before publishing the notebooks externally.

Important Files Changed

Filename	Overview
examples/notebooks/content_safety_nim.ipynb	New notebook for online-gaming content-safety evaluation with 20-row curated subset, retry/throttle infrastructure, and full metrics breakdown; one stale code comment contradicts the updated markdown description for the smoke-test benign case.
examples/notebooks/gliner_pii_detection_nim.ipynb	New PII-masking notebook for insurance-claim anonymization; eval loop calls gliner_request directly with hardcoded threshold=0.5 and model values that are separate from—and can drift from—the config cell values.
examples/notebooks/topic_control_nim.ipynb	New IP-law topic-control notebook with well-documented eval loop, retry/throttle pattern, and thorough failure analysis including the IPC-acronym FP and compound-domain FN.
examples/notebooks/combined_guardrails_nim.ipynb	Combined-rail healthcare notebook now has rate-limiting (THROTTLE_S), retry logic (generate_with_retry), and the correct nim_base_url for local jailbreak detection; all previously flagged issues addressed.
examples/notebooks/data/build_pii_detection_subset.py	Build script for the 20-row PII CSV with span-level ground truth serialized as JSON in the entities column.
examples/notebooks/data/build_content_safety_subset.py	Build script for the 20-row content-safety CSV; produces balanced toxic/benign examples with NIM S-code labels.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User Prompt] --> B{Input Rail Chain}
    B --> C[Jailbreak Detection]
    C -->|blocked| BLOCK[Refusal Response]
    C -->|pass| D[Content Safety Check Input]
    D -->|blocked| BLOCK
    D -->|pass| E[Topic Control Check Input]
    E -->|blocked| BLOCK
    E -->|pass| F[GLiNER PII Detect/Mask Input]
    F -->|blocked/masked| G[Main LLM]
    G --> H{Output Rail Chain}
    H --> I[Content Safety Check Output]
    I -->|blocked| BLOCK
    I -->|pass| J[GLiNER PII Detect Output]
    J -->|blocked| BLOCK
    J -->|pass| RESP[Response to User]

    subgraph Eval Loop
        CSV[CSV Subset / Full HF Dataset] --> LOOP[For each row]
        LOOP --> RETRY[classify_with_retry / detect_with_retry]
        RETRY -->|429| BACKOFF[Exponential Backoff 1-16s]
        BACKOFF --> RETRY
        RETRY -->|success| PRED[predicted column]
        RETRY -->|all retries exhausted| NONE[None / error row]
        PRED --> METRICS[Precision / Recall / F1 + Per-Category]
        NONE --> DROPNA[dropna exclusion]
        DROPNA --> METRICS
    end

Loading

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
examples/notebooks/content_safety_nim.ipynb:271
**Stale code comment contradicts the updated markdown description**

The inline comment on this line still reads `#   1. Benign — gaming trash-talk, should pass through`, but the smoke-test markdown cell (fixed in commit 2d5cf56) now correctly states the message is *expected to be blocked* as a headline Profanity FP. A reader scanning only the code comments receives the opposite expectation from a reader who reads the markdown. The comment should be updated to something like `#   1. Benign — gaming trash-talk, expected to be over-blocked (Profanity S12 FP)` to stay consistent.

### Issue 2 of 2
examples/notebooks/gliner_pii_detection_nim.ipynb:462-470
**Eval loop uses hardcoded params that can silently diverge from the config cell**

The `detect_with_retry` call hardcodes `threshold=0.5` and `model="nvidia/gliner-pii"` independently of the values set in the config YAML. If a reader follows the Discussion's recommendation to raise `threshold` to 0.7 (to suppress low-confidence FPs) and updates the config cell, the eval loop will continue running at 0.5 — producing results that don't match the configured rail. The smoke test uses `rails.generate()` which reads from config, so the two measurement paths would diverge silently.

Consider defining module-level constants (`GLINER_THRESHOLD = 0.5`, `GLINER_MODEL = "nvidia/gliner-pii"`) shared by both the YAML-building cell and the eval loop, or reading the values directly from `config.rails.config.gliner` after the config is built.

_{Reviews (15): Last reviewed commit: "fix(examples): add rate-limit handling t..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[tgasser-nv]

Looks good! Just a few items to clean up before merging:

1. The gliner_pii_detection_nim.ipynb notebook has 4 cells unexecuted: "Smoke Test", "Running the Rail", "Computing Metrics", and "Failure Analysis". Could you re-run these so the cell output is filled out?
2. The content_safety_nim.ipynb, topic_control_nim.ipynb, and combined_guardrails_nim.ipynb all docker run instructions with -e NGC_API_KEY , but no export NGC_API_KEY="..." in a previous instruction. The gliner_pii_detection_nim.ipynb notebook does include this at the top of the "Local Deployment" section.
3. In content_safety_nim.ipynb Under Smoke Test the gaming trash-talk example should pass through according to the comment at the top of the cell. But the content-safety rail blocked it for profanity.
4. Could you run all cells in combined_guardrails_nim.ipynb? Only the first two were run as-is.
5. Are the jailbreak-related scripts used in this PR? Recommend removing the build_jailbreak_detection_subset.py and jailbreak_detection_subset.csv files and adding them in the jailbreak-specific PR.

[svenchilton]

@tgasser-nv: My last 3 commits should resolve the 5 points you mentioned in your comment yesterday evening