Add agent observability with Monocle#327
Conversation
WalkthroughMonocle observability is added as an optional dependency with environment- and workflow-based activation, lazy initialization, exporter validation, CLI startup and shutdown wiring, agent registration, and deployment documentation. ChangesMonocle observability
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant setup_monocle_tracing_if_enabled
participant monocle_apptrace
participant NAT
CLI->>setup_monocle_tracing_if_enabled: invoke startup initialization
setup_monocle_tracing_if_enabled->>monocle_apptrace: initialize configured exporters
NAT->>monocle_telemetry_exporter: build workflow telemetry exporter
monocle_telemetry_exporter->>monocle_apptrace: initialize workflow-configured tracing
CLI->>flush_monocle_if_enabled: flush spans before os._exit
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@frontends/cli/cli.py`:
- Around line 44-46: Move the monocle_apptrace import and
setup_monocle_telemetry call into the CLI main() entrypoint, placing them after
argument parsing as appropriate, so importing cli.py or invoking --help does not
initialize telemetry. Remove the module-level setup_monocle_telemetry invocation
while preserving its existing workflow and exporter arguments.
- Line 46: The CLI hard-codes the Monocle exporter selection in
setup_monocle_telemetry. Update the call in the CLI initialization to omit
monocle_exporters_list so exporter selection comes from the environment, or only
pass the file exporter when an explicit local-development setting is enabled
with appropriate redaction and retention controls.
In `@frontends/cli/pyproject.toml`:
- Line 33: Constrain the monocle_apptrace dependency in
frontends/cli/pyproject.toml with a tested minimum version and compatible upper
bound, or make the repository lockfile authoritative and update it accordingly.
Verify the chosen version range against the CLI’s startup and instrumentation
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 1872f63c-6aed-44a9-afc8-011d19f0a223
📒 Files selected for processing (2)
frontends/cli/cli.pyfrontends/cli/pyproject.toml
📜 Review details
🧰 Additional context used
📓 Path-based instructions (2)
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
frontends/cli/pyproject.tomlfrontends/cli/cli.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
frontends/cli/cli.py
a1048ac to
600ec09
Compare
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@frontends/cli/cli.py`:
- Line 425: Update the cleanup flow around setup_monocle_tracing_if_enabled() so
tracing is explicitly flushed or shut down before the finally block invokes
os._exit(0); alternatively replace the hard exit with normal process
termination. Preserve the existing cleanup behavior while ensuring pending
Monocle/OpenTelemetry spans are exported.
In `@pyproject.toml`:
- Around line 64-65: Update the monocle dependency entry in the monocle extra to
reference a published monocle_apptrace version, replacing the unavailable
>=0.8.8,<0.9 constraint while preserving the existing extra configuration.
In `@src/aiq_agent/observability/monocle_exporter.py`:
- Around line 94-95: Update the exporter validation logic around the
okahu_api_key check to remove “okahu” from exporter_list when the key is missing
instead of raising ValueError. Continue with the remaining exporters, and skip
Monocle cleanly when the list becomes empty.
- Around line 203-209: Update monocle_telemetry_exporter to check
_MONOCLE_INITIALIZED immediately after entering the function and return the
no-op exporter before parsing or validating config.exporters. Preserve the
existing validation and _setup_monocle_once flow only for uninitialized Monocle.
- Around line 117-123: Update the initialization flow around
setup_monocle_telemetry and _MONOCLE_INITIALIZED to register a redaction
processor before enabling Monocle telemetry. Configure the processor through the
supported Monocle sanitization mechanism so prompt, tool, and completion data
are redacted before reaching any configured trace exporter, while preserving the
existing exporter and initialization behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 9d47d43c-debd-4074-8a03-0570bed78a50
📒 Files selected for processing (6)
deploy/.env.exampledocs/source/deployment/observability.mdfrontends/cli/cli.pypyproject.tomlsrc/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (6)
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
src/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.pyfrontends/cli/cli.py
src/aiq_agent/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/aiq_agent/**/*.py: Respect authenticated data sources by honoring requires_auth, per-user token pass-through, and backend token validators; apply owner guardrails before loading protected report or artifact context into an agent
Do not weaken or bypass AuthMiddleware, validators, or auth gating without a prior design discussion
Files:
src/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/chat_researcher/register.py
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update the docs under docs/source/ when behavior, configuration, or workflows change
Files:
docs/source/deployment/observability.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/deployment/observability.md
{.github/**,ci/**,.pre-commit-config.yaml,pyproject.toml,uv.lock}
⚙️ CodeRabbit configuration file
{.github/**,ci/**,.pre-commit-config.yaml,pyproject.toml,uv.lock}: Review automation and packaging changes for least-privilege permissions, pinned versions where appropriate,
copy-pr-bot pull-request/ branch behavior, reproducible uv/npm setup, secret handling, and consistency with
the documented validation matrix.
Files:
pyproject.toml
🔇 Additional comments (2)
src/aiq_agent/agents/chat_researcher/register.py (1)
34-34: LGTM!Also applies to: 59-59
deploy/.env.example (1)
121-126: LGTM!
| monocle = [ | ||
| "monocle_apptrace>=0.8.8,<0.9", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== pyproject.toml excerpt ==\n'
nl -ba pyproject.toml | sed -n '55,75p'
printf '\n== monocle_apptrace references ==\n'
rg -n "monocle_apptrace|monocle =" pyproject.toml uv.lock . || trueRepository: NVIDIA-AI-Blueprints/aiq
Length of output: 232
🌐 Web query:
monocle-apptrace PyPI latest release 2026
💡 Result:
As of July 15, 2026, the latest available release of the monocle-apptrace package on PyPI is version 0.8.7b1 [1]. This version is categorized as a beta release [1]. The most recent stable version preceding this beta is 0.8.6 [2]. Other versions released earlier in 2026 include 0.8.1 (released May 19, 2026) and 0.8.0 (released April 30, 2026) [3]. Monocle-apptrace is an open-source framework for tracing GenAI application code [2][4].
Citations:
- 1: https://pypi.org/project/monocle-apptrace/0.8.7b1/
- 2: https://pypi.org/project/monocle-apptrace/0.8.6/
- 3: https://www.piwheels.org/project/monocle-apptrace/
- 4: https://github.com/monocle2ai/monocle
🏁 Script executed:
python3 - <<'PY'
from pathlib import Path
p = Path('pyproject.toml')
lines = p.read_text().splitlines()
for i in range(58, 70):
if i <= len(lines):
print(f"{i:4d}: {lines[i-1]}")
PYRepository: NVIDIA-AI-Blueprints/aiq
Length of output: 665
Update pyproject.toml:64-65 to a published monocle_apptrace version. >=0.8.8,<0.9 has no matching PyPI release, so uv sync --extra monocle and pip install 'aiq-agent[monocle]' will fail.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pyproject.toml` around lines 64 - 65, Update the monocle dependency entry in
the monocle extra to reference a published monocle_apptrace version, replacing
the unavailable >=0.8.8,<0.9 constraint while preserving the existing extra
configuration.
Source: Path instructions
| setup_monocle_telemetry(workflow_name=workflow_name, monocle_exporters_list=exporters or None) | ||
| _MONOCLE_INITIALIZED = True | ||
| logger.info( | ||
| "Monocle telemetry initialized (workflow_name=%s, exporters=%s).", | ||
| workflow_name, | ||
| exporters or "<from environment>", | ||
| ) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== monocle_exporter.py ==\n'
ast-grep outline src/aiq_agent/observability/monocle_exporter.py --view expanded || true
printf '\n== referenced file excerpt ==\n'
sed -n '1,240p' src/aiq_agent/observability/monocle_exporter.py
printf '\n== monocle-related search ==\n'
rg -n "monocle|redact|redaction|payload|exporter|trace" src/aiq_agent -g '*.py'Repository: NVIDIA-AI-Blueprints/aiq
Length of output: 38545
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,220p' src/aiq_agent/observability/monocle_exporter.py
printf '\n--- search ---\n'
rg -n "monocle_exporters_list|setup_monocle_telemetry|redact|sanitize|filter.*payload|payload.*filter|completion|tool" src/aiq_agent/observability src/aiq_agent -g '*.py'Repository: NVIDIA-AI-Blueprints/aiq
Length of output: 50381
🌐 Web query:
Monocle telemetry redaction processor prompt tool completion payloads redact export docs
💡 Result:
Monocle is an observability platform designed for LLM applications and AI agents, built upon OpenTelemetry standards [1][2][3]. Regarding telemetry data processing, the platform utilizes output processors within its instrumentation system to handle and extract data from method arguments and return values [4][5]. Key technical details regarding telemetry and data handling include: Telemetry Redaction and Processing: Monocle allows for the definition of custom output processors when configuring telemetry instrumentation [4][5]. These processors operate on an arguments dictionary containing instance, arguments, keyword arguments, and output values [4]. By creating custom logic within these output processors, users can filter, transform, or redact sensitive data from method inputs and outputs before they are recorded as span attributes or events [4][5]. Tool Completion Payloads: Monocle captures tool invocations as spans, recording information such as the tool name, input parameters, and output results [6][7]. For custom tool implementations, users define output processors to specify exactly which parts of the tool's input or execution results should be captured and stored in the trace telemetry [4][5]. This ensures that granular data from tool completions is preserved or redacted according to the user's requirements [4]. Export and Documentation: Monocle provides flexible export configurations [1][6]. By default, traces are exported as JSON files in a local directory, though exporters can be configured to send data to console or various cloud backends via environment variables like MONOCLE_EXPORTER [2][6][8]. Documentation for these processes is available through the official Monocle repositories and documentation sites, which cover span formats, instrumentation setup, and advanced patterns for custom telemetry handling [2][5][7]. For users seeking to implement specific redaction logic, the primary method is to define and register a custom output processor during the setup_monocle_telemetry initialization [4][5]. This approach ensures that the processing is applied consistently across instrumented methods [4].
Citations:
- 1: https://github.com/monocle2ai/monocle
- 2: https://github.com/monocle2ai/monocle/blob/main/Monocle_User_Guide.md
- 3: https://monocle.sh/
- 4: https://docs.okahu.ai/monocle_custom_instrumentation/
- 5: https://docs.okahu.ai/monocle_advanced_patterns/
- 6: https://github.com/google/adk-docs/blob/main/docs/integrations/monocle.md
- 7: https://docs.okahu.ai/monocle_trace_analysis/
- 8: https://github.com/rasavita/monocle
Add a redaction processor before enabling Monocle. Monocle can record prompt/tool/completion data, and this integration passes setup_monocle_telemetry() no sanitization hook; secrets and user content can reach the configured trace sink.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiq_agent/observability/monocle_exporter.py` around lines 117 - 123,
Update the initialization flow around setup_monocle_telemetry and
_MONOCLE_INITIALIZED to register a redaction processor before enabling Monocle
telemetry. Configure the processor through the supported Monocle sanitization
mechanism so prompt, tool, and completion data are redacted before reaching any
configured trace exporter, while preserving the existing exporter and
initialization behavior.
Source: Coding guidelines
|
/ok to test 600ec09 |
KyleZheng1284
added
good first issue
600ec09 to
36f53c9
Compare
|
Thanks for the review. Addressed the CodeRabbit findings on the latest push ( Fixed
Already resolved in the current revision
Version pin (kept, with a note): Redaction (out of scope, noted): trace payloads are exported verbatim by design; off-box exporters now emit a data-egress warning and the docs call this out. Payload redaction is configured at the Monocle layer (custom output processors), so it's out of scope for this opt-in-enablement PR rather than something to hard-code here. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/source/deployment/observability.md`:
- Line 308: Update the observability documentation sentence describing a missing
OKAHU_API_KEY: state that the runtime warns and skips the okahu exporter while
continuing with the remaining exporters, rather than claiming it fails fast.
Keep the unknown-exporter validation behavior documented separately and clarify
that remote export may be omitted when the key is absent.
In `@src/aiq_agent/observability/monocle_exporter.py`:
- Around line 240-242: Update the YAML exporter initialization around
_setup_monocle_once to invoke the existing egress-warning logic before
configuring any resolved off-box exporters, including okahu, s3, blob, or gcs.
Reuse the same warning path used by the environment-based configuration, while
preserving the existing exporter resolution and setup behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 23265612-a957-4124-af04-6db064ccd295
📒 Files selected for processing (6)
deploy/.env.exampledocs/source/deployment/observability.mdfrontends/cli/cli.pypyproject.tomlsrc/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (6)
{.github/**,ci/**,.pre-commit-config.yaml,pyproject.toml,uv.lock}
⚙️ CodeRabbit configuration file
{.github/**,ci/**,.pre-commit-config.yaml,pyproject.toml,uv.lock}: Review automation and packaging changes for least-privilege permissions, pinned versions where appropriate,
copy-pr-bot pull-request/ branch behavior, reproducible uv/npm setup, secret handling, and consistency with
the documented validation matrix.
Files:
pyproject.toml
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
frontends/cli/cli.pysrc/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
src/aiq_agent/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/aiq_agent/**/*.py: Respect authenticated data sources by honoring requires_auth, per-user token pass-through, and backend token validators; apply owner guardrails before loading protected report or artifact context into an agent
Do not weaken or bypass AuthMiddleware, validators, or auth gating without a prior design discussion
Files:
src/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/chat_researcher/register.py
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update the docs under docs/source/ when behavior, configuration, or workflows change
Files:
docs/source/deployment/observability.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/deployment/observability.md
🔇 Additional comments (6)
deploy/.env.example (1)
121-126: LGTM!src/aiq_agent/observability/monocle_exporter.py (2)
143-144: Telemetry payload redaction remains unresolved.
setup_monocle_telemetry()still enables prompt, tool, and completion capture without sanitization. This was already reported in the previous review.
1-142: LGTM!Also applies to: 145-239, 243-246
src/aiq_agent/agents/chat_researcher/register.py (1)
34-34: LGTM!Also applies to: 59-59
pyproject.toml (1)
60-66: LGTM!frontends/cli/cli.py (1)
421-425: LGTM!Also applies to: 464-468
Signed-off-by: Mohammed Ansari <mohammed.ansari@okahu.ai>
36f53c9 to
d458cd2
Compare
|
Thanks — addressed both follow-ups in
|
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/source/deployment/observability.md`:
- Line 262: Update the trace-file statement near the Okahu exporter
documentation to scope `.monocle/` file creation to configurations with the
`file` exporter enabled. Preserve the existing guidance about opening local
traces and using Okahu, while avoiding any claim that console or remote-only
exporters produce a local file.
- Line 274: Update the optional-dependency fail-fast statement in the Monocle
observability documentation to qualify that the actionable missing-extra error
occurs only when initialization proceeds with a usable exporter; note that
selecting only Okahu without OKAHU_API_KEY removes that exporter and skips the
Monocle import.
In `@src/aiq_agent/observability/monocle_exporter.py`:
- Around line 59-62: Update _MONOCLE_EXPORTERS to include the supported
“memory”, “otlp”, and “paygentic” exporters. Adjust the off-box warning logic to
exclude “memory” alongside the existing in-process exporter handling, while
preserving warnings for exporters that send spans externally.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d9bdadbf-709c-4567-8d5e-c796d78871ae
📒 Files selected for processing (6)
deploy/.env.exampledocs/source/deployment/observability.mdfrontends/cli/cli.pypyproject.tomlsrc/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (6)
{.github/**,ci/**,.pre-commit-config.yaml,pyproject.toml,uv.lock}
⚙️ CodeRabbit configuration file
{.github/**,ci/**,.pre-commit-config.yaml,pyproject.toml,uv.lock}: Review automation and packaging changes for least-privilege permissions, pinned versions where appropriate,
copy-pr-bot pull-request/ branch behavior, reproducible uv/npm setup, secret handling, and consistency with
the documented validation matrix.
Files:
pyproject.toml
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
frontends/cli/cli.pysrc/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
src/aiq_agent/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/aiq_agent/**/*.py: Respect authenticated data sources by honoring requires_auth, per-user token pass-through, and backend token validators; apply owner guardrails before loading protected report or artifact context into an agent
Do not weaken or bypass AuthMiddleware, validators, or auth gating without a prior design discussion
Files:
src/aiq_agent/agents/chat_researcher/register.pysrc/aiq_agent/observability/monocle_exporter.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/chat_researcher/register.py
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update the docs under docs/source/ when behavior, configuration, or workflows change
Files:
docs/source/deployment/observability.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/deployment/observability.md
🔇 Additional comments (7)
deploy/.env.example (1)
121-126: LGTM!src/aiq_agent/observability/monocle_exporter.py (2)
97-103: LGTM!
241-253: LGTM!src/aiq_agent/agents/chat_researcher/register.py (1)
34-34: LGTM!Also applies to: 59-59
frontends/cli/cli.py (2)
421-425: LGTM!
464-468: LGTM!pyproject.toml (1)
64-66: 📐 Maintainability & Code QualityCheck
uv.lockfor the new extra
If this repository commitsuv.lock, regenerate it with the new optional dependency so locked installs stay reproducible.
|
|
||
| [Monocle](https://github.com/monocle2ai/monocle) is an open-source, OpenTelemetry-based tracer for agentic applications. It instruments the frameworks already in use (LangChain, LangGraph, and others) and records each run end-to-end -- LLM calls, agent steps, and tool and MCP invocations, with their inputs, outputs, timings, and token counts. It is packaged as an **optional** tracing backend and is disabled unless you both install it and enable it -- the default install and default behavior are unchanged. | ||
|
|
||
| Each run writes one trace file to `.monocle/` in the working directory; open it in the [Monocle VS Code extension](https://marketplace.visualstudio.com/items?itemName=OkahuAI.monocle-apptrace) to inspect the span timeline and token counts. Connect to [Okahu](https://www.okahu.ai), an agent-observability platform, to analyze traces across runs and run trace-based and agentic evaluations (via the `okahu` exporter). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Scope the local trace-file claim to the file exporter.
The documented configuration permits console and remote-only exporter lists, which do not necessarily produce a local .monocle/ file. Clarify that the file is written when file is enabled.
Suggested wording
-Each run writes one trace file to `.monocle/` in the working directory; open it in the [Monocle VS Code extension](https://marketplace.visualstudio.com/items?itemName=OkahuAI.monocle-apptrace) to inspect the span timeline and token counts.
+When the `file` exporter is enabled, each run writes a trace file to `.monocle/` in the working directory; open it in the [Monocle VS Code extension](https://marketplace.visualstudio.com/items?itemName=OkahuAI.monocle-apptrace) to inspect the span timeline and token counts.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Each run writes one trace file to `.monocle/` in the working directory; open it in the [Monocle VS Code extension](https://marketplace.visualstudio.com/items?itemName=OkahuAI.monocle-apptrace) to inspect the span timeline and token counts. Connect to [Okahu](https://www.okahu.ai), an agent-observability platform, to analyze traces across runs and run trace-based and agentic evaluations (via the `okahu` exporter). | |
| When the `file` exporter is enabled, each run writes a trace file to `.monocle/` in the working directory; open it in the [Monocle VS Code extension](https://marketplace.visualstudio.com/items?itemName=OkahuAI.monocle-apptrace) to inspect the span timeline and token counts. Connect to [Okahu](https://www.okahu.ai), an agent-observability platform, to analyze traces across runs and run trace-based and agentic evaluations (via the `okahu` exporter). |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/source/deployment/observability.md` at line 262, Update the trace-file
statement near the Okahu exporter documentation to scope `.monocle/` file
creation to configurations with the `file` exporter enabled. Preserve the
existing guidance about opening local traces and using Okahu, while avoiding any
claim that console or remote-only exporters produce a local file.
Source: Path instructions
| pip install 'aiq-agent[monocle]' | ||
| ``` | ||
|
|
||
| Enabling Monocle without the extra installed fails fast with an actionable error naming the install command; when Monocle is not enabled, `monocle_apptrace` is never imported. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Qualify the optional-dependency fail-fast behavior.
If okahu is the only selected exporter and OKAHU_API_KEY is missing, runtime removes that exporter and skips Monocle before importing monocle_apptrace; therefore enabling the feature does not always fail fast when the extra is absent. State that the error occurs when initialization proceeds with a usable exporter.
Suggested wording
-Enabling Monocle without the extra installed fails fast with an actionable error naming the install command; when Monocle is not enabled, `monocle_apptrace` is never imported.
+When Monocle initialization proceeds without the extra installed, it fails fast with an actionable error naming the install command; configurations with no usable exporters are skipped before `monocle_apptrace` is imported.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/source/deployment/observability.md` at line 274, Update the
optional-dependency fail-fast statement in the Monocle observability
documentation to qualify that the actionable missing-extra error occurs only
when initialization proceeds with a usable exporter; note that selecting only
Okahu without OKAHU_API_KEY removes that exporter and skips the Monocle import.
Source: Path instructions
| # Manual mirror of monocle_apptrace's supported exporters, kept local so a typo | ||
| # fails fast with a clear message instead of an opaque upstream error. Update | ||
| # this tuple when a monocle_apptrace bump adds or renames an exporter. | ||
| _MONOCLE_EXPORTERS = ("file", "console", "okahu", "s3", "blob", "gcs") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n 'monocle_apptrace|_MONOCLE_EXPORTERS|_warn_off_box' pyproject.toml src/aiq_agent/observability/monocle_exporter.py
curl -fsSL \
https://raw.githubusercontent.com/monocle2ai/monocle/c46dcd62bcd59f03c2ce76ef8fbe62a47cde5550/apptrace/src/monocle_apptrace/exporters/monocle_exporters.py \
| sed -n '/monocle_exporters: Dict/,/def get_monocle_exporter/p'Repository: NVIDIA-AI-Blueprints/aiq
Length of output: 3099
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '55,130p' src/aiq_agent/observability/monocle_exporter.pyRepository: NVIDIA-AI-Blueprints/aiq
Length of output: 3340
Add the missing Monocle exporters
memory, otlp, and paygentic are valid Monocle 0.8.8 exporters; keeping them out of _MONOCLE_EXPORTERS rejects supported configs. Also exclude memory from the off-box warning so in-memory spans don't trigger an egress warning.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiq_agent/observability/monocle_exporter.py` around lines 59 - 62, Update
_MONOCLE_EXPORTERS to include the supported “memory”, “otlp”, and “paygentic”
exporters. Adjust the off-box warning logic to exclude “memory” alongside the
existing in-process exporter handling, while preserving warnings for exporters
that send spans externally.
2 participants
Summary
Adds Monocle observability to the agent. Monocle is an OpenTelemetry-based tracer for LLM applications. With it enabled, each run is recorded as a structured trace: the agent and graph invocations, tool calls, LLM inferences, token usage, and timings. The change is additive and does not alter application logic.
What this adds
Instrumentation is one setup call plus one dependency:
frontends/cli/pyproject.toml: addsmonocle_apptrace.frontends/cli/cli.py: callssetup_monocle_telemetry(workflow_name="nvidia-aiq", monocle_exporters_list="file")at startup.That call auto-instruments the frameworks already in use (LangGraph and the LLM clients), so there is no per-tool or per-call wiring to maintain. Traces are written to
.monocle/by default.What you get
Each run produces a trace of all the agents and tools that were triggered: which agent ran, which tools were called (
web_search_tool), and what LLM inferences happened. In effect, it's the path the agent took to answer a given question. This is useful for developers building the agent, since they can see how it actually behaved on a run. The same traces are also a good basis for a behavioral test suite, an integration test that asserts on that behavior, and I've opened a companion PR that shows how that works: behavioral test suite. You can open the trace files directly, view them in the Monocle VS Code extension, or send them to Okahu for analysis across many runs.Example trace (Okahu VS Code Extension)
The AI-Q (NAT) research agent driving the model and calling
web_search_tool, captured as trace spans.Summary by CodeRabbit
monocleoptional dependency extra (aiq-agent[monocle]).PS: if Monocle looks useful, a ⭐ helps the project (https://github.com/monocle2ai/monocle). And if you want to turn these traces into tests, that's the companion PR (#328).