feat: enhance tracing system with OpenTelemetry semantic conventions

[Pouyanpi]

PR Description

This PR introduces a comprehensive enhancement to the NeMo Guardrails tracing and telemetry infrastructure, providing improved observability, standardized telemetry formats, and privacy controls for production deployments.

Screenshots

Nemoguard NIMs with parallel input rails:

Example of Span attributes

High-Level Impact

Strategic Goals

• Standardization: Align with OpenTelemetry semantic conventions for GenAI applications
• Flexibility: Support multiple span formats to maintain backward compatibality
• Privacy: Implement controls to protect sensitive prompt/response data
• Observability: Enhanced tracking of LLM interactions and guardrail executions, fixed span timestamps and relationshipts etc.

Key Features

1. Configurable Span Formats

• Legacy Format: Lightweight, backward-compatible format with minimal overhead
• OpenTelemetry Format: Full compliance with OTel semantic conventions for GenAI
• Configuration via span_format field in TracingConfig

2. Privacy First Content Capture

• New enable_content_capture flag to control prompt/response recording
• Defaults to disabled for privacy protection
• Granular control over what gets recorded in telemetry

3. Enhanced LLM Provider Tracking

• Added model_name and model_provider parameters to LLM calls
• Improved attribution and debugging capabilities
• Better support for multi-provider deployments

Technical Implementation

Architecture Changes

New Core Components

• SpanFormat Enum: Type-safe span format definitions
• SpanExtractor Hierarchy: V1 (legacy) and V2 (OTel) extractors
• Span Models: Typed span representations (InteractionSpan, RailSpan, ActionSpan, LLMSpan)
• OTel Constants: Comprehensive semantic convention attributes

Enhanced Components

• Tracer: Now accepts span format and content capture configuration
• OpenTelemetryAdapter: Refactored for better v2 span handling

Data Flow

Generation Log → SpanExtractor → Typed Spans → Adapter → Telemetry Backend
                     ↓               ↓
                 (v1 or v2)    (legacy or OTel)

OpenTelemetry Semantic Conventions

Implements GenAI semantic conventions including:

• gen_ai.request.model
• gen_ai.request.max_tokens
• gen_ai.response.finish_reasons
• gen_ai.usage.input_tokens
• gen_ai.usage.output_tokens
• Custom guardrails attributes under guardrails.* namespace

Breaking Changes

None - Full backward compatibility maintained

Migration Guide

Existing deployments will continue to work without changes. To adopt new features:

# Enable OpenTelemetry format with content capture
config = RailsConfig(
    tracing={
        "span_format": "opentelemetry",  # or "legacy" for legacy
        "enable_content_capture": True    # defaults to False
    }
)

Testing

• ✅ 7 new test files with comprehensive coverage
• ✅ Tests for both v1 and v2 span formats
• ✅ OTel semantic convention compliance validation
• ✅ Mixed v1/v2 span handling
• ✅ Format conversion and validation

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.86166% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.59%. Comparing base (a5e8b4c) to head (be51f33).

Files with missing lines	Patch %	Lines
nemoguardrails/tracing/span_extractors.py	93.47%	9 Missing ⚠️
nemoguardrails/tracing/spans.py	94.44%	8 Missing ⚠️
nemoguardrails/tracing/adapters/filesystem.py	46.15%	7 Missing ⚠️
nemoguardrails/rails/llm/llmrails.py	66.66%	1 Missing ⚠️
nemoguardrails/tracing/span_formatting.py	96.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1331      +/-   ##
===========================================
+ Coverage    71.04%   71.59%   +0.54%     
===========================================
  Files          162      168       +6     
  Lines        16398    16861     +463     
===========================================
+ Hits         11650    12071     +421     
- Misses        4748     4790      +42     

Flag	Coverage Δ
python	71.59% <94.86%> (+0.54%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
nemoguardrails/actions/llm/utils.py	80.00% <100.00%> (+0.06%)	⬆️
nemoguardrails/logging/explain.py	60.00% <100.00%> (+1.17%)	⬆️
nemoguardrails/rails/llm/config.py	90.73% <100.00%> (+0.08%)	⬆️
nemoguardrails/tracing/__init__.py	100.00% <100.00%> (ø)
nemoguardrails/tracing/adapters/base.py	70.58% <100.00%> (ø)
nemoguardrails/tracing/adapters/opentelemetry.py	95.52% <100.00%> (+1.64%)	⬆️
nemoguardrails/tracing/constants.py	100.00% <100.00%> (ø)
nemoguardrails/tracing/interaction_types.py	100.00% <100.00%> (ø)
nemoguardrails/tracing/span_format.py	100.00% <100.00%> (ø)
nemoguardrails/tracing/tracer.py	100.00% <100.00%> (ø)
... and 5 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cparisien]

I'm good with this. I reviewed it from the perspective of understanding what we're doing and that it lines up with expectations. Tim is going into more depth on OTel and testing via Jaeger, I'll leave that to him.

[tgasser-nv]

Looks good! I have a couple of high-level questions.

• Are we restricted to only instrumenting LLM calls (do we do the same for API calls to third-party integration APIs?)
• The Gen AI Opentelemetry definitions are in development phase (pre-alpha). Is there any way to keep our definitions in sync with them over time? Or we just need to re-check every time we bump the Open Telemetry library.
• Do you have any rough idea the load of logging telemetry like this (i.e. how much CPU does it consume?)