feat: implement Epic 07 Phase 5: versioned cost accounting on usage_events, OTel metric instruments, and span-helper wiring - #170
Conversation
…vents, OTel metric instruments, and span-helper wiring
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughChangesObservability accounting
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Application
participant CostRegistry
participant PricingTable
participant UsageStore
participant UsageEvent
Application->>CostRegistry: initialize settings
CostRegistry->>PricingTable: load and validate pricing
PricingTable-->>CostRegistry: return pricing table
UsageStore->>CostRegistry: calculate usage cost
CostRegistry-->>UsageStore: return cost and pricing version
UsageStore->>UsageEvent: persist usage and cost
sequenceDiagram
participant LLMOrWorkflow
participant Tracing
participant MetricInstruments
participant OpenTelemetry
LLMOrWorkflow->>Tracing: complete request or workflow action
Tracing->>MetricInstruments: record metric data
MetricInstruments->>OpenTelemetry: emit normalized attributes and measurement
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend-python/app/ai/observability/cost/calculator.py`:
- Around line 34-46: Update CostCalculator.price() to return (None, None) when
usage.prompt_tokens or usage.completion_tokens is negative, before looking up
pricing or calculating cost. Preserve the existing handling for missing usage,
missing token values, and unavailable pricing entries.
In `@backend-python/app/ai/observability/cost/pricing.py`:
- Around line 121-128: Update the pricing validation in the model-loading logic
around the existing type and range checks to also reject non-finite numeric
values by applying math.isfinite(value). Preserve the current errors for invalid
and negative rates, and add loader tests covering both NaN and positive/negative
infinity inputs.
In `@backend-python/app/ai/observability/tracing/provider_wrapper.py`:
- Around line 34-39: Update the provider wrapper around complete_chat,
complete_chat_with_tools, and stream_chat so exceptions from each provider call
or stream iteration record record_llm_request_metrics with succeeded=False and
no usage before re-raising the original error; retain the existing successful
metrics path with usage.
In `@backend-python/app/ai/workflow/manager.py`:
- Around line 525-526: Update cancel_run to
record_workflow_approval_pending_delta(-1) after the cancellation checkpoint
succeeds when the run’s prior status is WAITING_APPROVAL, while preserving
behavior for other statuses. Add a cancellation test covering a WAITING_APPROVAL
run and verifying the pending-approval count is decremented.
In `@backend-python/app/db/usage.py`:
- Around line 57-62: Move the record_llm_cost_metric call from before
self._session.flush() to after persistence succeeds, ensuring duplicate usage
events cannot emit a cost metric. In the transaction-owned path, defer emission
until the transaction commits; preserve the existing provider, model, and
cost_usd values.
In `@backend-python/config/model_pricing.yaml`:
- Around line 11-14: Update the gemini-3.1-flash-lite pricing entry in
model_pricing.yaml to use Google’s standard rates: $0.00025 input per 1K tokens
and $0.0015 output per 1K tokens.
In `@backend-python/tests/ai/observability/test_metrics_instruments.py`:
- Around line 51-56: Update the metric_reader fixture around
MetricInstruments.initialize so the OpenTelemetry test MeterProvider is
installed only once at module or session scope, rather than replacing the global
provider per fixture instance. Ensure each test’s InMemoryMetricReader still
observes emitted metrics, either by sharing the scoped provider/reader or by
injecting the provider through MeterRegistry.
In `@backend-python/tests/ai/observability/test_usage_cost_persistence.py`:
- Around line 188-194: Update the pricing setup around ModelPricingTable.load
and CostRegistry._calculator to load a temporary pricing file with a new pricing
version and modified rate instead of reloading _CANONICAL_PRICING unchanged.
Assert that second uses the updated cost and pricing version, while first
retains the original cost and version.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a44141ed-1793-4f3a-82f7-d7231c27034d
📒 Files selected for processing (22)
backend-python/.env.examplebackend-python/alembic/versions/0008_observability_usage_cost.pybackend-python/app/ai/observability/__init__.pybackend-python/app/ai/observability/cost/__init__.pybackend-python/app/ai/observability/cost/calculator.pybackend-python/app/ai/observability/cost/pricing.pybackend-python/app/ai/observability/metrics/instruments.pybackend-python/app/ai/observability/metrics/labels.pybackend-python/app/ai/observability/tracing/provider_wrapper.pybackend-python/app/ai/observability/tracing/spans.pybackend-python/app/ai/tools/executor.pybackend-python/app/ai/workflow/engine/executor.pybackend-python/app/ai/workflow/manager.pybackend-python/app/core/config.pybackend-python/app/db/models.pybackend-python/app/db/usage.pybackend-python/app/main.pybackend-python/config/model_pricing.yamlbackend-python/tests/ai/observability/test_cost_calculator.pybackend-python/tests/ai/observability/test_metrics_instruments.pybackend-python/tests/ai/observability/test_package_init.pybackend-python/tests/ai/observability/test_usage_cost_persistence.py
| if usage is None: | ||
| return None, None | ||
| if usage.prompt_tokens is None or usage.completion_tokens is None: | ||
| return None, None | ||
|
|
||
| entry = self._table.lookup(provider, model) | ||
| if entry is None: | ||
| return None, None | ||
|
|
||
| cost = (usage.prompt_tokens / 1000.0) * entry.input_usd_per_1k + ( | ||
| usage.completion_tokens / 1000.0 | ||
| ) * entry.output_usd_per_1k | ||
| return cost, self._table.pricing_version |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
ast-grep outline backend-python/app/providers/base.py --match ProviderUsage --view expanded
rg -n -C 5 'class ProviderUsage|prompt_tokens|completion_tokens' backend-python/app/providers/base.pyRepository: pateatlau/fullstack-ai-platform
Length of output: 1060
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Calculator and SqlUsageStore references:"
rg -n -C 4 'calculate.*cost|record\(|record_usage|cost.*usage|ProviderUsage|prompt_tokens|completion_tokens' backend-python/app/ai backend-python/app | head -n 240
echo
echo "Candidate files:"
git ls-files backend-python/app | rg '(calculator|usage|store|base|provider)'Repository: pateatlau/fullstack-ai-platform
Length of output: 20778
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "calculator.py"
cat -n backend-python/app/ai/observability/cost/calculator.py
echo
echo "usage_service.py"
cat -n backend-python/app/services/usage_service.py
echo
echo "db usage models"
fd -a 'usage.py|tables.py|schema.py|models.py' backend-python/app/db | while read -r f; do
echo "--- $f"
wc -l "$f"
rg -n -C 4 'class .*Usage|Usage|input_tokens|completion_tokens|prompt_tokens|price|cost|record' "$f" || true
done
echo
echo "observability usages"
rg -n -C 5 'calculate|record_usage|SqlUsageStore|UsageStore|ProviderUsage' backend-python/app/services backend-python/app/ai backend-python/app/dbRepository: pateatlau/fullstack-ai-platform
Length of output: 35777
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "ChatService usage recording slices:"
sed -n '792,850p' backend-python/app/services/chat_service.py
sed -n '1040,1105p' backend-python/app/services/chat_service.py
sed -n '170,225p' backend-python/app/ai/agent/adapters/chat_adapter.py
echo
echo "SqlUsageStore.record bounded slice:"
sed -n '28,95p' backend-python/app/db/usage.py
echo
echo "Behavioral probe of current bounds and calculation:"
python3 - <<'PY'
from dataclasses import dataclass
from typing import Any
`@dataclass`(frozen=True)
class PricingEntry:
input_usd_per_1k: float = 0.0
output_usd_per_1k: float = 0.0
`@dataclass`(frozen=True)
class ModelPricingTable:
data: dict[tuple[str, str], PricingEntry]
pricing_version = "probe"
def lookup(self, provider: str, model: str) -> PricingEntry | None:
return self.data.get((provider, model))
class CostCalculator:
def __init__(self, table: ModelPricingTable) -> None:
self._table = table
def price(self, provider: str, model: str, usage: Any | None) -> tuple[float | None, str | None]:
if usage is None:
return None, None
if usage.prompt_tokens is None or usage.completion_tokens is None:
return None, None
entry = self._table.lookup(provider, model)
if entry is None:
return None, None
cost = (usage.prompt_tokens / 1000.0) * entry.input_usd_per_1k + (
usage.completion_tokens / 1000.0
) * entry.output_usd_per_1k
return cost, self._table.pricing_version
ProviderUsage = type("ProviderUsage", (), {})
ProviderUsage = dataclass(frozen=True)(type(
"ProviderUsage", (), {
"__annotations__": {"prompt_tokens": "int | None", "completion_tokens": "int | None", "total_tokens": "int | None"}
}
))
ProviderUsage.prompt_tokens = None
ProviderUsage.completion_tokens = None
ProviderUsage.total_tokens = None
calculator = CostCalculator(ModelPricingTable({("p", "m"): PricingEntry(1.0, 2.0)}))
for tokens in [
(10, 5),
(0, 0),
(-1, 5),
(10, -2),
(-3, -4),
]:
usage = ProviderUsage(prompt_tokens=tokens[0], completion_tokens=tokens[1], total_tokens=None)
print(tokens, calculator.price("p", "m", usage))
PYRepository: pateatlau/fullstack-ai-platform
Length of output: 9296
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "ChatService surrounding error handling:"
sed -n '830,915p' backend-python/app/services/chat_service.py
sed -n '220,345p' backend-python/app/ai/agent/adapters/chat_adapter.py
echo
echo "Simple arithmetic probe:"
python3 - <<'PY'
def current_price(prompt_tokens, completion_tokens, in_price, out_price):
if prompt_tokens is None or completion_tokens is None:
return None
return (prompt_tokens / 1000.0) * in_price + (completion_tokens / 1000.0) * out_price
for prompt_tokens, completion_tokens in [
(10, 5),
(0, 0),
(-1, 5),
(10, -2),
(-3, -4),
]:
print(f"input={prompt_tokens}, completion={completion_tokens} -> cost={current_price(prompt_tokens, completion_tokens, 1.0, 2.0)}")
PYRepository: pateatlau/fullstack-ai-platform
Length of output: 6014
Reject negative usage before calculating cost.
ProviderUsage accepts negative integer token counts, and SqlUsageStore.record() persists those inputs into usage_events; valid pricing rows would then produce a negative cost_usd. Reject prompt_tokens < 0 or completion_tokens < 0 in CostCalculator.price(), or return (None, None).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/app/ai/observability/cost/calculator.py` around lines 34 - 46,
Update CostCalculator.price() to return (None, None) when usage.prompt_tokens or
usage.completion_tokens is negative, before looking up pricing or calculating
cost. Preserve the existing handling for missing usage, missing token values,
and unavailable pricing entries.
| if not isinstance(value, (int, float)) or isinstance(value, bool): | ||
| raise ObservabilityConfigError( | ||
| f"models[{index}].{field_name} must be a finite number >= 0." | ||
| ) | ||
| if value < 0: | ||
| raise ObservabilityConfigError( | ||
| f"models[{index}].{field_name} must be >= 0." | ||
| ) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Reject non-finite pricing rates.
.nan and .inf pass these checks because both are float values and neither is less than zero. The loader then accepts invalid rates despite the error text requiring finite values. This can persist invalid cost_usd values and emit invalid cost metrics.
Use math.isfinite(value) in this condition. Add .nan and .inf loader test cases.
Proposed fix
+import math
+
- if not isinstance(value, (int, float)) or isinstance(value, bool):
+ if (
+ not isinstance(value, (int, float))
+ or isinstance(value, bool)
+ or not math.isfinite(value)
+ ):
raise ObservabilityConfigError(
f"models[{index}].{field_name} must be a finite number >= 0."
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if not isinstance(value, (int, float)) or isinstance(value, bool): | |
| raise ObservabilityConfigError( | |
| f"models[{index}].{field_name} must be a finite number >= 0." | |
| ) | |
| if value < 0: | |
| raise ObservabilityConfigError( | |
| f"models[{index}].{field_name} must be >= 0." | |
| ) | |
| import math | |
| if ( | |
| not isinstance(value, (int, float)) | |
| or isinstance(value, bool) | |
| or not math.isfinite(value) | |
| ): | |
| raise ObservabilityConfigError( | |
| f"models[{index}].{field_name} must be a finite number >= 0." | |
| ) | |
| if value < 0: | |
| raise ObservabilityConfigError( | |
| f"models[{index}].{field_name} must be >= 0." | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/app/ai/observability/cost/pricing.py` around lines 121 - 128,
Update the pricing validation in the model-loading logic around the existing
type and range checks to also reject non-finite numeric values by applying
math.isfinite(value). Preserve the current errors for invalid and negative
rates, and add loader tests covering both NaN and positive/negative infinity
inputs.
| record_llm_request_metrics( | ||
| provider=provider, | ||
| model=model, | ||
| succeeded=succeeded, | ||
| total_tokens=usage.total_tokens if usage is not None else None, | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Record failed provider requests.
Line 34 runs only after the inner provider completes. If complete_chat, complete_chat_with_tools, or stream_chat raises, the code emits no failed llm_requests_total metric. Catch provider errors around each inner call or iteration, record succeeded=False with no usage, then re-raise the error.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/app/ai/observability/tracing/provider_wrapper.py` around lines
34 - 39, Update the provider wrapper around complete_chat,
complete_chat_with_tools, and stream_chat so exceptions from each provider call
or stream iteration record record_llm_request_metrics with succeeded=False and
no usage before re-raising the original error; retain the existing successful
metrics path with usage.
| record_workflow_approval_pending_delta(-1) | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Decrement pending approvals on cancellation.
Line 525 handles approval decisions only. cancel_run can change a WAITING_APPROVAL run to CANCELLED, but it does not record record_workflow_approval_pending_delta(-1). Each such cancellation leaves workflow_approval_pending_count elevated. Decrement after the cancellation checkpoint succeeds when the prior status was WAITING_APPROVAL, and add a cancellation test.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/app/ai/workflow/manager.py` around lines 525 - 526, Update
cancel_run to record_workflow_approval_pending_delta(-1) after the cancellation
checkpoint succeeds when the run’s prior status is WAITING_APPROVAL, while
preserving behavior for other statuses. Add a cancellation test covering a
WAITING_APPROVAL run and verifying the pending-approval count is decremented.
| if cost_usd is not None: | ||
| record_llm_cost_metric( | ||
| provider=provider, | ||
| model=model, | ||
| cost_usd=cost_usd, | ||
| ) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Emit the cost metric only after successful persistence.
record_llm_cost_metric runs before await self._session.flush(). If a retry supplies an existing request_id, the unique usage-event constraint rejects the row after this metric has already been emitted. This overcounts LLM cost while the database correctly prevents duplicate usage events.
Move metric emission after a successful flush. If the caller owns the transaction, emit after commit through the transaction lifecycle.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/app/db/usage.py` around lines 57 - 62, Move the
record_llm_cost_metric call from before self._session.flush() to after
persistence succeeds, ensuring duplicate usage events cannot emit a cost metric.
In the transaction-owned path, defer emission until the transaction commits;
preserve the existing provider, model, and cost_usd values.
| - provider: gemini | ||
| model: gemini-3.1-flash-lite | ||
| input_usd_per_1k: 0.000075 | ||
| output_usd_per_1k: 0.0003 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Correct the Gemini 3.1 Flash-Lite standard rates.
These values equal $0.075 input and $0.30 output per 1M tokens. Google lists standard rates of $0.25 input and $1.50 output per 1M tokens. The current values understate persisted and emitted cost by 3.33× for input and 5× for output. (ai.google.dev)
Proposed fix
- input_usd_per_1k: 0.000075
- output_usd_per_1k: 0.0003
+ input_usd_per_1k: 0.00025
+ output_usd_per_1k: 0.0015📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - provider: gemini | |
| model: gemini-3.1-flash-lite | |
| input_usd_per_1k: 0.000075 | |
| output_usd_per_1k: 0.0003 | |
| - provider: gemini | |
| model: gemini-3.1-flash-lite | |
| input_usd_per_1k: 0.00025 | |
| output_usd_per_1k: 0.0015 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/config/model_pricing.yaml` around lines 11 - 14, Update the
gemini-3.1-flash-lite pricing entry in model_pricing.yaml to use Google’s
standard rates: $0.00025 input per 1K tokens and $0.0015 output per 1K tokens.
| reader = InMemoryMetricReader() | ||
| provider = MeterProvider(metric_readers=[reader]) | ||
| metrics.set_meter_provider(provider) | ||
| MeterRegistry._initialized = True | ||
| MeterRegistry._enabled = True | ||
| MetricInstruments.initialize() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate files =="
git ls-files | rg '(^backend-python/tests/ai/observability/test_metrics_instruments\.py$|metrics|meter|instruments)'
echo "== diff/stat =="
git diff --stat || true
git diff -- backend-python/tests/ai/observability/test_metrics_instruments.py | sed -n '1,220p' || true
echo "== target file outline/section =="
wc -l backend-python/tests/ai/observability/test_metrics_instruments.py || true
sed -n '1,140p' backend-python/tests/ai/observability/test_metrics_instruments.py
echo "== MeterRegistry/initialize references =="
rg -n "class MeterRegistry|def initialize|MeterRegistry|metrics\.set_meter_provider|MeterProvider|InMemoryMetricReader|set_meter_provider" backend-python -SRepository: pateatlau/fullstack-ai-platform
Length of output: 11638
🌐 Web query:
OpenTelemetry Python metrics set_meter_provider accepts first global provider
💡 Result:
In OpenTelemetry Python, the set_meter_provider function is designed to set the global MeterProvider only once [1][2]. If you attempt to call it multiple times, the library will log a warning indicating that overriding the current MeterProvider is not allowed [1][2][3]. Internally, this mechanism is enforced using a "set-once" lock or check [1]. When set_meter_provider is called for the first time, the provided MeterProvider is stored as the global instance [1]. Any subsequent calls are ignored in terms of changing the global provider, and the library issues a warning [1][2][3]. This ensures that the global configuration remains consistent throughout the application lifecycle.
Citations:
- 1: https://opentelemetry-python.readthedocs.io/en/stable/%5Fmodules/opentelemetry/metrics/%5Finternal.html
- 2: https://opentelemetry-python.readthedocs.io/en/latest/api/metrics.html
- 3: Added a warning log when the existing TracerProvider and MeterProvider are overridden open-telemetry/opentelemetry-python#856
🌐 Web query:
OpenTelemetry Python metrics NoOpMeterProvider instruments get_meter
💡 Result:
In the OpenTelemetry Python API, the NoOpMeterProvider is the default implementation provided when no other meter provider is configured [1][2]. Its purpose is to allow applications to interact with the OpenTelemetry metrics API without requiring a fully operational SDK, thereby avoiding telemetry production and minimizing computational overhead [1][3][4]. When you call get_meter(name, version=None, schema_url=None, attributes=None) on a NoOpMeterProvider, it returns a NoOpMeter [1][2]. The NoOpMeter, in turn, provides "no-op" versions of instruments (such as NoOpCounter, NoOpHistogram, etc.) when its create_* methods are called [3][5]. These instrument objects perform no operations, hold no state, and do not aggregate or emit any telemetry data [5][4]. Key characteristics of this behavior include: 1. No Validation: Consistent with the OpenTelemetry specification, the NoOpMeterProvider accepts the provided parameters (name, version, schema_url, attributes) but does not validate them [4]. 2. No State: Neither the NoOpMeterProvider nor the instruments it creates hold any configuration or operational state, ensuring a minimal memory footprint [4]. 3. Silence: To prevent interference with application logic, the NoOpMeterProvider and its associated objects do not produce errors or log messages during operations [4]. In many production scenarios, the global default provider is actually a _ProxyMeterProvider, which buffers instrumentation calls until a concrete SDK implementation is loaded and set via set_meter_provider, at which point it upgrades the buffered instruments to real, operational ones [5]. The NoOpMeterProvider is typically reserved for instances where no telemetry processing is desired or explicitly configured [1][5].
Citations:
- 1: https://opentelemetry-python.readthedocs.io/en/latest/api/metrics.html
- 2: https://github.com/open-telemetry/opentelemetry-python/blob/eed100c3/opentelemetry-api/src/opentelemetry/metrics/_internal/__init__.py
- 3: https://github.com/open-telemetry/opentelemetry-python/blob/eed100c3/opentelemetry-api/src/opentelemetry/metrics/__init__.py
- 4: https://opentelemetry.io/docs/specs/otel/metrics/noop/
- 5: https://deepwiki.com/open-telemetry/opentelemetry-python/4-metrics-system
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== pyproject dependency versions =="
for f in backend-python/pyproject.toml pyproject.toml backend-python/Pipfile backend-python/Pipfile.lock; do
[ -f "$f" ] && { echo "-- $f"; sed -n '/opentelemetry/p;/otel/p;/metrics/p' "$f" || true; }
done
echo "== instrument/meter implementations =="
wc -l backend-python/app/ai/observability/metrics/instruments.py backend-python/app/ai/observability/metrics/meter.py
sed -n '1,180p' backend-python/app/ai/observability/metrics/meter.py
sed -n '1,160p' backend-python/app/ai/observability/metrics/instruments.pyRepository: pateatlau/fullstack-ai-platform
Length of output: 8081
🌐 Web query:
"proxy meter provider" "overriding" "set_meter_provider" OpenTelemetry Python metrics
💡 Result:
In OpenTelemetry Python, the set_meter_provider function is designed to set the global MeterProvider only once [1][2]. If you attempt to call set_meter_provider again, it will log a warning and ignore the subsequent attempt [1][3]. When you call set_meter_provider for the first time, it interacts with a internal _ProxyMeterProvider [1][4]. The proxy mechanism exists to allow instrumentation libraries to create meters before a concrete SDK provider has been configured [5]. Here is how the override process and proxy interaction work: 1. Initial State: Before set_meter_provider is called, get_meter_provider returns a _ProxyMeterProvider [1][4]. Any meters or instruments created at this stage are "proxy" objects [4][5]. 2. Setting the Provider: When you call set_meter_provider with a concrete SDK MeterProvider, the _ProxyMeterProvider is notified via its on_set_meter_provider method [1][4]. 3. Resolving Proxies: Upon receiving the concrete provider, the proxy informs its existing _ProxyMeters and proxy instruments to switch to the "real" backing objects provided by the SDK [1][4]. This ensures that telemetry recorded via initial proxies is correctly routed to the configured SDK [5]. 4. Overriding Constraints: Because the API intentionally enforces a single-assignment policy for the global provider, you cannot programmatically override the MeterProvider once it has been set [1][2]. Attempts to do so result in a warning being logged [1][3]. If you find yourself needing to change the provider, this is generally considered an anti-pattern in the current OpenTelemetry Python design, as seen in ongoing community discussions regarding the inability to reset global providers [6]. If your application architecture requires dynamic configuration, you should configure the provider before any global telemetry initialization occurs, or use a custom implementation that handles configuration changes internally (such as a wrapper provider) rather than relying on the OpenTelemetry global API for runtime re-configuration.
Citations:
- 1: https://github.com/open-telemetry/opentelemetry-python/blob/eed100c3/opentelemetry-api/src/opentelemetry/metrics/_internal/__init__.py
- 2: https://opentelemetry-python.readthedocs.io/en/latest/api/metrics.html
- 3: https://opentelemetry-python.readthedocs.io/en/latest/%5Fmodules/opentelemetry/metrics/%5Finternal.html
- 4: https://github.com/open-telemetry/opentelemetry-python/blob/eed100c3/opentelemetry-api/tests/metrics/test_meter_provider.py
- 5:
ProxyMetermust return proxy instruments or users may get stale instruments in the API open-telemetry/opentelemetry-python#2144 - 6: Allow resetting global providers open-telemetry/opentelemetry-python#4557
Install the test meter provider once.
metrics.set_meter_provider sets the global provider only once. Replacing it in each metric_reader fixture makes later tests continue to use the first provider’s meters, so their InMemoryMetricReader does not receive emitted metrics. Use one module- or session-scoped provider, or inject the provider through MeterRegistry.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/tests/ai/observability/test_metrics_instruments.py` around
lines 51 - 56, Update the metric_reader fixture around
MetricInstruments.initialize so the OpenTelemetry test MeterProvider is
installed only once at module or session scope, rather than replacing the global
provider per fixture instance. Ensure each test’s InMemoryMetricReader still
observes emitted metrics, either by sharing the scoped provider/reader or by
injecting the provider through MeterRegistry.
| settings = Settings( | ||
| openai_api_key="test-key", | ||
| observability_enabled=True, | ||
| observability_cost_pricing_version="2026-08", | ||
| ) | ||
| table_v2 = ModelPricingTable.load(settings, pricing_file=_CANONICAL_PRICING) | ||
| CostRegistry._calculator = CostCalculator(table_v2) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Exercise an actual pricing-table change.
This reloads the same file with the same required version, 2026-08. table_v2 therefore has the same prices and pricing version as the first calculator.
Load a temporary pricing file with a new version and changed rate. Assert that second has the new cost and version while first retains its original values.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend-python/tests/ai/observability/test_usage_cost_persistence.py` around
lines 188 - 194, Update the pricing setup around ModelPricingTable.load and
CostRegistry._calculator to load a temporary pricing file with a new pricing
version and modified rate instead of reloading _CANONICAL_PRICING unchanged.
Assert that second uses the updated cost and pricing version, while first
retains the original cost and version.
1 participant
Summary
Implements Epic 07 Phase 5 — Token & Cost Metrics: approximate, versioned dollar cost on
usage_events, plus the OTel counters/histograms Epic 06 pre-declared (and LLM/tool/agent equivalents).ModelPricingTableloads git-trackedconfig/model_pricing.yamlwith startup version lock (observability_cost_pricing_version)CostCalculatorpricesProviderUsageinsideSqlUsageStore.record()only — unknown models →NULL, never blocking0008_observability_usage_costadds nullablecost_usd/pricing_versionand index(provider, model, created_at)normalize_metric_label(); wired through span helpers /TracingLLMProvider/ workflow manager+executor (independent of trace sampling)NULLTest plan
uv run pytest tests/ai/observability/ -q(78 passed)pre-commit run --all-filesuv run alembic upgrade headon target DB before deployMigration
Summary by CodeRabbit