Skip to content

Commit 3617397

Browse files
doudouOUCqwencoderwenshao
authored
feat(core): Align GenAI telemetry with ARMS (#7536)
* feat(core): align GenAI telemetry with ARMS Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(core): remove estimated token usage splits Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(core): address GenAI telemetry review feedback Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> --------- Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com>
1 parent 947b632 commit 3617397

30 files changed

Lines changed: 1890 additions & 173 deletions
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# GenAI and ARMS field alignment
2+
3+
## Scope and standards baseline
4+
5+
This design aligns the first set of Qwen Code span attributes whose names,
6+
types, and meanings agree between OpenTelemetry GenAI semantic conventions and
7+
Alibaba Cloud ARMS LLM Trace. It does not change span names, span kinds,
8+
parenting, retry topology, or sensitive payload collection.
9+
10+
The OpenTelemetry GenAI convention is still Development status. This change is
11+
pinned to commit
12+
[`2e994c6d59a93bb4fc1752c5378eedb9b8e14d6b`](https://github.com/open-telemetry/semantic-conventions-genai/tree/2e994c6d59a93bb4fc1752c5378eedb9b8e14d6b):
13+
14+
- [Inference spans](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/2e994c6d59a93bb4fc1752c5378eedb9b8e14d6b/docs/gen-ai/gen-ai-spans.md)
15+
- [Agent spans](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/2e994c6d59a93bb4fc1752c5378eedb9b8e14d6b/docs/gen-ai/gen-ai-agent-spans.md)
16+
- [GenAI registry](https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/2e994c6d59a93bb4fc1752c5378eedb9b8e14d6b/model/gen-ai/registry.yaml)
17+
18+
The ARMS baseline is [LLM Trace field definitions](https://help.aliyun.com/zh/arms/application-monitoring/developer-reference/llm-trace-field-definition-description).
19+
An upgrade to either baseline requires regenerating and reviewing this matrix.
20+
21+
## Field contract
22+
23+
| Span | Standard attributes emitted in this phase | Source and omission rule |
24+
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
25+
| LLM | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, `gen_ai.request.model` | Written at span creation. Conversation ID is the existing session ID. |
26+
| LLM response | `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons` | Provider response data only. Missing response model is omitted rather than replaced with the request model. All candidate finish reasons are ordered by candidate index. |
27+
| LLM output | `gen_ai.output.type` | Gemini and Vertex AI only, and only when an explicit response MIME type or one unambiguous response modality is sent on the wire. |
28+
| LLM usage | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens` | Only provider-reported non-negative safe integers. Explicit zero is retained. When only a total is reported, input/output are omitted instead of estimated. |
29+
| Tool | `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`, `gen_ai.tool.type=function`, `gen_ai.tool.call.id` | Tool call ID prefers the provider/model ID and falls back to Qwen Code's internal ID. |
30+
| Agent | `gen_ai.operation.name=invoke_agent`, `gen_ai.agent.name`, `gen_ai.agent.description`, `gen_ai.conversation.id`, optional `gen_ai.request.model` | Description uses the existing 1024-UTF-16-code-unit truncation threshold and never splits surrogate pairs. Internal invocation IDs remain private. |
31+
32+
Private attributes without an exact standard equivalent remain available for
33+
compatibility. Exact-equivalent private aliases and invalid GenAI aliases are
34+
removed without a dual-write period:
35+
36+
| Removed attribute | Replacement |
37+
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
38+
| LLM `qwen-code.model` | `gen_ai.request.model`; interaction spans continue using `qwen-code.model` because they are not GenAI inference spans |
39+
| LLM `response_id` | `gen_ai.response.id`; API response/error logs retain their existing `response_id` schema |
40+
| LLM `input_tokens` | `gen_ai.usage.input_tokens` when the provider reports an input breakdown |
41+
| LLM `output_tokens` | `gen_ai.usage.output_tokens` when the provider reports an output breakdown |
42+
| LLM `cached_input_tokens` | `gen_ai.usage.cache_read.input_tokens` when the provider reports cache reads |
43+
| `qwen-code.tool` Span `tool.name` | `gen_ai.tool.name`; blocked-on-user and hook spans continue using `tool.name` |
44+
| `gen_ai.usage.cached_tokens` | `gen_ai.usage.cache_read.input_tokens` when the provider reports cache reads |
45+
| `gen_ai.server.time_to_first_token` | No common attribute; continue querying private `ttft_ms` |
46+
| `gen_ai.usage.reasoning_tokens` | No ARMS/GenAI common attribute in this baseline; continue querying private `thoughts_token_count` |
47+
48+
## Provider and operation resolution
49+
50+
Resolution is a pure function over the effective content-generator config. It
51+
never returns a URL, credential, arbitrary proxy hostname, or a value inferred
52+
from the model name.
53+
54+
1. Qwen OAuth and an exact `DASHSCOPE_PROXY_BASE_URL` match resolve to
55+
`dashscope`.
56+
2. A boundary-safe hostname match recognizes Alibaba Model Studio endpoints and
57+
internal Alibaba gateways, Azure OpenAI, and the supported third-party
58+
endpoints (DeepSeek, xAI, Mistral, MiniMax, Z.AI, ModelScope, MiMo,
59+
OpenRouter, and Requesty).
60+
3. If the host is unknown, a known `apiKeyEnvKey` identifies the configured
61+
provider. Host identity wins on conflict.
62+
4. Unknown endpoints fall back to the protocol provider: `openai`,
63+
`anthropic`, `gcp.gemini`, or `gcp.vertex_ai`.
64+
65+
OpenAI-compatible, Anthropic, and Qwen OAuth requests use operation `chat`.
66+
Gemini and Vertex AI requests use `generate_content`.
67+
68+
## Response and usage provenance
69+
70+
Provider converters attach internal provenance to normalized Gemini usage
71+
objects with a `WeakMap`. It records whether a cache-read field was actually
72+
present and Anthropic cache-creation tokens. This preserves the public response
73+
JSON shape and lets garbage collection follow the normalized usage object.
74+
75+
When an OpenAI-compatible provider reports only `total_tokens`, the normalized
76+
total remains available to existing internal consumers, but no input/output
77+
split is synthesized and neither standard usage attribute is emitted.
78+
79+
OpenAI `response.model`/`chunk.model` and Anthropic message model are preserved
80+
as `modelVersion`. A missing provider model remains missing for tracing;
81+
request-model fallback remains limited to existing API logs and UI behavior.
82+
Stream merging carries the last known provider model and usage provenance into
83+
the terminal response. Anthropic `message_start` input and cache usage is
84+
attached to the first subsequent yielded chunk so partial stream failures retain
85+
provider-reported usage without synthesizing an output count.
86+
87+
## ARMS configuration
88+
89+
ARMS automatic GenAI application recognition requires this resource attribute:
90+
91+
```json
92+
{
93+
"telemetry": {
94+
"resourceAttributes": {
95+
"acs.arms.service.feature": "genai_app"
96+
}
97+
}
98+
}
99+
```
100+
101+
Qwen Code does not inject that vendor-specific resource attribute or
102+
`gen_ai.span.kind`. ARMS can infer LLM, Tool, and Agent roles from
103+
`gen_ai.operation.name`.
104+
105+
## Deferred work
106+
107+
- Request sampling, choice, maximum-output, and stop fields need hooks after
108+
provider adapters finalize the wire request.
109+
- `seed` and `top_k` have incompatible ARMS and GenAI types in the baselines.
110+
- Messages, instructions, tool definitions, arguments, and results require a
111+
standard JSON schema, privacy controls, and payload caps.
112+
- Embedding needs a correct requested-model lifecycle before tracing.
113+
- ARMS time-to-first-token and OpenTelemetry time-to-first-chunk differ in name,
114+
unit, and meaning, so private `ttft_ms` remains authoritative.
115+
- Full GenAI span naming, CLIENT span kind, and logical retry topology are a
116+
separate compliance project.

docs/design/telemetry-llm-request-timing-design.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
11
# LLM Request Timing Decomposition Design (P3 Phase 4)
22

3+
> **GenAI attribute migration:**
4+
> [`gen-ai-arms-field-alignment.md`](./gen-ai-arms-field-alignment.md) supersedes
5+
> this document's `gen_ai.usage.cached_tokens`,
6+
> `gen_ai.server.time_to_first_token`, and
7+
> `gen_ai.usage.reasoning_tokens` aliases, and replaces the LLM Span's
8+
> `qwen-code.model`, `input_tokens`, `output_tokens`, and
9+
> `cached_input_tokens` aliases with standard GenAI attributes. The private
10+
> timing and reasoning attributes described here remain valid.
11+
312
> Issue #3731 — Phase 4 of hierarchical session tracing. Adds time-to-first-token, request-setup duration, sampling duration, and per-attempt retry telemetry to the `qwen-code.llm_request` span so operators can answer "why was this LLM call slow?" without guessing.
413
>
514
> Builds on Phase 1 (#4126), Phase 1.5 (#4302), Phase 2 (#4321). Independent of Phase 3 (#4410, in review) — recommended to land Phase 3 first so Phase 4's per-attempt fields aggregate cleanly under subagent subtrees.

docs/design/telemetry-subagent-spans-design.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# Subagent Trace Tree Design (P3 Phase 3)
22

3+
> **GenAI attribute migration:**
4+
> [`gen-ai-arms-field-alignment.md`](./gen-ai-arms-field-alignment.md) supersedes
5+
> this document's use of `gen_ai.provider.name=qwen-code` and the temporary
6+
> `gen_ai.agent.id`. The `qwen-code.subagent.*` lifecycle, identity, parenting,
7+
> and linking design described here remains valid.
8+
39
> Issue #3731 — Phase 3 of hierarchical session tracing. Adds a `qwen-code.subagent` span so subagent invocations get isolated, queryable trace structure instead of interleaving silently under the parent `qwen-code.interaction` span.
410
>
511
> Builds on Phase 1 (#4126), Phase 1.5 (#4302), and Phase 2 (#4321).

docs/developers/development/telemetry.md

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -822,10 +822,12 @@ Distributed tracing spans form a tree rooted at `qwen-code.interaction`. Each in
822822
- **Attributes**: `session.id`, `qwen-code.prompt_id`, `qwen-code.message_type`, `qwen-code.model`, `qwen-code.approval_mode`, `interaction.sequence`, `interaction.duration_ms`, `qwen-code.turn_status` ("ok"/"error"/"cancelled")
823823

824824
- `qwen-code.llm_request`: Wraps a single LLM API call.
825-
- **Attributes**: `session.id`, `qwen-code.model`, `qwen-code.prompt_id`, `llm_request.context` ("subagent"/"interaction"/"standalone"), `gen_ai.request.model`, `duration_ms`, `input_tokens`, `output_tokens`, `cached_input_tokens`, `ttft_ms`, `request_setup_ms`, `attempt`, `retry_total_delay_ms`, `sampling_ms`, `output_tokens_per_second`, `success`, `error`, `response_id`, `finish_reason`, `thoughts_token_count`, `subagent_name`, `error_type`, `error_status_code`
825+
- **GenAI attributes**: `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, `gen_ai.request.model`, optional `gen_ai.output.type`, `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens`
826+
- **Compatibility attributes**: `session.id`, `qwen-code.prompt_id`, `llm_request.context` ("subagent"/"interaction"/"standalone"), `duration_ms`, `ttft_ms`, `request_setup_ms`, `attempt`, `retry_total_delay_ms`, `sampling_ms`, `output_tokens_per_second`, `success`, `error`, `finish_reason`, `thoughts_token_count`, `subagent_name`, `error_type`, `error_status_code`
827+
- Standard response fields come from the provider response. Standard token fields are emitted only for provider-reported non-negative safe integers. If the provider reports only a total token count, input/output usage is omitted rather than estimated.
826828

827829
- `qwen-code.tool`: Wraps the full tool lifecycle (approval wait + execution).
828-
- **Attributes**: `session.id`, `tool.name`, `duration_ms`, `success`, `error`
830+
- **Attributes**: `session.id`, `gen_ai.operation.name` (`execute_tool`), `gen_ai.tool.name`, `gen_ai.tool.type` (`function`), `gen_ai.tool.call.id`, `tool.call_id`, `duration_ms`, `success`, `error`
829831

830832
- `qwen-code.tool.execution`: Wraps the tool execution phase (after approval).
831833
- **Attributes**: `session.id`, `duration_ms`, `success`, `error`
@@ -837,7 +839,25 @@ Distributed tracing spans form a tree rooted at `qwen-code.interaction`. Each in
837839
- **Attributes**: `session.id`, `hook_event` ("PreToolUse"/"PostToolUse"/"PostToolUseFailure"/"PostToolBatch"), `tool.name`, `tool.use_id` (optional), `is_interrupt` (boolean, optional), `duration_ms`, `success`, `should_proceed` (optional), `should_stop` (optional), `block_type` (optional), `error` (optional)
838840

839841
- `qwen-code.subagent`: Wraps a single subagent invocation.
840-
- **Attributes**: `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.conversation.id`, `qwen-code.subagent.id`, `qwen-code.subagent.name`, `qwen-code.subagent.invocation_kind` ("foreground"/"fork"/"background"), `qwen-code.subagent.is_built_in`, `qwen-code.subagent.depth`, `qwen-code.subagent.status`, `qwen-code.subagent.terminate_reason`, `qwen-code.subagent.duration_ms`
842+
- **Attributes**: `gen_ai.operation.name` (`invoke_agent`), `gen_ai.agent.name`, `gen_ai.agent.description`, `gen_ai.conversation.id`, optional `gen_ai.request.model`, `qwen-code.subagent.id`, `qwen-code.subagent.name`, `qwen-code.subagent.invocation_kind` ("foreground"/"fork"/"background"), `qwen-code.subagent.is_built_in`, `qwen-code.subagent.depth`, `qwen-code.subagent.status`, `qwen-code.subagent.terminate_reason`, `qwen-code.subagent.duration_ms`
843+
844+
#### GenAI field migration and ARMS recognition
845+
846+
LLM spans now use `gen_ai.request.model`, `gen_ai.response.id`, and `gen_ai.usage.*` without the exact-equivalent private `qwen-code.model`, `response_id`, `input_tokens`, `output_tokens`, and `cached_input_tokens` aliases. Tool spans similarly use `gen_ai.tool.name` without `tool.name`; blocked-on-user and hook spans keep `tool.name` because they are not GenAI Tool spans. The invalid aliases `gen_ai.usage.cached_tokens`, `gen_ai.server.time_to_first_token`, and `gen_ai.usage.reasoning_tokens` are no longer emitted. Use `gen_ai.usage.cache_read.input_tokens` for provider-reported cache reads; continue using the private `ttft_ms` and `thoughts_token_count` fields where no GenAI/ARMS-common replacement exists. The full version-pinned contract and deferred fields are documented in [GenAI and ARMS field alignment](../../design/gen-ai-arms-field-alignment.md).
847+
848+
To make ARMS recognize exported spans as a GenAI application, configure its resource feature explicitly:
849+
850+
```json
851+
{
852+
"telemetry": {
853+
"resourceAttributes": {
854+
"acs.arms.service.feature": "genai_app"
855+
}
856+
}
857+
}
858+
```
859+
860+
Qwen Code does not inject this ARMS-specific resource attribute or `gen_ai.span.kind`. ARMS can infer LLM, Tool, and Agent roles from `gen_ai.operation.name`.
841861

842862
- `qwen-code.daemon.request`: Wraps a daemon HTTP request.
843863
- **Attributes**: `http.request.method`, `http.route`, `qwen-code.daemon.operation`, `session.id`, `http.response.status_code`

0 commit comments

Comments
 (0)