fix: token analyzers look for wrong artifact name#1883
Closed
Conversation
The token usage analyzer and optimizer workflows were downloading `agent-artifacts` to find `token-usage.jsonl`, but the actual artifact name is `firewall-audit-logs` (nested under `logs/api-proxy-logs/token-usage.jsonl`). This caused the analyzers to report 'no token data' for workflows like Smoke Copilot even though the data was present. Updated all 4 workflows: - copilot-token-usage-analyzer.md - claude-token-usage-analyzer.md - copilot-token-optimizer.md - claude-token-optimizer.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the token usage analyzer/optimizer workflow prompts to download the correct artifact (firewall-audit-logs) when locating token-usage.jsonl, aligning the analyzers with how AWF workflows actually upload token usage logs.
Changes:
- Update Copilot + Claude token usage analyzer prompts to reference
firewall-audit-logs(and thelogs/api-proxy-logs/location). - Update Copilot + Claude token optimizer prompts to download
firewall-audit-logsinstead of the previously referenced artifact name.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/copilot-token-usage-analyzer.md | Updates analyzer instructions to download firewall-audit-logs and locate token logs within it. |
| .github/workflows/claude-token-usage-analyzer.md | Same as above, scoped to Claude workflows and provider filtering guidance. |
| .github/workflows/copilot-token-optimizer.md | Updates optimizer artifact download instructions to use firewall-audit-logs. |
| .github/workflows/claude-token-optimizer.md | Updates optimizer artifact download instructions to use firewall-audit-logs. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 2
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
|
Smoke test results:
|
Contributor
Smoke Test Results
Overall: PASS
|
Closed
lpcox
added a commit
that referenced
this pull request
Apr 10, 2026
Replace manual artifact downloading (gh run download --name firewall-audit-logs) with gh aw logs --json in all 4 token analysis workflows. This aligns with the approach used by gh-aw's own token-audit and token-optimizer workflows. Benefits: - Eliminates dependency on knowing the correct artifact name (the root cause of the bug fixed in PR #1883) - Gets pre-aggregated structured data (token counts, costs, turns) instead of parsing raw token-usage.jsonl - Handles artifact naming backward/forward compat automatically - Pre-downloads data in steps block (faster, more reliable) Changes: - copilot-token-usage-analyzer.md: replaced manual run discovery + artifact download with gh aw logs --engine copilot --json pre-step - claude-token-usage-analyzer.md: same for --engine claude - copilot-token-optimizer.md: added gh aw logs --engine copilot pre-step for 7-day data, removed manual artifact download - claude-token-optimizer.md: same for --engine claude - shared/mcp/gh-aw.md: new shared component to install gh-aw CLI (mirroring upstream gh-aw's shared/mcp/gh-aw.md) Refs: #1883, github/gh-aw#25683 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Apr 10, 2026
lpcox
added a commit
that referenced
this pull request
Apr 10, 2026
Replace manual artifact downloading (gh run download --name firewall-audit-logs) with gh aw logs --json in all 4 token analysis workflows. This aligns with the approach used by gh-aw's own token-audit and token-optimizer workflows. Benefits: - Eliminates dependency on knowing the correct artifact name (the root cause of the bug fixed in PR #1883) - Gets pre-aggregated structured data (token counts, costs, turns) instead of parsing raw token-usage.jsonl - Handles artifact naming backward/forward compat automatically - Pre-downloads data in steps block (faster, more reliable) Changes: - copilot-token-usage-analyzer.md: replaced manual run discovery + artifact download with gh aw logs --engine copilot --json pre-step - claude-token-usage-analyzer.md: same for --engine claude - copilot-token-optimizer.md: added gh aw logs --engine copilot pre-step for 7-day data, removed manual artifact download - claude-token-optimizer.md: same for --engine claude - shared/mcp/gh-aw.md: new shared component to install gh-aw CLI (mirroring upstream gh-aw's shared/mcp/gh-aw.md) Refs: #1883, github/gh-aw#25683 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The token usage analyzer and optimizer workflows were downloading
agent-artifactsto findtoken-usage.jsonl, but the actual artifact uploaded by all AWF workflows isfirewall-audit-logs. The token data lives atlogs/api-proxy-logs/token-usage.jsonlinside that artifact.This caused the daily reports to claim 'no token data' for workflows like Smoke Copilot, even though the data was present and accessible.
Fix
Updated all 4 token analysis workflows to download
firewall-audit-logsinstead ofagent-artifacts:copilot-token-usage-analyzer.mdclaude-token-usage-analyzer.mdcopilot-token-optimizer.mdclaude-token-optimizer.mdImpact
Future token usage reports should now correctly capture data from all workflows that use
--enable-api-proxy, including Smoke Copilot (which has cli-proxy enabled — useful for measuring cli-proxy token efficiency).