Skip to content

Commit e2f2610

Browse files
chrisdocchrisdoc
committed
feat: add privacy-safe telemetry
1 parent b2cf50d commit e2f2610

39 files changed

Lines changed: 1369 additions & 299 deletions

‎.changeset/privacy-safe-telemetry.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"hevy-mcp": patch
3+
---
4+
5+
Add privacy-safe MCP telemetry taxonomy, bounded tool outcomes, session compatibility signals, and dashboard guidance.

‎docs/telemetry-dashboards.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Privacy-reviewed telemetry dashboards
2+
3+
These panels use only the fields in the
4+
[telemetry data dictionary](./telemetry-data-dictionary.md). The examples use
5+
Honeycomb-style metric names and dimensions; Sentry span panels should apply
6+
the equivalent span attribute filters.
7+
8+
## Product usage
9+
10+
| Panel | Source | Grouping/filter | Question answered |
11+
| ---------------------- | ---------------------- | ------------------------------------------------------------- | ------------------------------------------------------------ |
12+
| Feature adoption | `mcp.tool.invocations` | `hevy.feature`, `mcp.tool.operation` | Which Hevy areas and operations are used? |
13+
| Read/write split | `mcp.tool.invocations` | `mcp.tool.kind` | Are users primarily reading or writing? |
14+
| Discovery entry points | `mcp.tool.invocations` | `mcp.tool.operation = search`, `mcp.tool.name` | Which bounded search tools start workflows? |
15+
| Workflow adoption | `mcp.tool.invocations` | `hevy.feature = workflows`, `mcp.tool.operation` | How often are training summaries and routine discovery used? |
16+
| Write adoption | `mcp.tool.invocations` | `mcp.tool.kind = write`, `hevy.feature`, `mcp.tool.operation` | How frequently are writes made by feature? |
17+
18+
Example aggregate query:
19+
20+
```text
21+
COUNT mcp.tool.invocations
22+
GROUP BY hevy.feature, mcp.tool.kind, mcp.tool.operation
23+
WHERE transport = "stdio"
24+
```
25+
26+
## Tool reliability
27+
28+
| Panel | Source | Grouping/filter | Question answered |
29+
| --------------------------- | ---------------------- | -------------------------------------------------------------- | ------------------------------------------------------------- |
30+
| User-visible outcome rate | `mcp.tool.outcomes` | `mcp.tool.name`, `outcome` | Which tools return MCP errors versus succeed? |
31+
| Thrown-error rate | `mcp.tool.errors` | `mcp.tool.name`, `error_type` | Which tools raise uncaught failures? |
32+
| Client compatibility | `mcp.tool.outcomes` | `client_name`, `client_version`, `protocol_version`, `outcome` | Which sanitized client/protocol combinations fail more often? |
33+
| Tool latency | `mcp.tool.duration_ms` | `hevy.feature`, `mcp.tool.operation`, `outcome` | Which bounded feature operations are slow? |
34+
| Result shape versus latency | `mcp.tool.duration_ms` | `mcp.tool.result.item_count_bucket`, `hevy.feature` | Do large result shapes correlate with latency? |
35+
36+
Returned `isError: true` responses use `outcome=returned_error`; thrown
37+
exceptions use `outcome=thrown_error`. Do not infer user-visible failure rate
38+
from `mcp.tool.errors` alone.
39+
40+
## Hevy API reliability
41+
42+
| Panel | Source | Grouping/filter | Question answered |
43+
| ------------------------- | ---------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------- |
44+
| API status friction | `hevy.api.calls` | normalized `endpoint`, `status_code`, `method` | Which endpoint/status pairs fail? |
45+
| Stale-resource signals | `hevy.api.calls` | `status_code = 404`, normalized `endpoint` | Are 404s concentrated in resource lookup or pagination? |
46+
| Conflicting writes | `hevy.api.calls` | `status_code = 409`, normalized `endpoint`, `method` | Are duplicate/conflicting writes concentrated in one feature? |
47+
| Retry and timeout signals | `hevy.api.calls` | `status_code`, `retry_count_bucket`, `error_category`, `error_code` | How often do retries, rate limits, and timeouts occur? |
48+
| API duration | `hevy.api.duration_ms` | normalized `endpoint`, `method` | Which API operations are slow? |
49+
50+
Endpoints are normalized before telemetry. Dynamic path segments are replaced
51+
with placeholders; raw IDs never reach these panels.
52+
53+
## Performance and workflows
54+
55+
| Panel | Source | Grouping/filter | Question answered |
56+
| ----------------------------- | ----------------------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------- |
57+
| Workflow pages scanned | `mcp.tool.get-training-summary`, `mcp.tool.search-routines` | `workflow.pagination.*.pages` | Which workflows scan the most pages? |
58+
| Cache hit/miss | `mcp.tool.get-training-summary`, `mcp.tool.search-routines` | `workflow.cache_status` | How effective is catalog caching? |
59+
| Workflow scan versus duration | `mcp.tool.get-training-summary`, `mcp.tool.search-routines` | `workflow.items_scanned` | Does scan size correlate with latency? |
60+
| Session lifecycle | `mcp.session.started`, `mcp.session.ended` | sanitized client fields, `termination_category` | Do sessions end cleanly or fail during startup/connect/tool execution? |
61+
| Session shape | `mcp.session.ended` | `session_duration_bucket`, `tool_calls_bucket` | How long are sessions and how many tools do they call? |
62+
63+
## Retention and access review
64+
65+
Approved application policy for these dashboards:
66+
67+
- aggregate metrics: 90 days;
68+
- Sentry/OTel traces containing sanitized client metadata or the pseudonymous
69+
user hash: 30 days;
70+
- user-hash troubleshooting views: 24 hours of access and no saved per-user
71+
dashboard or query;
72+
- dashboard access owners: repository maintainers and the on-call operator;
73+
- no export of prompt, argument, result, title, notes, dates, identifiers, or
74+
measurements.
75+
76+
Backend retention settings must be configured to match this policy before a
77+
panel is published. A review owner must re-check the Sentry MCP options and the
78+
telemetry dictionary whenever the SDK or dashboard definitions change.
79+
80+
## Publication checklist
81+
82+
Automated guards that must remain green:
83+
84+
- [x] Sentry MCP input capture is explicitly disabled; `src/index.test.ts`
85+
asserts `recordInputs: false`.
86+
- [x] Sentry MCP output capture is explicitly disabled; `src/index.test.ts`
87+
asserts `recordOutputs: false`.
88+
- [x] Metric dimensions have fixed taxonomies or bounded sanitization;
89+
`src/tools/register.test.ts` and `src/utils/telemetry-wrapper.test.ts`
90+
cover the declared fields.
91+
- [x] Raw queries, IDs, dates, titles, notes, descriptions, and measurements
92+
are excluded; privacy regression tests assert secret-sentinel absence.
93+
- [x] Returned MCP errors and thrown errors have separate outcomes;
94+
`src/utils/telemetry-wrapper.test.ts` covers both paths.
95+
- [x] Session termination and sanitized client/protocol fields are covered by
96+
`src/utils/mcp-session-observability.test.ts`.
97+
- [x] Backend retention and dashboard access policy are documented above.
98+
99+
Before publishing or changing a panel:
100+
101+
- [ ] Configure backend retention to the policy above.
102+
- [ ] Confirm repository maintainers and the on-call operator own access.
103+
- [ ] Re-run the privacy regression tests and re-review the data dictionary
104+
whenever Sentry, OpenTelemetry, or dashboard definitions change.

‎docs/telemetry-data-dictionary.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Privacy-safe telemetry data dictionary
2+
3+
This dictionary is the contract for Sentry spans and OpenTelemetry metrics. The
4+
telemetry implementation must fail closed when a value is not in an allowlist
5+
or bounded bucket.
6+
7+
## Approved bounded dimensions
8+
9+
| Field | Allowed values | Applies to |
10+
| ------------------------------ | -------------------------------------------------------------------------------------- | --------------------------------------------- |
11+
| `hevy.feature` | `workouts`, `routines`, `templates`, `measurements`, `folders`, `profile`, `workflows` | Tool spans and tool metrics |
12+
| `mcp.tool.kind` | `read`, `write` | Tool spans and tool metrics |
13+
| `mcp.tool.operation` | `list`, `get`, `search`, `create`, `update`, `count`, `sync` | Tool spans and tool metrics |
14+
| `outcome` / `mcp.tool.outcome` | `success`, `returned_error`, `thrown_error` | Tool outcome and duration metrics; tool spans |
15+
| Result count buckets | `0`, `1`, `2-10`, `11-50`, `51+` | Result-shape spans and tool duration metrics |
16+
| Retry count buckets | `0`, `1`, `2-10`, `11-50`, `51+` | API spans and API calls/duration metrics |
17+
| Session termination | `clean`, `startup_failure`, `connect_failure`, `tool_failure`, `unknown` | Session metrics |
18+
| Session duration buckets | `<1s`, `1-10s`, `10-60s`, `1-5m`, `5m+` | Session metrics |
19+
| Tool-call buckets | `0`, `1`, `2-10`, `11-50`, `51+` | Session metrics |
20+
| Cache status | `hit`, `miss`, `not-used` | Workflow spans |
21+
| API method | HTTP method from the client allowlist | API spans and metrics |
22+
| API endpoint | Normalized static endpoint or a placeholder path containing no identifier | API spans and metrics |
23+
| HTTP status | Numeric status code | API diagnostics and metrics |
24+
25+
API error categories and codes are emitted only after `createSafeErrorDiagnostic`
26+
normalization. Categories are the finite `SafeErrorCategory` union; codes are
27+
the finite allowlist in `error-policy.ts`. Neither field contains an upstream
28+
message or arbitrary error value.
29+
The exact tool name remains available as `mcp.tool.name` / `tool_name` for
30+
short-lived debugging. It is not a product taxonomy dimension.
31+
32+
## Structural fields
33+
34+
The tool wrapper may record argument key names from the fixed schema, total
35+
argument-key count as a bucket, and the following structural values:
36+
37+
- presence flags for IDs, dates, timestamps, queries, and muscle-group filters;
38+
- count buckets for pagination and limit fields;
39+
- booleans such as `includeCustom` and `refresh`;
40+
- result content-block count and structured-content presence;
41+
- result item, exercise, and set count buckets;
42+
- `has_notes`, `folder_selected`, and `uses_rep_ranges` booleans;
43+
- workflow page counts, bounded workflow name, cache status, and `items_scanned`.
44+
45+
These fields describe shape only. They never contain a value from the argument
46+
or result body.
47+
48+
## Session and client fields
49+
50+
The stdio initialize message may provide client name, client version, and MCP
51+
protocol version. Each value is trimmed, restricted to a safe printable
52+
character set, and limited to 64 characters; malformed or missing values become
53+
`unknown`. The transport is always `stdio` for this path. Metrics never contain
54+
a session ID, request ID, progress token, prompt, argument, result, or user
55+
hash. The server version is supplied by the service resource (`service.version`)
56+
and server lifecycle spans.
57+
58+
The pseudonymous user hash is span-only correlation data. It is not a metric
59+
dimension and must not be used to construct per-user behavior histories.
60+
61+
## Explicitly prohibited fields
62+
63+
Never send or inspect for telemetry:
64+
65+
- MCP prompts, prompt arguments, tool arguments, or tool result text;
66+
- raw queries, workout/routine/folder/exercise-template IDs, request IDs, or
67+
progress tokens;
68+
- workout titles, descriptions, notes, exercise names, routine names, or folder
69+
names;
70+
- exact dates or timestamps from tool arguments or returned records;
71+
- body measurements, weights, reps, distances, durations, or other measurement
72+
values;
73+
- arbitrary client metadata or unnormalized endpoint paths.
74+
75+
## Regression guard
76+
77+
`src/index.test.ts`, `src/utils/telemetry-wrapper.test.ts`,
78+
`src/utils/stdio-observability.test.ts`, and
79+
`src/utils/mcp-session-observability.test.ts` assert the capture settings,
80+
allowlisted attributes, sanitized client metadata, and secret-sentinel absence.
81+
Any telemetry field change must update this dictionary and its regression tests
82+
in the same change.

‎src/index.test.ts‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -90,12 +90,15 @@ vi.mock("./utils/telemetry.js", () => ({
9090

9191
vi.mock("./utils/metrics.js", () => ({
9292
toolInvocations: { add: vi.fn() },
93+
toolOutcomes: { add: vi.fn() },
9394
toolErrors: { add: vi.fn() },
9495
toolDuration: { record: vi.fn() },
9596
apiCalls: { add: vi.fn() },
9697
apiDuration: { record: vi.fn() },
9798
stdioParseErrors: { add: vi.fn() },
9899
serverStartups: { add: vi.fn() },
100+
sessionStarted: { add: vi.fn() },
101+
sessionEnded: { add: vi.fn() },
99102
}));
100103

101104
vi.mock("@opentelemetry/api", () => ({
@@ -186,6 +189,10 @@ describe("Server entry", () => {
186189
it("creates an MCP server instance after validating the API key", async () => {
187190
const server = await createServer({ config: { apiKey: "test-key" } });
188191
expect(server).toBeDefined();
192+
expect(testDoubles.sentry.wrapMcpServerWithSentry).toHaveBeenCalledWith(
193+
expect.anything(),
194+
{ recordInputs: false, recordOutputs: false },
195+
);
189196
expect(testDoubles.getUserInfo).toHaveBeenCalledTimes(1);
190197
expect(testDoubles.getUserInfo.mock.invocationCallOrder[0]).toBeLessThan(
191198
testDoubles.mcpServerConstructor.mock.invocationCallOrder[0],
@@ -516,9 +523,12 @@ describe("Server entry", () => {
516523
);
517524
expect(spanNames).toContain("mcp.server.run");
518525
expect(spanNames).toContain("mcp.server.connect");
519-
expect(testDoubles.installGracefulShutdown).toHaveBeenCalledWith({
520-
target: expect.objectContaining({ close: testDoubles.close }),
521-
});
526+
expect(testDoubles.installGracefulShutdown).toHaveBeenCalledWith(
527+
expect.objectContaining({
528+
target: expect.objectContaining({ close: testDoubles.close }),
529+
onComplete: expect.any(Function),
530+
}),
531+
);
522532
expect(testDoubles.connect.mock.invocationCallOrder[0]).toBeLessThan(
523533
testDoubles.installGracefulShutdown.mock.invocationCallOrder[0] ?? 0,
524534
);

‎src/index.ts‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ import { createNodeHevyClientOptions } from "./utils/hevy-client-observability.j
2121
import { createClient } from "./utils/hevyClient.js";
2222
import { withObservability } from "./utils/observability-wrapper.js";
2323
import { createInstrumentedStdioTransport } from "./utils/stdio-observability.js";
24+
import {
25+
recordMcpSessionTermination,
26+
resolveSessionTerminationCategory,
27+
} from "./utils/mcp-session-observability.js";
2428
import { scheduleUpdateCheck } from "./utils/version-check.js";
2529

2630
const name = serviceName;
@@ -177,7 +181,10 @@ function buildServer(apiKey: string) {
177181
clientOptions: createNodeHevyClientOptions(),
178182
wrapHandler: withObservability,
179183
wrapServer: (baseServer) =>
180-
Sentry.wrapMcpServerWithSentry(baseServer),
184+
Sentry.wrapMcpServerWithSentry(baseServer, {
185+
recordInputs: false,
186+
recordOutputs: false,
187+
}),
181188
onToolsRegistered: (count) =>
182189
span.setAttribute("mcp.tools.count", count),
183190
});
@@ -232,6 +239,7 @@ export async function runServer() {
232239
setCurrentUserHash(initialUserHash);
233240
Sentry.setUser({ id: initialUserHash });
234241
}
242+
let connectAttempted = false;
235243

236244
await tracer.startActiveSpan(
237245
"mcp.server.run",
@@ -251,6 +259,7 @@ export async function runServer() {
251259
const transport = createInstrumentedStdioTransport(
252260
new StdioServerTransport(),
253261
);
262+
connectAttempted = true;
254263

255264
await tracer.startActiveSpan(
256265
"mcp.server.connect",
@@ -275,10 +284,19 @@ export async function runServer() {
275284
packageName: serviceName,
276285
currentVersion: serviceVersion,
277286
});
278-
installGracefulShutdown({ target: server });
287+
installGracefulShutdown({
288+
target: server,
289+
onComplete: (succeeded) =>
290+
recordMcpSessionTermination(
291+
resolveSessionTerminationCategory(succeeded),
292+
),
293+
});
279294

280295
span.setStatus({ code: SpanStatusCode.OK });
281296
} catch (e) {
297+
recordMcpSessionTermination(
298+
connectAttempted ? "connect_failure" : "startup_failure",
299+
);
282300
span.setStatus({ code: SpanStatusCode.ERROR });
283301
throw e;
284302
} finally {

‎src/shared-server.ts‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,20 +7,23 @@ import {
77
SERVER_VERSION,
88
} from "./server-metadata.js";
99
import { registerHevyTools } from "./tools/register.js";
10-
import { withErrorHandling } from "./utils/error-handler.js";
1110
import { createExerciseTemplateCatalog } from "./utils/exercise-template-catalog.js";
1211
import { createClient } from "./utils/hevyClient.js";
1312
import type { HevyClient } from "./utils/hevyClient.js";
1413
import type { HevyClientOptions } from "./utils/hevyClientKubb.js";
1514
import { createMcpClientLogger } from "./utils/mcp-client-logger.js";
16-
import { createToolRuntime } from "./tools/tool-runtime.js";
15+
import {
16+
createToolRuntime,
17+
defaultToolHandlerWrapper,
18+
} from "./tools/tool-runtime.js";
19+
import type { ToolHandlerWrapper } from "./tools/tool-runtime.js";
1720

1821
export interface SharedServerOptions {
1922
apiKey: string;
2023
clientOptions?: HevyClientOptions;
21-
hevyClient?: HevyClient;
2224
onToolsRegistered?: (count: number) => void;
23-
wrapHandler?: typeof withErrorHandling;
25+
hevyClient?: HevyClient;
26+
wrapHandler?: ToolHandlerWrapper;
2427
wrapServer?: (server: McpServer) => McpServer;
2528
}
2629

@@ -70,8 +73,7 @@ export function createSharedMcpServer(options: SharedServerOptions): McpServer {
7073
const runtime = createToolRuntime({
7174
client: hevyClient,
7275
catalog: createExerciseTemplateCatalog(hevyClient),
73-
logger,
74-
wrapHandler: options.wrapHandler ?? withErrorHandling,
76+
wrapHandler: options.wrapHandler ?? defaultToolHandlerWrapper,
7577
});
7678
const counting = createToolCountingServer(server);
7779

0 commit comments

Comments
 (0)