Skip to content

Add structured logging, MCP request logging, and an OpenTelemetry pipeline across server apps#1328

Open
FelineStateMachine wants to merge 2 commits into
UsefulSoftwareCo:mainfrom
FelineStateMachine:otel-observability
Open

Add structured logging, MCP request logging, and an OpenTelemetry pipeline across server apps#1328
FelineStateMachine wants to merge 2 commits into
UsefulSoftwareCo:mainfrom
FelineStateMachine:otel-observability

Conversation

@FelineStateMachine

@FelineStateMachine FelineStateMachine commented Jul 5, 2026

Copy link
Copy Markdown

Motivation

Outside of apps/cloud, the server apps have no configured logger (Effect's
default pretty printer), no log or metric export, and no tracer, so the
roughly 145 existing Effect.withSpan call sites in core packages export
nothing. MCP protocol traffic is only visible through the EXECUTOR_MCP_DEBUG
console hook. This makes self-hosted and local deployments hard to operate and
debug.

This PR adds structured logging, MCP request logging, and an OTLP export
pipeline (traces, logs, metrics) to all four server apps. When no OTLP
endpoint is configured, export is disabled and only the structured console
logger is active. Runtime behavior is otherwise unchanged.

Changes

Two commits.

1. Structured logging, MCP request logging, and OTLP telemetry

  • New package packages/core/observability (@executor-js/observability).
    It exports observabilityLayer({ serviceName, endpoint, headers, logLevel }),
    which provides a structured JSON console logger and OTLP/HTTP exporters for
    traces, logs, and metrics. It is built on effect/unstable/observability
    (fetch-based, no async_hooks), so the same layer runs on Bun and on
    Cloudflare workerd. No new npm dependencies.
  • Structured mcp.* log events in packages/hosts/mcp and
    packages/hosts/cloudflare: mcp.tool.start and mcp.tool.end with
    outcome (success, paused, approval_required, error) and duration,
    mcp.tool.internal_error with a correlation id, session lifecycle events
    (created, closed, idle_expire, init_failed, alarm), mcp.auth.outcome,
    dispatch results, and elicitation events at debug level. Two metrics:
    mcp.tool.calls (counter) and mcp.tool.duration_ms (histogram), tagged by
    tool and outcome. Code payloads are not logged; only
    mcp.execute.code_length, matching the existing span attribute convention.
  • Wiring in apps/local, apps/host-selfhost, apps/host-cloudflare, and
    apps/cloud. Cloud keeps its existing trace pipeline and error-capture
    correlation and gains only the logs and metrics layers (the shared package
    takes traces: false for this). apps/local additionally runs its
    in-process MCP surface through a dedicated runtime so tool calls log through
    the same pipeline and flush on shutdown.
  • Docs: package role in AGENTS.md, configuration section in RUNNING.md.
  • EXECUTOR_MCP_DEBUG is unchanged and coexists with the new events.

2. SpanKind.CONSUMER on MCP tool-call spans

mcp.host.tool.execute and mcp.host.tool.resume spans now carry OTel
SpanKind.CONSUMER. Without a kind, these are root INTERNAL spans with no
HTTP attributes wherever no HTTP server span wraps the call (for example the
stdio transport), and span classifiers in observability backends commonly
discard such roots. CONSUMER marks them as remotely initiated units of work
processed by this server, which matches their semantics.

Design notes

  • Each app builds the layer from its own environment source (process env on
    Bun, bindings on Workers) and merges it into the boot seam of
    ExecutorApp.make. Shared packages contain no environment reads and no
    conditional wiring.
  • The console logger writes JSON lines to stderr, not stdout, because the MCP
    stdio transport uses the process's stdout as its JSON-RPC channel. A stdout
    logger corrupts the protocol stream; the existing stdio integration test
    catches this.
  • Log lines include trace_id and span_id from the fiber's current span, so
    console output correlates with exported traces the same way OTLP log records
    do.
  • The MCP tool server already captures its Effect context at construction and
    re-enters it per SDK callback, so the host app's logger and tracer reach
    every tool call without new plumbing. SDK lifecycle callbacks that run
    outside Effect (for example transport close) emit single-line JSON directly.
  • The OTLP logger is added to the current logger set rather than replacing it,
    so console and OTLP both receive every record.

Configuration

Variable Default Effect
OTEL_EXPORTER_OTLP_ENDPOINT unset OTLP/HTTP base URL; /v1/{traces,logs,metrics} appended. Unset disables export.
OTEL_EXPORTER_OTLP_HEADERS none Standard key=value,key2=value2 format.
LOG_LEVEL info Minimum level for console and OTLP logs.

On the Workers apps these are bindings rather than process env.

Testing and verification

  • format:check, lint, typecheck, and test pass.
  • New unit tests: logger output shape and level filtering, header parsing,
    disabled-when-unconfigured behavior, OTLP layer teardown; MCP host tests
    assert mcp.tool.start/mcp.tool.end fire with outcome, duration, and
    execution id, and that defects produce the correlation-id error log.
  • Payload check against a local OTLP collector stub: spans arrive on
    /v1/traces, trace-correlated log records on /v1/logs, both metric series
    on /v1/metrics.
  • Deployed to a self-hosted OTLP backend from this branch (apps/host-selfhost
    and apps/local). Screenshots below. With the endpoint unset, there is no
    network activity and existing e2e tests (including the cloud
    error-correlation contract test) pass unchanged.

MCP tool call classified as a task via SpanKind.CONSUMER, with tool
attributes and child spans:

MCP tool call task detail

Endpoint grouped by parametrized route with database child spans:

Endpoint detail

Disclosure

This change was developed with an LLM coding assistant (Claude), directed and
reviewed by the submitter. Design decisions, code, tests, and the verification
above were produced in that workflow and checked against the repository's full
test gates before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant