The task is to create a document that summarizes the work done and provides evidence of completion. According to the reporting guidelines, this can be achieved by collating links to all relevant commits, pull requests, and other source code management actions. Once the report is finalized, it will be sent to our program manager.
Milestone 6 called for implementing comprehensive observability features that enable developers to monitor, debug, and optimize federated applications in production environments. The contract specified work in the following areas:
All of this work shipped in Fedify 2.3.0 (released June 24, 2026). The deliverables actually produced are:
-
Comprehensive OpenTelemetry metrics for Fedify's federation runtime: metrics now cover HTTP request counts and duration, outbound delivery outcomes, inbox processing duration, ActivityPub fanout, inbox/outbox activity lifecycle, queue task processing, collection dispatch and pagination, WebFinger lookup and handling, actor discovery, key lookup, remote document fetches, document cache behavior, and object lookup.
-
Queue depth and queue processing visibility: MessageQueue now supports optional getDepth() reporting, with implementations for the in-process, parallel, AMQP, MySQL/MariaDB, PostgreSQL, Redis, and SQLite queue backends. Fedify also emits queue enqueue, start, completion, failure, duration, and in-flight task metrics for inbox, outbox, and fanout work.
-
Signature verification and key-fetch timing metrics: Fedify now measures verification duration for HTTP Signatures, Linked Data Signatures, and Object Integrity Proofs, and separately measures public-key lookup latency so operators can distinguish remote key-fetch delays from local verification cost.
-
Federation health indicators: delivery success and permanent-failure metrics, remote-host-scoped document and key lookup metrics, WebFinger and actor discovery result metrics, and circuit breaker state-change metrics now make peer-server reliability visible without exposing high-cardinality actor IDs, activity IDs, inbox URLs, or handles.
-
Outbound delivery circuit breaker: queued outbox delivery now includes a host-level circuit breaker that stops repeatedly retrying unreachable peer servers, holds deliveries while the circuit is open, probes after a recovery delay, and emits state-change telemetry.
-
Production monitoring documentation: the new Production monitoring guide on fedify.dev turns Fedify's metrics into Prometheus queries, dashboard panels, and alert rules for queue backlog, delivery failures, permanent delivery failures, inbox latency, signature verification latency, and related operational signals.
-
Runnable monitoring example: examples/monitoring/ provides a local OpenTelemetry Collector, Prometheus, Grafana, dashboard JSON, alert rules, synthetic Fedify-shaped metrics, validation scripts, and Docker Compose setup so operators can try the monitoring stack before connecting a real application.
-
ActivityPub-specific benchmarking tools: @fedify/cli now includes fedify bench, with YAML/JSON scenario suites, JSON Schema validation, signed inbox benchmarks, WebFinger, actor, object, fanout, failure, and mixed scenarios, queue-drain measurement, Markdown/JSON/text reports, and safety checks for public targets.
-
Benchmark comparison gates: fedify bench compare can check out base and head Git refs, run the same benchmark suite against both, account for measured noise, and fail CI when the head regresses beyond the configured allowance.
All technical details, code changes, and completion evidence can be found in the linked resources and in the release discussion at #821.
Now that the work for Milestone 6, “Observability and Monitoring,” is complete, we need to prepare and submit a progress report to the Sovereign Tech Agency as stipulated in our service agreement. This report is a prerequisite for invoicing the sixth milestone payment.
The task is to create a document that summarizes the work done and provides evidence of completion. According to the reporting guidelines, this can be achieved by collating links to all relevant commits, pull requests, and other source code management actions. Once the report is finalized, it will be sent to our program manager.
Milestone scope
Milestone 6 called for implementing comprehensive observability features that enable developers to monitor, debug, and optimize federated applications in production environments. The contract specified work in the following areas:
All of this work shipped in Fedify 2.3.0 (released June 24, 2026). The deliverables actually produced are:
Comprehensive OpenTelemetry metrics for Fedify's federation runtime: metrics now cover HTTP request counts and duration, outbound delivery outcomes, inbox processing duration, ActivityPub fanout, inbox/outbox activity lifecycle, queue task processing, collection dispatch and pagination, WebFinger lookup and handling, actor discovery, key lookup, remote document fetches, document cache behavior, and object lookup.
Queue depth and queue processing visibility:
MessageQueuenow supports optionalgetDepth()reporting, with implementations for the in-process, parallel, AMQP, MySQL/MariaDB, PostgreSQL, Redis, and SQLite queue backends. Fedify also emits queue enqueue, start, completion, failure, duration, and in-flight task metrics for inbox, outbox, and fanout work.Signature verification and key-fetch timing metrics: Fedify now measures verification duration for HTTP Signatures, Linked Data Signatures, and Object Integrity Proofs, and separately measures public-key lookup latency so operators can distinguish remote key-fetch delays from local verification cost.
Federation health indicators: delivery success and permanent-failure metrics, remote-host-scoped document and key lookup metrics, WebFinger and actor discovery result metrics, and circuit breaker state-change metrics now make peer-server reliability visible without exposing high-cardinality actor IDs, activity IDs, inbox URLs, or handles.
Outbound delivery circuit breaker: queued outbox delivery now includes a host-level circuit breaker that stops repeatedly retrying unreachable peer servers, holds deliveries while the circuit is open, probes after a recovery delay, and emits state-change telemetry.
Production monitoring documentation: the new Production monitoring guide on fedify.dev turns Fedify's metrics into Prometheus queries, dashboard panels, and alert rules for queue backlog, delivery failures, permanent delivery failures, inbox latency, signature verification latency, and related operational signals.
Runnable monitoring example:
examples/monitoring/provides a local OpenTelemetry Collector, Prometheus, Grafana, dashboard JSON, alert rules, synthetic Fedify-shaped metrics, validation scripts, and Docker Compose setup so operators can try the monitoring stack before connecting a real application.ActivityPub-specific benchmarking tools:
@fedify/clinow includesfedify bench, with YAML/JSON scenario suites, JSON Schema validation, signed inbox benchmarks, WebFinger, actor, object, fanout, failure, and mixed scenarios, queue-drain measurement, Markdown/JSON/text reports, and safety checks for public targets.Benchmark comparison gates:
fedify bench comparecan check out base and head Git refs, run the same benchmark suite against both, account for measured noise, and fail CI when the head regresses beyond the configured allowance.Releases
All work for this milestone shipped in a single release:
Related issues and PRs
OpenTelemetry metrics
Queue depth and federation health
Production monitoring documentation and examples
Performance benchmarking tools
benchmarkModeand the cooperative benchmark endpoints #782: Benchmarking:benchmarkModeand the cooperative benchmark endpointsfedify benchengine, scenario format, and JSON schema hosting #783: Benchmarking:fedify benchengine, scenario format, and JSON schema hostingfedify benchcommand with its scenario and report schemas #791: Thefedify benchcommand with its scenario and report schemasfedify benchsafety explicit for staging targets #795: Makefedify benchsafety explicit for staging targetsfedify bench#802: Run remaining benchmark scenarios infedify benchfedify bench compareA/B mode and CI gate robustness #786: Benchmarking:fedify bench compareA/B mode and CI gate robustnessAll technical details, code changes, and completion evidence can be found in the linked resources and in the release discussion at #821.