Skip to content

Add POSIX telemetry - #27379

Merged
bmehta001 merged 43 commits into
mainfrom
bhamehta/posix-telemetry
Jul 24, 2026
Merged

Add POSIX telemetry#27379
bmehta001 merged 43 commits into
mainfrom
bhamehta/posix-telemetry

Conversation

@bmehta001

@bmehta001 bmehta001 commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds 1DS-backed telemetry to ONNX Runtime on Linux, macOS, Android, and iOS. Windows continues to use its existing ETW provider, with aligned metadata, redaction, and unit-test suppression. WebAssembly remains telemetry-free.

No public ABI is changed.

Behavior and controls

  • Emits one process-level ProcessInfo event plus low-frequency session, model, execution-provider, error, and aggregate runtime events.
  • Uses deterministic session-level sampling so correlated events are retained or dropped together. The current sampling rate is 100%, and retained events carry 1DS popSample metadata.
  • Serializes map-valued metadata deterministically to avoid multiple telemetry values for identical logical data.
  • Aggregates successful inference counts and durations in RuntimePerf. POSIX EvaluationStart/EvaluationStop are no-ops, and SystemMetrics follows the RuntimePerf cadence instead of running on every inference.
  • Native telemetry is compiled only when configured with --use_telemetry; direct build.py/CMake builds can omit it completely.
  • ORT_DISABLE_TELEMETRY=1 is a full non-Windows opt-out when set before initialization: no uploader, events, persistent device identifier, or offline cache is created. The decision is latched for the process lifetime.
  • Runtime telemetry APIs suppress non-essential events. ONNX Runtime may already have emitted the minimal initialization event before the API can be called.
  • CI hard-suppresses POSIX telemetry; ORT unit-test processes suppress both POSIX telemetry and Windows ETW.

Privacy, safety, and reliability

  • Desktop builds send a product-salted hash of a locally generated per-user UUID; the raw UUID is never uploaded. Android and iOS retain the platform identity supplied by 1DS.
  • Persistent device identity uses owner-only storage, bounded no-follow reads, dirfd-relative file operations, atomic publication, and serialized corruption repair shared with ONNX Runtime GenAI.
  • Free-text fields scrub filesystem paths and cap payload size without splitting UTF-8 code points.
  • Every complete POSIX telemetry operation - event construction, redaction, property insertion, and SDK emission - is contained by a no-throw boundary. Diagnostic failures are also contained.
  • SDK initialization is transactional: partially created managers are released before configuration teardown, and logger state is published only after setup completes.
  • Logging holds a shared lock while 1DS accepts an event; initialization and shutdown retain exclusive locking. Concurrent logging is not serialized by ORT.
  • Shutdown is configured not to wait for network upload and remains safe during process teardown.

Build and package integration

  • Uses the cpp-client-telemetry vcpkg port when available, with a pinned FetchContent fallback.
  • Links the SDK statically where supported and includes its dependency metadata and third-party notices.
  • Official Linux, macOS, Android, and iOS package configurations enable telemetry.
  • Official Android AARs package and initialize the 1DS Java HTTP transport, including relocated bridge classes and required JNI exports so they can coexist with ONNX Runtime GenAI.
  • Apple framework builds use the platform HTTP transport and static packaging path.
  • An optional tenant-token override is emitted into a generated header rather than a compiler command line or pipeline YAML.

Performance

A Release benchmark using mul_1.onnx, alternating enabled/disabled samples, CPU affinity, isolated caches, and blocked network showed that the original synchronous per-inference 1DS events were too expensive:

Scenario Before After hot-path removal
Single thread 100.4 us enabled vs 3.5 us disabled (28.7x) Difference below benchmark noise
Eight threads 11.3k enabled vs 306.5k disabled runs/s (96.3% loss) 294.3k enabled vs 295.5k disabled runs/s (0.4% difference)

Network upload remains asynchronous inside 1DS; this change removes serialized SDK intake from each successful Run().

Validation

  • Telemetry-enabled Linux builds and focused environment, redaction, no-throw, and device-ID tests.
  • Windows build/test coverage for shared telemetry helpers and ETW metadata changes.
  • Android AAR build plus x86_64 emulator end-to-end session creation.
  • Backend verification of ProcessInfo, SessionCreationStart, SessionCreation, and SessionCreationEnd, including shared process correlation and platform device identity.
  • Concurrent first-run and corrupted-file device-ID recovery stress tests.
  • Lint, material code review, and repeated Copilot review rounds.

@bmehta001
bmehta001 requested a review from Copilot February 18, 2026 19:04
@bmehta001 bmehta001 self-assigned this Feb 18, 2026
@bmehta001 bmehta001 changed the title Add POSIX telemetry Add POSIX telemetry [DRAFT] Feb 18, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can commit the suggested changes from lintrunner.

Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.h Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.h Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.h Outdated
Comment thread tools/ci_build/build.py Outdated
Comment thread tools/ci_build/build_args.py Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Fixed
Comment thread onnxruntime/core/platform/posix/telemetry.h Fixed
Comment thread tools/ci_build/build.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces cross-platform build support for telemetry by extending the existing Windows ETW telemetry to non-Windows platforms via the 1DS (cpp_client_telemetry) SDK, wiring it through the build scripts and CMake.

Changes:

  • Adds a new POSIX telemetry provider implementation using the 1DS SDK and swaps it into the POSIX Env when enabled.
  • Extends build and CMake plumbing to fetch/link cpp_client_telemetry on non-Windows and to expose a cross-platform --use_telemetry build flag.
  • Updates top-level build wrappers and third-party notices for the new dependency.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
tools/ci_build/build_args.py Moves --use_telemetry to cross-platform args.
tools/ci_build/build.py Always sets -Donnxruntime_USE_TELEMETRY=ON/OFF based on --use_telemetry.
onnxruntime/core/platform/windows/telemetry.cc Include ordering tweak only.
onnxruntime/core/platform/posix/telemetry.h Introduces POSIX telemetry provider interface (1DS).
onnxruntime/core/platform/posix/telemetry.cc Implements POSIX telemetry provider using cpp_client_telemetry.
onnxruntime/core/platform/posix/env.cc Uses PosixTelemetry provider when USE_1DS_TELEMETRY is defined.
cmake/onnxruntime_common.cmake Adds POSIX telemetry sources/defines and links 1DS + system libs when enabled.
cmake/onnxruntime_1ds_telemetry.cmake New helper module for enabling 1DS telemetry on non-Windows.
cmake/external/onnxruntime_external_deps.cmake Fetches cpp_client_telemetry only when telemetry is enabled on non-Windows.
cmake/deps.txt Adds cpp_client_telemetry dependency entry.
cmake/CMakeLists.txt Includes the new 1DS telemetry CMake module.
build.sh Now passes --use_telemetry by default.
build.bat Now passes --use_telemetry by default.
ThirdPartyNotices.txt Adds cpp_client_telemetry license text (and an additional KleidiAi block).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.h
Comment thread onnxruntime/core/platform/posix/telemetry.cc
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread build.bat Outdated
Comment thread ThirdPartyNotices.txt Outdated
Comment thread cmake/external/onnxruntime_external_deps.cmake Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can commit the suggested changes from lintrunner.

Comment thread onnxruntime/core/platform/posix/telemetry.h Outdated
@bmehta001
bmehta001 requested a review from a team as a code owner March 5, 2026 10:07
@bmehta001
bmehta001 force-pushed the bhamehta/posix-telemetry branch from fded7ba to 2f18217 Compare June 11, 2026 06:40
@bmehta001
bmehta001 force-pushed the bhamehta/posix-telemetry branch from 178e702 to b0eaeb7 Compare June 23, 2026 17:14
@bmehta001
bmehta001 requested a review from Copilot June 23, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread build.sh Outdated
Comment thread cmake/onnxruntime_1ds_telemetry.cmake Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.h
Comment thread onnxruntime/core/platform/posix/telemetry.cc

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread cmake/onnxruntime_1ds_telemetry.cmake Outdated
Comment thread cmake/onnxruntime_common.cmake
Comment thread ThirdPartyNotices.txt

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Comment thread onnxruntime/core/platform/posix/telemetry.h
Comment thread onnxruntime/core/platform/posix/telemetry.cc
Comment thread onnxruntime/core/platform/posix/telemetry.cc
Comment thread build.sh Outdated
Comment thread cmake/vcpkg.json

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Comment thread build.sh Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc
Comment thread onnxruntime/core/platform/posix/telemetry.cc
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/windows/telemetry.cc
Comment thread onnxruntime/core/platform/posix/device_id.cc Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Comment thread build.sh Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/env.cc Outdated
Comment thread cmake/vcpkg-configuration.json
Comment thread cmake/onnxruntime_1ds_telemetry.cmake Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 7 comments.

Comment thread build.sh Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
Comment thread onnxruntime/core/platform/posix/telemetry.cc Outdated
@bmehta001
bmehta001 requested a review from Copilot June 23, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants