wip: pytest support#80
Draft
gnufede wants to merge 26 commits into
Draft
Conversation
Implement initial Python platform detection and Pytest framework support. What: - Add Python platform layer (internal/platform/python.go) for runtime detection - Add Pytest framework implementation (internal/framework/pytest.go) for test discovery and execution - Update platform detection to support "python" platform - Add Python environment tag collection script Why: Enable ddtest to support Python projects using Pytest, expanding beyond Ruby-only support. Breakdown: - Python platform: Detects Python version, verifies datadog-test-lib installation, collects OS/runtime tags - Pytest framework: Implements test discovery via pytest --collect-only, file discovery via glob, and test execution - Integration: Both follow existing Ruby/Rspec patterns for consistency Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Include ddtrace pytest plugin by setting PYTEST_ADDOPTS environment variable, mirroring Ruby's RUBYOPT approach for automatic instrumentation. - Add GetPlatformEnv() to Python platform to set PYTEST_ADDOPTS="-p ddtrace.pytest_plugin" - Skip if already set in environment (respects user overrides) - Pass platform env to framework in DetectFramework() call This ensures tests are automatically instrumented with Datadog tracing for CI Visibility without requiring manual pytest configuration. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Change from plugin notation to pytest's cleaner --ddtrace flag. Add validation for both required packages: - datadog-test-lib >= 0.1.0 - ddtrace >= 1.0.0 Refactor SanityCheck to use shared checkPackageVersion() method to avoid code duplication and make version checking more robust. This ensures both Datadog and ddtrace libraries are installed and meet minimum version requirements before running tests. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…on library In Python, ddtrace IS the test instrumentation library (equivalent to datadog-ci in Ruby). Remove the separate datadog-test-lib requirement and simplify to only check ddtrace >= 4.10.3. Also refactor SanityCheck to inline the version checking since there's now only one package to verify. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Change GetPlatformEnv() to append --ddtrace to any existing PYTEST_ADDOPTS value instead of overwriting it. This allows users to set custom pytest options while still ensuring ddtrace is enabled. If PYTEST_ADDOPTS is already set: PYTEST_ADDOPTS += ' --ddtrace' If PYTEST_ADDOPTS is not set: PYTEST_ADDOPTS = '--ddtrace' Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When --tests-location was set, the pattern was resolved and logged but never passed to pytest --collect-only, so the flag had no effect on discovery while correctly constraining DiscoverTestFiles (the fast glob fallback). Fix: when tests-location is set, resolve the glob to actual file paths via DiscoverTestFiles and append them to the pytest --collect-only command. When using the default pattern, pass no file args and let pytest handle discovery with its own config. Regression tests added for both cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pip show only works when pip is the package manager. importlib.metadata queries Python's own package metadata and works with any installer: pip, uv, poetry, conda, etc. The output is a bare version string (e.g. "4.10.3") so parsePipShowVersion is no longer needed and is removed along with the strings import that was only used by it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…fault
Previously the default was hardcoded to tests/**/*_test.py which:
- only matched *_test.py, missing the more common test_*.py convention
- assumed a tests/ root dir, breaking projects using test/, src/tests/, etc.
Now testPatterns() is resolved with the following priority:
1. --tests-location flag (explicit user override, unchanged)
2. pytest.ini / pyproject.toml / tox.ini / setup.cfg — read testpaths and
python_files and combine them into patterns (e.g. tests/**/test_*.py)
3. Built-in fallback: **/{test_*,*_test}.py (matches both naming
conventions everywhere, same as buildkite/test-engine-client)
DiscoverTestFiles now globs over all patterns, deduplicating results.
The TOML parser (pelletier/go-toml/v2, already a transitive dep) is
promoted to a direct dependency for pyproject.toml support.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
anmarchenko
reviewed
Jun 10, 2026
| return tests, nil | ||
| } | ||
|
|
||
| func (p *PyTest) DiscoverTestFiles() ([]string, error) { |
Member
There was a problem hiding this comment.
you can probably reuse globTestFiles function defined in framework.go
Member
Author
There was a problem hiding this comment.
globTestFiles was used in line 88, I guess you mean to try yo simplify this function.
A small refactor around it is in commit c31a67d
Member
There was a problem hiding this comment.
yes, exactly, I was just under the impression that this is too long
Replace testPatterns() []string with testPattern() string that collapses multiple testpaths/python_files into brace-expansion syntax supported by doublestar, so DiscoverTestFiles can call globTestFiles directly like RSpec. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…mentations Replace DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED/FILE with the canonical DD_CI_TEST_DISCOVERY_MODE_ENABLED/DD_CI_TEST_DISCOVERY_OUTPUT_PATH env vars used by both dd-trace-py and datadog-ci-rb. Also remove --collect-only -q from DiscoverTests: the dd-trace-py plugin handles collection and exits via pytest_collection_finish, so the flag is redundant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…veryEnv Revert the rename to DD_CI_TEST_DISCOVERY_MODE_ENABLED / DD_CI_TEST_DISCOVERY_OUTPUT_PATH. The dd-trace-py plugin now reads DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED and DD_TEST_OPTIMIZATION_DISCOVERY_FILE, so framework.go stays unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Python packages use PEP 440 format (e.g. 4.12.0rc1) while our version parser expects semver-style hyphens (4.12.0-rc1). Normalize before parsing so that pre-release ddtrace builds pass the version check correctly. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When discovery is intentionally cancelled (TIA disabled, or 0 skippable tests returned), the killed subprocess is expected — not a failure. Check ctx.Err() at both log sites and emit DEBUG instead of WARN. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This reverts commit 694904a.
When the context is cancelled (TIA disabled or 0 skippable tests returned), exec.CommandContext wraps context.Canceled in the returned error. Use errors.Is(err, context.Canceled) to detect this — not ctx.Err() from outside — so the log site stays honest about what it's checking. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
exec.CommandContext only wraps context.Canceled in the exit error when
cmd.WaitDelay is set. Without it the error is a plain *exec.ExitError
("signal: killed"), so errors.Is(err, context.Canceled) is always false.
Check ctx.Err() != nil instead — the context is the authoritative source
for whether the subprocess was killed intentionally.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
These belong in a separate PR (#82) since they affect all platforms, not just pytest. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
ddtrace's pytest plugin reads DD_TEST_OPTIMIZATION_MANIFEST_FILE (not TEST_OPTIMIZATION_MANIFEST_FILE) to activate manifest mode, where it reads settings and skippable tests from the cache files written during ddtest plan instead of making redundant HTTP calls per worker. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ddtrace's manifest mode now distinguishes version 1 (Bazel — skipping disabled) from version 2 (ddtest — skipping applied from cached skippable_tests.json). Bumping to 2 opts ddtest workers into reading the cached skippable tests fetched during the plan phase. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The dd-trace-py approach changed: instead of using manifest version to distinguish Bazel from ddtest, get_skippable_tests now returns a no-op only when DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES is set (the Bazel payload-files signal), so no version bump is needed here. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
python_env.py hardcoded "python" for runtime.name and used sys.platform
("linux") for os.platform. Both ddtrace.testing and the public CI
visibility plugin use platform.python_implementation() ("CPython") and
platform.system() ("Linux"). The skippable tests filter is exact string
equality, so every API response entry was being silently dropped,
causing tiaSkippableTestsCount=0 on every run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- main.go: read DDTEST_LOG_LEVEL or DD_LOG_LEVEL env vars at startup and configure slog at LevelDebug; without this all slog.Debug calls were silently dropped (Go slog defaults to INFO) - client.go: log up to 5 sample skippable test keys at debug level (format: test.bundle.suite.name.params) so the plan-phase query can be compared against discovery output - discovered_tests.go: log up to 5 sample discovered test IDs at debug level and promote "Test will be skipped" from absent to debug Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The test calls DetectPlatform() which runs SanityCheck() against the real Python environment. CI doesn't have ddtrace installed, so it always failed. Guard with t.Skip when python or ddtrace is absent so it runs as an integration test where the deps are present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
staticcheck SA1012: do not pass a nil Context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Motivation
Additional Notes
How to test the change?