| title | CI |
|---|---|
| description | Our test setup and how our CI files are generated. |
| sidebar_order | 20 |
Our test suite consists of a common part that tests basic SDK functionality, and an integrations part that tests individual integrations.
Since the number of integrations to test is so large, the tests in CI are split into multiple groups (roughly by integration type) so that they can run in parallel. So there is, for example, a GitHub workflow that runs all our feature flag integrations, and another one that runs the test suites of all GraphQL integrations, and so on. These workflow YAMLs are auto-generated by this script.
The ultimate source of truth of what should be tested and on which versions (both Python versions as well as package versions) is stored in tox.ini. This file is also auto-generated each week by a GitHub cron job that scans PyPI for new releases and updates the test matrix with them. The script that does this lives in scripts/populate_tox and is often referred to as "toxgen".
There are multiple components to the test setup that runs in CI:
- on a weekly basis,
scripts/populate_toxcreates a PR that updatestox.iniwith new releases tox.iniis where the configuration (what to test and what versions) is stored and read fromscripts/split-tox-gh-actions/split-tox-gh-actions.pyis the script that defines the test groups and generates the resulting YAML filesscripts/split-tox-gh-actions/templatescontains the jinja2 template snippets used to generate the YAML files.github/workflows/test-integrations-*.ymlare the resulting workflow configuration YAMLs
If you update any of the components without keeping the rest in sync, for instance by making a change in one of the test-integrations-*.yml files that is not reflected in the templates in scripts/split-tox-gh-actions/templates, they will be overwritten next time they're auto-generated.