Migrate conda packaging to rattler-build (v1 recipe format) for GitHub Actions CI - #3031
Migrate conda packaging to rattler-build (v1 recipe format) for GitHub Actions CI#3031ekomarova wants to merge 14 commits into
Conversation
3991bb9 to
ee3ec67
Compare
b9750b9 to
b2a68e2
Compare
|
@antonwolfy @ndgrigorian It looks like I've managed to stabilize GitHub CI, so please take a look at it when you have time. I would also consider getting rid of installing |
|
|
||
| tests: | ||
| - script: | ||
| - python -c "import dpnp; print(dpnp.__version__)" |
There was a problem hiding this comment.
We need to add run_test.bat and run_test.sh running explicitly, since rattler-build will not automatically do that anymore:
tests:
# quick smoke checks
- script:
- pip check
- python -c "import dpnp; print(dpnp.__version__)"
- python -m dpctl -f
requirements:
run:
- pip
# full suite via the existing scripts
- script: ${{ "run_test.bat" if win else "run_test.sh" }}
requirements:
run:
- pytest
- setuptools
- if: match(python, "<3.14")
then: scipy| read -ra RATTLER_CHANNELS <<< "$(echo "${{ env.channels-list }}" | sed 's/--override-channels//')" | ||
| rattler-build build \ | ||
| --recipe conda-recipe/rattler_recipe.yaml \ | ||
| --no-include-recipe \ |
There was a problem hiding this comment.
We need to pass --test skip , since we are running tests only in separate dedicated job
| # Keep bld.bat as the single real source file and | ||
| # just copy it under the name rattler-build needs, so both build systems | ||
| # keep working off the same script without maintaining two duplicates | ||
| - name: Prepare build.bat for rattler-build |
There was a problem hiding this comment.
It's possible to specify the names of build scripts explicitly in recipe file:
build:
script:
file: ${{ "bld.bat" if win else "build.sh" }}and that step can be dropped then
| script: | ||
| env: | ||
| WHEELS_OUTPUT_FOLDER: ${{ env.get("WHEELS_OUTPUT_FOLDER", default="") }} | ||
| OVERRIDE_INTEL_IPO: ${{ env.get("OVERRIDE_INTEL_IPO", default="") }} |
There was a problem hiding this comment.
OVERRIDE_INTEL_IPO is only applicable to Windows
| dynamic_linking: | ||
| missing_dso_allowlist: | ||
| - "*DPCTLSyclInterface*" | ||
| - "*/dpnp_backend_c.dll" |
There was a problem hiding this comment.
"*/dpnp_backend_c.dll" is only applicable to Windows
| script: | ||
| env: | ||
| WHEELS_OUTPUT_FOLDER: ${{ env.get("WHEELS_OUTPUT_FOLDER", default="") }} | ||
| OVERRIDE_INTEL_IPO: ${{ env.get("OVERRIDE_INTEL_IPO", default="") }} |
There was a problem hiding this comment.
It'd be better to update bld.bat also, to use more robust check: if not "%OVERRIDE_INTEL_IPO%"=="", because the default behavior might be changed now a bit (empty string "" rather then undefined env variable)
| shell: bash -el {0} | ||
| run: | | ||
| DESCRIBE=$(git describe --tags --long) | ||
| echo "GIT_DESCRIBE_TAG=$(echo "$DESCRIBE" | sed -E 's/-[0-9]+-g[0-9a-f]+$//')" >> "$GITHUB_ENV" |
There was a problem hiding this comment.
It seems rattler-build support that on recipe level also. But that can be left for the follow-up.
| run: conda-build --no-test --python "${{ env.python-conda-spec }}" --numpy 2.0 ${{ env.channels-list }} conda-recipe | ||
| shell: bash -el {0} | ||
| run: | | ||
| read -ra RATTLER_CHANNELS <<< "$(echo "${{ env.channels-list }}" | sed 's/--override-channels//')" |
There was a problem hiding this comment.
To reduce the duplication we can define all using build options as build-args in prefix-dev/rattler-build-action
| @@ -0,0 +1,121 @@ | |||
| schema_version: 1 | |||
There was a problem hiding this comment.
Please populate changelog stating that rattler-build compatible version of recipe has ben added.
Add rattler-build support for GitHub Actions CI
Summary
This PR adds rattler-build as the package-building tool for the Conda package GitHub Actions workflow, alongside a new
rattler_recipe.yaml(v1 recipe schema) generated from the existingmeta.yaml.meta.yamlis left untouched and fully functional. This PR only changes what runs on GitHub Actions; it does not retire or replace theconda-buildpath.What changed
.github/workflows/conda-package.yml: build steps now call rattler-build instead of conda-buildconda-recipe/rattler_recipe.yaml: new file, the rattler-build (v1 schema) equivalent ofmeta.yaml. A handful of things had to be restructured because v1 format must always be valid plain YAML (no Jinja control-flow blocks)environments/build_conda_pkg.yml: no longer needed for useconda-recipe/bld.bat: unchanged, still the source file forconda-build; a CI step copies it tobuild.baton Windows runners since that's the filenamerattler-buildexpectsWhy these specific changes were needed
CONDA_BLD path —
rattler-buildoutputs tooutput/<platform>/relative to the working directory by default, unlikeconda-build's $CONDA_PREFIX/conda-bld/Compute version from git describestep —conda-buildrunsgit describeinternally before rendering to populate GIT_DESCRIBE_TAG/GIT_DESCRIBE_NUMBER.rattler-buildhas no equivalent, sorattler_recipe.yamlnow reads these viaenv.get(...), and we compute + export them ourselves beforehand.Prepare build.bat for rattler-buildstep —rattler-buildexpectsbuild.bat(symmetric withbuild.sh), whileconda-build(still used in non-GitHub CI) only recognizesbld.bat. Rather than duplicating the file,bld.batstays the single source and gets copied tobuild.batat CI time on Windows.RATTLER_CHANNELS array —
channels-listincludes--override-channels, whichrattler-build's CLI doesn't recognize, so it's stripped before use. Expanded into a bash array (instead of a bare $VAR) so each-c <channel>reachesrattler-buildas a separate argument — also fixes a shellcheck SC2086 warning.Removed usage of
conda-verify— it's a conda-build-specific post-build checker;rattler-buildvalidates internally, so it's no longer needed.Removed the two guard blocks and the
pyproject.tomlloop inrattler_recipe.yaml— v1 format must always be valid plain YAML, so Jinja control-flow ({% if %}/{% for %}) isn't supported. The version guard checks are dropped (an invalid override still fails to build, just via a less friendly solver error instead of a custom message), and the dependency list previously generated frompyproject.toml's [build-system]requires is now static — it needs manual updates if those requirements change.Added
tests: - script: ...block — v1 schema requires an explicit test type; the oldtest: requires:block installed test deps without running anything, which isn't valid anymore. Added a minimal import/version smoke testReplace
minicondawithrattler-buildaction in the build job —rattler-buildis a standalone binary with no conda/Python dependency, so building conda packages with it doesn't need a full conda distribution on the runner. The installation takes about 1s