-
Notifications
You must be signed in to change notification settings - Fork 29
Migrate conda packaging to rattler-build (v1 recipe format) for GitHub Actions CI #3031
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
9af7c57
4c677df
9d616be
b15031c
ee3ec67
b2a68e2
b2bf4a8
939c991
4e0c6d0
8ba4cc1
da53abb
af48c14
9321553
d1820fc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,8 +58,6 @@ jobs: | |
| shell: ${{ matrix.os == 'windows-2022' && 'cmd /C CALL {0}' || 'bash -el {0}' }} | ||
|
|
||
| env: | ||
| build-conda-pkg-env: 'environments/build_conda_pkg.yml' | ||
| build-env-name: 'build' | ||
| python-label: ${{ endsWith(matrix.python_spec, 't') && format('{0}t', matrix.python) || matrix.python }} | ||
| python-conda-spec: ${{ matrix.python_spec || matrix.python }} | ||
|
|
||
|
|
@@ -75,46 +73,74 @@ jobs: | |
| persist-credentials: false | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Setup miniconda | ||
| id: setup_miniconda | ||
| continue-on-error: true | ||
| uses: conda-incubator/setup-miniconda@8ee1f361103df19b6f8c8655fd3967a8ecb162d5 # v4.0.1 | ||
| # rattler-build is a standalone binary with no conda/Python dependency, | ||
| # so building conda packages with it doesn't need a full conda | ||
| # distribution on the runner - only rattler-build itself needs to be on | ||
| # PATH | ||
| - name: Setup rattler-build | ||
| uses: prefix-dev/rattler-build-action@1ca5f45832f419a46d1326ccc5861d7e14d67c44 # v0.2.39 | ||
| with: | ||
| miniforge-version: latest | ||
| use-mamba: 'true' | ||
| conda-remove-defaults: 'true' | ||
| environment-file: ${{ env.build-conda-pkg-env }} | ||
| activate-environment: ${{ env.build-env-name }} | ||
|
|
||
| - name: ReSetup miniconda | ||
| if: steps.setup_miniconda.outcome == 'failure' | ||
| uses: conda-incubator/setup-miniconda@8ee1f361103df19b6f8c8655fd3967a8ecb162d5 # v4.0.1 | ||
| with: | ||
| miniforge-version: latest | ||
| use-mamba: 'true' | ||
| conda-remove-defaults: 'true' | ||
| environment-file: ${{ env.build-conda-pkg-env }} | ||
| activate-environment: ${{ env.build-env-name }} | ||
| setup-only: true | ||
|
|
||
| - name: List installed packages | ||
| run: mamba list | ||
| - name: Check rattler-build version | ||
| run: rattler-build --version | ||
|
|
||
| - name: Store conda paths as envs | ||
| - name: Store build output paths | ||
| shell: bash -el {0} | ||
| run: | | ||
| echo "CONDA_BLD=$CONDA_PREFIX/conda-bld/${{ runner.os == 'Linux' && 'linux' || 'win' }}-64/" | tr "\\\\" '/' >> "$GITHUB_ENV" | ||
| echo "CONDA_BLD=$GITHUB_WORKSPACE/output/${{ runner.os == 'Linux' && 'linux-64' || 'win-64' }}/" >> "$GITHUB_ENV" | ||
| echo "WHEELS_OUTPUT_FOLDER=$GITHUB_WORKSPACE${{ runner.os == 'Linux' && '/' || '\\' }}" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Compute version from git describe | ||
| shell: bash -el {0} | ||
| run: | | ||
| DESCRIBE=$(git describe --tags --long) | ||
| echo "GIT_DESCRIBE_TAG=$(echo "$DESCRIBE" | sed -E 's/-[0-9]+-g[0-9a-f]+$//')" >> "$GITHUB_ENV" | ||
| echo "GIT_DESCRIBE_NUMBER=$(echo "$DESCRIBE" | sed -E 's/^.*-([0-9]+)-g[0-9a-f]+$/\1/')" >> "$GITHUB_ENV" | ||
|
|
||
| # rattler-build looks for build.bat on Windows (build.sh/build.bat is the | ||
| # symmetric naming it expects), while conda-build is still used for this | ||
| # recipe in non-GitHub CI and required bld.bat. | ||
| # Keep bld.bat as the single real source file and | ||
| # just copy it under the name rattler-build needs, so both build systems | ||
| # keep working off the same script without maintaining two duplicates | ||
| - name: Prepare build.bat for rattler-build | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's possible to specify the names of build scripts explicitly in recipe file: build:
script:
file: ${{ "bld.bat" if win else "build.sh" }}and that step can be dropped then |
||
| if: runner.os == 'Windows' | ||
| shell: bash -el {0} | ||
| run: cp conda-recipe/bld.bat conda-recipe/build.bat | ||
|
|
||
| - name: Build conda package | ||
| id: build_conda_pkg | ||
| continue-on-error: true | ||
| run: conda-build --no-test --python "${{ env.python-conda-spec }}" --numpy 2.0 ${{ env.channels-list }} conda-recipe | ||
| shell: bash -el {0} | ||
| run: | | ||
| read -ra RATTLER_CHANNELS <<< "$(echo "${{ env.channels-list }}" | sed 's/--override-channels//')" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To reduce the duplication we can define all using build options as |
||
| rattler-build build \ | ||
| --recipe conda-recipe/rattler_recipe.yaml \ | ||
| --no-include-recipe \ | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We need to pass |
||
| --channel-priority disabled \ | ||
| --target-platform ${{ runner.os == 'Linux' && 'linux-64' || 'win-64' }} \ | ||
| --variant-config conda-recipe/conda_build_config.yaml \ | ||
| --variant "python=${{ env.python-conda-spec }}" \ | ||
| --variant numpy=2 \ | ||
| "${RATTLER_CHANNELS[@]}" | ||
| env: | ||
| MAX_BUILD_CMPL_MKL_VERSION: '2027.0a0' | ||
|
|
||
| - name: ReBuild conda package | ||
| if: steps.build_conda_pkg.outcome == 'failure' | ||
| run: conda-build --no-test --python "${{ env.python-conda-spec }}" --numpy 2.0 ${{ env.channels-list }} conda-recipe | ||
| shell: bash -el {0} | ||
| run: | | ||
| read -ra RATTLER_CHANNELS <<< "$(echo "${{ env.channels-list }}" | sed 's/--override-channels//')" | ||
| rattler-build build \ | ||
| --recipe conda-recipe/rattler_recipe.yaml \ | ||
| --no-include-recipe \ | ||
| --channel-priority disabled \ | ||
| --target-platform ${{ runner.os == 'Linux' && 'linux-64' || 'win-64' }} \ | ||
| --variant-config conda-recipe/conda_build_config.yaml \ | ||
| --variant "python=${{ env.python-conda-spec }}" \ | ||
| --variant numpy=2 \ | ||
| "${RATTLER_CHANNELS[@]}" | ||
| env: | ||
| MAX_BUILD_CMPL_MKL_VERSION: '2027.0a0' | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| schema_version: 1 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please populate changelog stating that rattler-build compatible version of recipe has ben added. |
||
|
|
||
| context: | ||
| max_compiler_and_mkl_version: ${{ env.get("MAX_BUILD_CMPL_MKL_VERSION", default="2027.0a0") }} | ||
| required_compiler_version: ${{ env.get("MIN_BUILD_CMPL_VERSION", default="2026.1.1") }} | ||
| required_mkl_version: "2026.0" | ||
| required_dpctl_version: 0.23.0dev0 | ||
|
|
||
| package: | ||
| name: dpnp | ||
| version: ${{ env.get("GIT_DESCRIBE_TAG") }} | ||
|
|
||
| source: | ||
| path: .. | ||
|
|
||
| build: | ||
| number: ${{ env.get("GIT_DESCRIBE_NUMBER") }} | ||
| script: | ||
| env: | ||
| WHEELS_OUTPUT_FOLDER: ${{ env.get("WHEELS_OUTPUT_FOLDER", default="") }} | ||
| OVERRIDE_INTEL_IPO: ${{ env.get("OVERRIDE_INTEL_IPO", default="") }} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It'd be better to update |
||
| # conda-build used to forward GIT_DESCRIBE_NUMBER into the build script | ||
| # automatically; rattler-build does not, so bld.bat's | ||
| # `wheel tags --remove --build %GIT_DESCRIBE_NUMBER%` was silently | ||
| # getting an empty value on Windows, which made that command fail and | ||
| # broke the wheel install step further down the script. | ||
| GIT_DESCRIBE_NUMBER: ${{ env.get("GIT_DESCRIBE_NUMBER") }} | ||
| dynamic_linking: | ||
| missing_dso_allowlist: | ||
| - "*DPCTLSyclInterface*" | ||
| - "*/dpnp_backend_c.dll" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| requirements: | ||
| host: | ||
| - python | ||
| - pip >=25.0 | ||
| - pybind11 >=3.0.2 | ||
| - python-build>=1.2.2 | ||
| - cmake>=3.31.6 | ||
| - cython>=3.1.0 | ||
| - dpctl >=${{ required_dpctl_version }} | ||
| - if: not win | ||
| then: ninja>=1.11.1 | ||
| - numpy | ||
| - scikit-build>=0.18.1 | ||
| - setuptools>=79.0.1 | ||
| - wheel>=0.45.1 | ||
| - versioneer ==0.29 | ||
| # versioneer dependency | ||
| - if: match(python, "<3.11") | ||
| then: tomli | ||
| # Do not use pyproject.toml for setting dependencies on OneAPI packages | ||
| - mkl-devel-dpcpp >=${{ required_mkl_version }},<${{ max_compiler_and_mkl_version }} | ||
| - onedpl-devel | ||
| - tbb-devel | ||
| - if: win | ||
| then: opencl-headers >=2025.06.13 | ||
| build: | ||
| - ${{ compiler('cxx') }} | ||
| - ${{ stdlib('c') }} | ||
| - ${{ compiler('dpcpp') }} >=${{ required_compiler_version }},<${{ max_compiler_and_mkl_version }} | ||
| run: | ||
| - python | ||
| - ${{ pin_compatible('dpctl', lower_bound='x.x.x', upper_bound=None) }} | ||
| - ${{ pin_compatible('dpcpp-cpp-rt', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('intel-sycl-rt', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('intel-cmplr-lib-rt', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('mkl', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('onemkl-sycl-blas', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('onemkl-sycl-dft', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('onemkl-sycl-lapack', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('onemkl-sycl-rng', lower_bound='x.x', upper_bound='x') }} | ||
| - ${{ pin_compatible('onemkl-sycl-vm', lower_bound='x.x', upper_bound='x') }} | ||
| - numpy | ||
| - if: linux | ||
| then: ocl-icd-system | ||
| - if: win | ||
| then: khronos-opencl-icd-loader | ||
| ignore_run_exports: | ||
| by_name: | ||
| - dpctl | ||
| - numpy | ||
| - python | ||
| # building with Intel DPC++ and do not import the MSVC runtime DLLs | ||
| - if: win | ||
| then: vc14_runtime | ||
| - if: win | ||
| then: ucrt | ||
|
|
||
| tests: | ||
| - script: | ||
| - python -c "import dpnp; print(dpnp.__version__)" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We need to add tests:
# quick smoke checks
- script:
- pip check
- python -c "import dpnp; print(dpnp.__version__)"
- python -m dpctl -f
requirements:
run:
- pip
# full suite via the existing scripts
- script: ${{ "run_test.bat" if win else "run_test.sh" }}
requirements:
run:
- pytest
- setuptools
- if: match(python, "<3.14")
then: scipy |
||
| requirements: | ||
| run: | ||
| - pytest | ||
| - setuptools | ||
| - if: match(python, "<3.14") | ||
| then: scipy | ||
|
|
||
| about: | ||
| license: BSD-3-Clause | ||
| license_file: LICENSE.txt | ||
| summary: Data Parallel Extension for NumPy | ||
| description: | | ||
| <strong>LEGAL NOTICE: Use of this software package is subject to the | ||
| software license agreement (as set forth above, in the license section of | ||
| the installed Conda package and/or the README file) and all notices, | ||
| disclaimers or license terms for third party or open source software | ||
| included in or with the software.</strong> | ||
| <br/><br/> | ||
| EULA: <a href="https://opensource.org/licenses/BSD-3-Clause" target="_blank">BSD-3-Clause</a> | ||
| <br/><br/> | ||
|
|
||
|
|
||
| homepage: https://github.com/IntelPython/dpnp | ||
|
|
||
| extra: | ||
| recipe-maintainers: | ||
| - antonwolfy | ||
| - vlad-perevezentsev | ||
| - ndgrigorian | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems rattler-build support that on recipe level also. But that can be left for the follow-up.