Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 53 additions & 27 deletions .github/workflows/conda-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,6 @@ jobs:
shell: ${{ matrix.os == 'windows-2022' && 'cmd /C CALL {0}' || 'bash -el {0}' }}

env:
build-conda-pkg-env: 'environments/build_conda_pkg.yml'
build-env-name: 'build'
python-label: ${{ endsWith(matrix.python_spec, 't') && format('{0}t', matrix.python) || matrix.python }}
python-conda-spec: ${{ matrix.python_spec || matrix.python }}

Expand All @@ -75,46 +73,74 @@ jobs:
persist-credentials: false
fetch-depth: 0

- name: Setup miniconda
id: setup_miniconda
continue-on-error: true
uses: conda-incubator/setup-miniconda@8ee1f361103df19b6f8c8655fd3967a8ecb162d5 # v4.0.1
# rattler-build is a standalone binary with no conda/Python dependency,
# so building conda packages with it doesn't need a full conda
# distribution on the runner - only rattler-build itself needs to be on
# PATH
- name: Setup rattler-build
uses: prefix-dev/rattler-build-action@1ca5f45832f419a46d1326ccc5861d7e14d67c44 # v0.2.39
with:
miniforge-version: latest
use-mamba: 'true'
conda-remove-defaults: 'true'
environment-file: ${{ env.build-conda-pkg-env }}
activate-environment: ${{ env.build-env-name }}

- name: ReSetup miniconda
if: steps.setup_miniconda.outcome == 'failure'
uses: conda-incubator/setup-miniconda@8ee1f361103df19b6f8c8655fd3967a8ecb162d5 # v4.0.1
with:
miniforge-version: latest
use-mamba: 'true'
conda-remove-defaults: 'true'
environment-file: ${{ env.build-conda-pkg-env }}
activate-environment: ${{ env.build-env-name }}
setup-only: true

- name: List installed packages
run: mamba list
- name: Check rattler-build version
run: rattler-build --version

- name: Store conda paths as envs
- name: Store build output paths
shell: bash -el {0}
run: |
echo "CONDA_BLD=$CONDA_PREFIX/conda-bld/${{ runner.os == 'Linux' && 'linux' || 'win' }}-64/" | tr "\\\\" '/' >> "$GITHUB_ENV"
echo "CONDA_BLD=$GITHUB_WORKSPACE/output/${{ runner.os == 'Linux' && 'linux-64' || 'win-64' }}/" >> "$GITHUB_ENV"
echo "WHEELS_OUTPUT_FOLDER=$GITHUB_WORKSPACE${{ runner.os == 'Linux' && '/' || '\\' }}" >> "$GITHUB_ENV"

- name: Compute version from git describe
shell: bash -el {0}
run: |
DESCRIBE=$(git describe --tags --long)
echo "GIT_DESCRIBE_TAG=$(echo "$DESCRIBE" | sed -E 's/-[0-9]+-g[0-9a-f]+$//')" >> "$GITHUB_ENV"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems rattler-build support that on recipe level also. But that can be left for the follow-up.

echo "GIT_DESCRIBE_NUMBER=$(echo "$DESCRIBE" | sed -E 's/^.*-([0-9]+)-g[0-9a-f]+$/\1/')" >> "$GITHUB_ENV"

# rattler-build looks for build.bat on Windows (build.sh/build.bat is the
# symmetric naming it expects), while conda-build is still used for this
# recipe in non-GitHub CI and required bld.bat.
# Keep bld.bat as the single real source file and
# just copy it under the name rattler-build needs, so both build systems
# keep working off the same script without maintaining two duplicates
- name: Prepare build.bat for rattler-build

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible to specify the names of build scripts explicitly in recipe file:

build:
  script:
    file: ${{ "bld.bat" if win else "build.sh" }}

and that step can be dropped then

if: runner.os == 'Windows'
shell: bash -el {0}
run: cp conda-recipe/bld.bat conda-recipe/build.bat

- name: Build conda package
id: build_conda_pkg
continue-on-error: true
run: conda-build --no-test --python "${{ env.python-conda-spec }}" --numpy 2.0 ${{ env.channels-list }} conda-recipe
shell: bash -el {0}
run: |
read -ra RATTLER_CHANNELS <<< "$(echo "${{ env.channels-list }}" | sed 's/--override-channels//')"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To reduce the duplication we can define all using build options as build-args in prefix-dev/rattler-build-action

rattler-build build \
--recipe conda-recipe/rattler_recipe.yaml \
--no-include-recipe \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to pass --test skip , since we are running tests only in separate dedicated job

--channel-priority disabled \
--target-platform ${{ runner.os == 'Linux' && 'linux-64' || 'win-64' }} \
--variant-config conda-recipe/conda_build_config.yaml \
--variant "python=${{ env.python-conda-spec }}" \
--variant numpy=2 \
"${RATTLER_CHANNELS[@]}"
env:
MAX_BUILD_CMPL_MKL_VERSION: '2027.0a0'

- name: ReBuild conda package
if: steps.build_conda_pkg.outcome == 'failure'
run: conda-build --no-test --python "${{ env.python-conda-spec }}" --numpy 2.0 ${{ env.channels-list }} conda-recipe
shell: bash -el {0}
run: |
read -ra RATTLER_CHANNELS <<< "$(echo "${{ env.channels-list }}" | sed 's/--override-channels//')"
rattler-build build \
--recipe conda-recipe/rattler_recipe.yaml \
--no-include-recipe \
--channel-priority disabled \
--target-platform ${{ runner.os == 'Linux' && 'linux-64' || 'win-64' }} \
--variant-config conda-recipe/conda_build_config.yaml \
--variant "python=${{ env.python-conda-spec }}" \
--variant numpy=2 \
"${RATTLER_CHANNELS[@]}"
env:
MAX_BUILD_CMPL_MKL_VERSION: '2027.0a0'

Expand Down
121 changes: 121 additions & 0 deletions conda-recipe/rattler_recipe.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
schema_version: 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please populate changelog stating that rattler-build compatible version of recipe has ben added.


context:
max_compiler_and_mkl_version: ${{ env.get("MAX_BUILD_CMPL_MKL_VERSION", default="2027.0a0") }}
required_compiler_version: ${{ env.get("MIN_BUILD_CMPL_VERSION", default="2026.1.1") }}
required_mkl_version: "2026.0"
required_dpctl_version: 0.23.0dev0

package:
name: dpnp
version: ${{ env.get("GIT_DESCRIBE_TAG") }}

source:
path: ..

build:
number: ${{ env.get("GIT_DESCRIBE_NUMBER") }}
script:
env:
WHEELS_OUTPUT_FOLDER: ${{ env.get("WHEELS_OUTPUT_FOLDER", default="") }}
OVERRIDE_INTEL_IPO: ${{ env.get("OVERRIDE_INTEL_IPO", default="") }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OVERRIDE_INTEL_IPO is only applicable to Windows

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be better to update bld.bat also, to use more robust check: if not "%OVERRIDE_INTEL_IPO%"=="", because the default behavior might be changed now a bit (empty string "" rather then undefined env variable)

# conda-build used to forward GIT_DESCRIBE_NUMBER into the build script
# automatically; rattler-build does not, so bld.bat's
# `wheel tags --remove --build %GIT_DESCRIBE_NUMBER%` was silently
# getting an empty value on Windows, which made that command fail and
# broke the wheel install step further down the script.
GIT_DESCRIBE_NUMBER: ${{ env.get("GIT_DESCRIBE_NUMBER") }}
dynamic_linking:
missing_dso_allowlist:
- "*DPCTLSyclInterface*"
- "*/dpnp_backend_c.dll"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"*/dpnp_backend_c.dll" is only applicable to Windows


requirements:
host:
- python
- pip >=25.0
- pybind11 >=3.0.2
- python-build>=1.2.2
- cmake>=3.31.6
- cython>=3.1.0
- dpctl >=${{ required_dpctl_version }}
- if: not win
then: ninja>=1.11.1
- numpy
- scikit-build>=0.18.1
- setuptools>=79.0.1
- wheel>=0.45.1
- versioneer ==0.29
# versioneer dependency
- if: match(python, "<3.11")
then: tomli
# Do not use pyproject.toml for setting dependencies on OneAPI packages
- mkl-devel-dpcpp >=${{ required_mkl_version }},<${{ max_compiler_and_mkl_version }}
- onedpl-devel
- tbb-devel
- if: win
then: opencl-headers >=2025.06.13
build:
- ${{ compiler('cxx') }}
- ${{ stdlib('c') }}
- ${{ compiler('dpcpp') }} >=${{ required_compiler_version }},<${{ max_compiler_and_mkl_version }}
run:
- python
- ${{ pin_compatible('dpctl', lower_bound='x.x.x', upper_bound=None) }}
- ${{ pin_compatible('dpcpp-cpp-rt', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('intel-sycl-rt', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('intel-cmplr-lib-rt', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('mkl', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('onemkl-sycl-blas', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('onemkl-sycl-dft', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('onemkl-sycl-lapack', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('onemkl-sycl-rng', lower_bound='x.x', upper_bound='x') }}
- ${{ pin_compatible('onemkl-sycl-vm', lower_bound='x.x', upper_bound='x') }}
- numpy
- if: linux
then: ocl-icd-system
- if: win
then: khronos-opencl-icd-loader
ignore_run_exports:
by_name:
- dpctl
- numpy
- python
# building with Intel DPC++ and do not import the MSVC runtime DLLs
- if: win
then: vc14_runtime
- if: win
then: ucrt

tests:
- script:
- python -c "import dpnp; print(dpnp.__version__)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add run_test.bat and run_test.sh running explicitly, since rattler-build will not automatically do that anymore:

tests:
  # quick smoke checks
  - script:
      - pip check
      - python -c "import dpnp; print(dpnp.__version__)"
      - python -m dpctl -f
    requirements:
      run:
        - pip
  # full suite via the existing scripts
  - script: ${{ "run_test.bat" if win else "run_test.sh" }}
    requirements:
      run:
        - pytest
        - setuptools
        - if: match(python, "<3.14")
          then: scipy

requirements:
run:
- pytest
- setuptools
- if: match(python, "<3.14")
then: scipy

about:
license: BSD-3-Clause
license_file: LICENSE.txt
summary: Data Parallel Extension for NumPy
description: |
<strong>LEGAL NOTICE: Use of this software package is subject to the
software license agreement (as set forth above, in the license section of
the installed Conda package and/or the README file) and all notices,
disclaimers or license terms for third party or open source software
included in or with the software.</strong>
<br/><br/>
EULA: <a href="https://opensource.org/licenses/BSD-3-Clause" target="_blank">BSD-3-Clause</a>
<br/><br/>


homepage: https://github.com/IntelPython/dpnp

extra:
recipe-maintainers:
- antonwolfy
- vlad-perevezentsev
- ndgrigorian
Loading