fix: Guard against removing package while uploading to it (#64)

* fix: Guard against removing package while uploading to it

* On Anaconda Cloud as of 2024-01-04, if a wheel is being uploaded to a
  package, but the package only has one wheel in it and is of the same
  name as the uploaded wheel, Anaconda Cloud will overwrite the file by
  _removing_ the file from the package index. However, when this happens
  it removes the entire package, and then the wheel that is in the
  process of being uploaded has no destination and the upload fails.
  To guard against this, ensure for each package that has a wheel being
  uploaded if:
   - there is only one release for the package
   - and only 1 file for that release
   - and the upload target wheel has the same name as the file
   - that the file (and so the package) is removed in advance of the
     upload.
* To make filtering names and versions from wheels easier, add a
  get_wheel_name_version function that uses as regex to lazily capture
  the package name as well as the version and then return these.
   - Examples of this working:
     * "matplotlib-3.9.0.dev0-pp39-pypy39_pp73-win_amd64.whl"
       matplotlib 3.9.0.dev0

     * "scikit_learn-1.5.dev0-cp39-cp39-win_amd64.whl"
       scikit_learn 1.5.dev0

     * "scipy-openblas64-0.3.26.186-py3-none-macosx_10_9_x86_64.whl"
       scipy-openblas64 0.3.26.186

     * "awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
       awkward_cpp 29
* As Anaconda Cloud normalizes package names, also distinguish between
  basename and package name to try to make things easier to keep track
  of when normalizing.
  * Example:
     - basename: test_package-0.0.1-py3-none-any.whl
     - basename prefix: test_package
     - package_name: test-package

* MNT: Add curl and jq to environment

* As curl and jq are now used in cmd.sh, they need to also be added to
  the conda environment.yml.
* The lower bounds are chosen as the latest values, but are not
  motivated by known problems.
* Rebuild the lock file.

* CI: Add a test to revmove a package

* Add a test that triggers the conditions for removal of a package from
  Anaconda Cloud in advance of its upload to avoid an error.

Author: matthewfeickert

.github/workflows/ci.yml

@@ -53,6 +53,12 @@ jobs:
           artifacts_path: dist
           anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}
 
+      - name: Test upload that forces removal first
+        uses: ./
+        with:
+          artifacts_path: dist
+          anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}
+
       - name: Build v0.0.2 wheel and sdist
         run: |
           # Bump version to avoid wheel name conflicts

cmd.sh

@@ -67,6 +67,70 @@ micromamba activate upload-nightly-action
 # trim trailing slashes from $INPUT_ARTIFACTS_PATH
 INPUT_ARTIFACTS_PATH="${INPUT_ARTIFACTS_PATH%/}"
 
+get_wheel_name_version() {
+    local wheel_name="$1"
+    if [[ "${wheel_name}" =~ ^([[:alnum:]_-]+)-([0-9][^-]+)-(.+)$ ]]; then
+        # return the package_name and version_number
+        local return_values=("${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}")
+        echo "${return_values[@]}"
+    else
+        echo "The wheel name ${1} does not follow the PEP 491 spec (https://peps.python.org/pep-0491/) and is invalid."
+        return 1
+    fi
+}
+
+# get the unique package names from all the wheels
+package_names=()
+for wheel_path in "${INPUT_ARTIFACTS_PATH}"/*.whl; do
+  # remove the INPUT_ARTIFACTS_PATH/ prefix (including the /)
+  wheel_name="${wheel_path#${INPUT_ARTIFACTS_PATH}/}"
+  read -r package_basename_prefix _ <<< "$(get_wheel_name_version ${wheel_name})"
+  package_names+=("${package_basename_prefix}")
+done
+package_names=($(tr ' ' '\n' <<< "${package_names[@]}" | sort --unique | tr '\n' ' '))
+
+# If the package version doesn't exist on the package index then there will
+# be no files to overwrite and the package can be uploaded safely.
+# If the package version exists, is the only version on the package index,
+# and only has one distribution file, then that package needs to be removed
+# from the index before a wheel of the same name can be uploaded again.
+# c.f. https://github.com/Anaconda-Platform/anaconda-client/issues/702
+
+for package_basename_prefix in "${package_names[@]}"; do
+  # normalize package_name to use '-' as delimiter
+  package_name="${package_basename_prefix//_/-}"
+
+  number_releases=$(curl --silent https://api.anaconda.org/package/"${ANACONDA_ORG}/${package_name}" | \
+    jq -r '.releases' | \
+    jq length)
+
+  if [ "${number_releases}" -eq 1 ]; then
+    # get any wheel for the package (they should all have the same version)
+    wheel_path=$(find "${INPUT_ARTIFACTS_PATH}" -name "${package_basename_prefix}-*.whl" -print -quit)
+    wheel_name="${wheel_path#${INPUT_ARTIFACTS_PATH}/}"
+    read -r _ package_version <<< "$(get_wheel_name_version ${wheel_name})"
+
+    number_files=$(curl --silent https://api.anaconda.org/release/"${ANACONDA_ORG}/${package_name}/${package_version}" | \
+      jq -r '.distributions' | \
+      jq length)
+
+    if [ "${number_files}" -eq 1 ]; then
+      distribution_name=$(curl --silent https://api.anaconda.org/release/"${ANACONDA_ORG}/${package_name}/${package_version}" | \
+        jq -r '.distributions[].basename')
+
+      if [ "${wheel_name}" = "${distribution_name}" ]; then
+        echo -e "\n# ${distribution_name} is the only distribution file uploaded for the package https://anaconda.org/${ANACONDA_ORG}/${package_name}"
+        echo "# To avoid https://github.com/Anaconda-Platform/anaconda-client/issues/702 remove the existing release before uploading."
+
+        echo -e "\n# Removing ${ANACONDA_ORG}/${package_name}/${package_version}"
+        anaconda --token "${ANACONDA_TOKEN}" remove \
+          --force \
+          "${ANACONDA_ORG}/${package_name}/${package_version}"
+      fi
+    fi
+  fi
+done
+
 # upload wheels
 echo "Uploading wheels to anaconda.org..."