Maintenance: Refactor artifact name strategy for parallel uploads

[dreamorosi]

Summary

GitHub actions/upload-artifact v4 has a racing condition when uploading artifacts with the same artifact name in parallel.

For example, our Lambda Layer CDK artifacts are uploaded on a per region deploy in parallel using the a static output name.

While in our repo the workflow ran successfully, the Python version of Powertools was not as lucky and their release failed partially (rerunning the failed jobs serves as a short term workaround).

Related to aws-powertools/powertools-lambda-python#3748

Why is this needed?

N/A

Which area does this relate to?

No response

Solution

Since it's a race condition on upload, we need a dynamic artifact name e.g., one per region. Then merge these artifacts on download.

Updating to a dynamic name

      - name: Download CDK layer artifact
        uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe # v4.1.2
        with:
          name: cdk-layer-stack
          path: cdk-layer-stack/
          merge-multiple: true

Downloading all artifacts for all regions

      - name: Save Layer ARN artifact
        if: ${{ inputs.stage == 'PROD' }}
        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1
        with:
          name: cdk-layer-stack-{{ matrix.region }}
          path: ./layer/cdk-layer-stack/* # NOTE: upload-artifact does not inherit working-directory setting.
          if-no-files-found: error
          retention-days: 1
          overwrite: true   # still needed to allow failed retries

Or using actions/upload-artifact#505

We'll need a testing environment to create a parallel job to test this out.

Acknowledgment

• This request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.