-
Notifications
You must be signed in to change notification settings - Fork 382
Migrate Python checks to CI Gate #11522
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
msyyc
wants to merge
17
commits into
microsoft:main
Choose a base branch
from
msyyc:msyyc-migrate-python-ci-gate
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
e1b262e
ci: migrate Python checks to CI Gate
msyyc 6cdf761
ci: share Python workflow template
msyyc c126768
ci: keep Python CI wrappers distinct
msyyc b0d6b2f
ci: fix Python workflow gating
msyyc 62cde21
ci: use local Python setup for Python CI
msyyc 6e9597f
ci: simplify Python CI workflow
msyyc 295e2f0
ci: add standalone Python CI Gate workflow
msyyc 60c7525
ci: avoid Python CI lockfile mutation
msyyc 9fa8055
ci: prepare Python test environment
msyyc 524568e
ci: align Python CI setup with integration flow
msyyc 2b191d9
ci: run Windows Python build and test together
msyyc 021600f
ci: upload Python wheel artifacts
msyyc 21f0f56
ci: restore Python artifacts into package path
msyyc cff9838
ci: declare Python ESLint dependencies
msyyc a3a70d1
ci: isolate Python formatter config
msyyc fbec5e5
ci: build Prettier plugin for Python format check
msyyc 9c8dab7
Merge branch 'main' into msyyc-migrate-python-ci-gate
msyyc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| changeKind: internal | ||
| packages: | ||
| - "@typespec/http-client-python" | ||
| --- | ||
|
|
||
| Move Python emitter PR validation into GitHub Actions CI Gate. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,242 @@ | ||
| name: Python CI | ||
|
|
||
| on: | ||
| workflow_call: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| build: | ||
| name: "Build & Regenerate (Linux)" | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - uses: ./.github/actions/setup | ||
|
|
||
| - uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Install repo dependencies | ||
| run: pnpm install | ||
|
|
||
| - name: Setup http-client-python | ||
| # Avoid npm lifecycle side effects and lockfile rewrites in CI; setup still runs build + generator install. | ||
| run: | | ||
| npm install --ignore-scripts --package-lock=false | ||
| npm run setup | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Prepare Python environment | ||
| # Run prepare.py directly so missing tox/dev dependencies fail the job instead of being swallowed by the npm wrapper. | ||
| run: venv/bin/python eng/scripts/setup/prepare.py | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Regenerate | ||
| run: npm run regenerate | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Pre-build wheels | ||
| run: | | ||
| venv/bin/python tests/install_packages.py build azure tests | ||
| venv/bin/python tests/install_packages.py build unbranded tests | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Check for unstaged changes | ||
| run: npm run check-unstaged | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Upload generated artifacts | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: python-generated | ||
| include-hidden-files: true | ||
| path: | | ||
| packages/http-client-python/tests/generated | ||
| packages/http-client-python/tests/.wheels | ||
| packages/http-client-python/dist | ||
| packages/http-client-python/generator/dist | ||
| retention-days: 1 | ||
|
|
||
| test: | ||
| name: "Mock API Tests" | ||
| needs: build | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - uses: ./.github/actions/setup | ||
|
|
||
| - uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Download generated artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: python-generated | ||
| path: packages/http-client-python | ||
|
|
||
| - name: Install repo dependencies | ||
| run: pnpm install | ||
|
|
||
| - name: Setup http-client-python | ||
| # Generated artifacts come from the build job; downstream jobs only need the Python package installed and test tools prepared. | ||
| run: | | ||
| npm install --ignore-scripts --package-lock=false | ||
| npm run install | ||
| venv/bin/python eng/scripts/setup/prepare.py | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Test | ||
| run: npm run test:generator -- --env=test | ||
| working-directory: packages/http-client-python | ||
|
|
||
| typecheck: | ||
| name: "Type Checking" | ||
| needs: build | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - uses: ./.github/actions/setup | ||
|
|
||
| - uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Download generated artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: python-generated | ||
| path: packages/http-client-python | ||
|
|
||
| - name: Install repo dependencies | ||
| run: pnpm install | ||
|
|
||
| - name: Setup http-client-python | ||
| # Generated artifacts come from the build job; downstream jobs only need the Python package installed and test tools prepared. | ||
| run: | | ||
| npm install --ignore-scripts --package-lock=false | ||
| npm run install | ||
| venv/bin/python eng/scripts/setup/prepare.py | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Mypy & Pyright | ||
| run: npm run test:generator -- --env=mypy,pyright | ||
| working-directory: packages/http-client-python | ||
|
|
||
| lint: | ||
| name: "Lint & Format" | ||
| needs: build | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - uses: ./.github/actions/setup | ||
|
|
||
| - uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Download generated artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: python-generated | ||
| path: packages/http-client-python | ||
|
|
||
| - name: Install repo dependencies | ||
| run: pnpm install | ||
|
|
||
| - name: Setup http-client-python | ||
| # Generated artifacts come from the build job; downstream jobs only need the Python package installed and test tools prepared. | ||
| run: | | ||
| npm install --ignore-scripts --package-lock=false | ||
| npm run install | ||
| venv/bin/python eng/scripts/setup/prepare.py | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Pylint | ||
| run: npm run test:generator -- --env=lint | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Lint source | ||
| run: npm run lint | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Build TypeSpec Prettier plugin | ||
| run: pnpm --filter "@typespec/prettier-plugin-typespec..." run build | ||
|
|
||
| - name: Format check source | ||
| run: npm run format:check | ||
| working-directory: packages/http-client-python | ||
|
|
||
| docs: | ||
| name: "Docs Validation" | ||
| needs: build | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - uses: ./.github/actions/setup | ||
|
|
||
| - uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Download generated artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: python-generated | ||
| path: packages/http-client-python | ||
|
|
||
| - name: Install repo dependencies | ||
| run: pnpm install | ||
|
|
||
| - name: Setup http-client-python | ||
| # Generated artifacts come from the build job; downstream jobs only need the Python package installed and test tools prepared. | ||
| run: | | ||
| npm install --ignore-scripts --package-lock=false | ||
| npm run install | ||
| venv/bin/python eng/scripts/setup/prepare.py | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: API View & Sphinx | ||
| run: npm run test:generator -- --env=apiview,sphinx | ||
| working-directory: packages/http-client-python | ||
|
|
||
| windows-test: | ||
| name: "Build & Regenerate && Test (Windows)" | ||
| runs-on: windows-latest | ||
| steps: | ||
| - name: Enable Git long paths | ||
| run: git config --system core.longpaths true | ||
|
|
||
| - uses: actions/checkout@v7 | ||
|
|
||
| - uses: ./.github/actions/setup | ||
|
|
||
| - uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Install repo dependencies | ||
| run: pnpm install | ||
|
|
||
| - name: Setup http-client-python | ||
| # Windows validates build/regenerate/test in one job, so run the full setup instead of restoring Linux artifacts. | ||
| run: | | ||
| npm install --ignore-scripts --package-lock=false | ||
| npm run setup | ||
| .\venv\Scripts\python.exe eng\scripts\setup\prepare.py | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Regenerate | ||
| run: npm run regenerate | ||
| working-directory: packages/http-client-python | ||
|
|
||
| - name: Test | ||
| run: npm run test:generator -- --env=test | ||
| working-directory: packages/http-client-python | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,7 @@ on: | |
|
|
||
| permissions: | ||
| contents: read | ||
| pull-requests: read | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
|
|
@@ -22,12 +23,13 @@ jobs: | |
| name: Detect Changes | ||
| runs-on: ubuntu-slim | ||
| outputs: | ||
| core: ${{ steps.filter.outputs.core }} | ||
| core: ${{ steps.core-filter.outputs.core }} | ||
| python: ${{ steps.python-filter.outputs.python }} | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| # cspell:ignore dorny | ||
| - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4 | ||
| id: filter | ||
| id: core-filter | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that tool was letting multiple filters to be defined in the same task |
||
| with: | ||
| predicate-quantifier: "every" | ||
| filters: | | ||
|
|
@@ -42,22 +44,39 @@ jobs: | |
| - '!packages/http-client-csharp/**' | ||
| - '!packages/http-client-java/**' | ||
| - '!packages/http-client-python/**' | ||
| - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4 | ||
| id: python-filter | ||
| with: | ||
| filters: | | ||
| python: | ||
| - 'packages/http-client-python/**' | ||
| - '.github/workflows/ci.yml' | ||
| - '.github/workflows/ci-python.yml' | ||
|
|
||
| core: | ||
| needs: changes | ||
| if: needs.changes.outputs.core == 'true' | ||
| uses: ./.github/workflows/core-ci.yml | ||
|
|
||
| python: | ||
| needs: changes | ||
| if: needs.changes.outputs.python == 'true' | ||
| uses: ./.github/workflows/ci-python.yml | ||
|
|
||
| ci-gate: | ||
| name: CI Gate | ||
| runs-on: ubuntu-slim | ||
| if: "!cancelled() || needs.core.result == 'cancelled'" | ||
| needs: [core] | ||
| if: "!cancelled() || contains(needs.*.result, 'cancelled')" | ||
| needs: [core, python] | ||
| steps: | ||
| - name: Validate CI results | ||
| run: | | ||
| if [[ "${{ needs.core.result }}" == "failure" || "${{ needs.core.result }}" == "cancelled" ]]; then | ||
| echo "Core CI failed or was cancelled" | ||
| exit 1 | ||
| fi | ||
| if [[ "${{ needs.python.result }}" == "failure" || "${{ needs.python.result }}" == "cancelled" ]]; then | ||
| echo "Python CI failed or was cancelled" | ||
| exit 1 | ||
| fi | ||
| echo "All CI checks passed or were appropriately skipped" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -55,9 +55,3 @@ extends: | |
| parameters: | ||
| DependsOn: InitStage | ||
| Condition: eq('true', stageDependencies.InitStage.outputs['InitJob.InitStep.RunJava']) | ||
|
|
||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do you need to delete some fileS? |
||
| # Run python stages if RunPython == true | ||
| - template: /packages/http-client-python/eng/pipeline/templates/ci-stages.yml | ||
| parameters: | ||
| DependsOn: InitStage | ||
| Condition: eq('true', stageDependencies.InitStage.outputs['InitJob.InitStep.RunPython']) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is intentional.
upload-artifactroots the artifact at the least common ancestor of the uploaded paths, which ispackages/http-client-pythonhere. The downloaded artifact contains top-leveltests/,dist/, andgenerator/, notpackages/http-client-python/....Downloading into
packages/http-client-pythonrestores those directories to the paths expected by the package scripts. Downloading to.was the previous issue: the package scripts then only saw checked-in placeholders underpackages/http-client-python/tests/generated, while the full generated artifact was restored under repo-roottests/.