GitHub Community
Parallel Steps #14484
Replies: 36 comments 63 replies
-
|
I'd like that too. There are scenarios where it doesn't make sense juggling artifacts between jobs. An idea how it could be used: steps:
- uses: actions/checkout@v3
- run: echo "Sequential step"
- parallel:
branch_id_1:
steps:
- run: echo "branch 1 sequential step 1"
- run: echo "branch 1 sequential step 2"
# Executes steps in parallel to branch1
branch_id_2:
steps:
- run: echo "branch 2 sequential step 1"
- run: echo "branch 2 sequential step 2"
- run: echo "More sequential steps after the whole parallel part is done." |
Beta Was this translation helpful? Give feedback.
-
|
This is a good approach as it mirrors Jenkins well-known solution for parallelizing stages. |
Beta Was this translation helpful? Give feedback.
-
|
Response #1 |
Beta Was this translation helpful? Give feedback.
-
|
just bumping https://github.com/qoomon/actions--parallel-steps https://github.com/orgs/community/discussions/14484#discussioncomment-12893306 |
Beta Was this translation helpful? Give feedback.
-
|
https://github.blog/news-insights/product-news/lets-talk-about-github-actions/
Thanks to @tempacc21 for noticing! Putting here as well for better visibility. |
Beta Was this translation helpful? Give feedback.
-
|
Alternative solution: |
Beta Was this translation helpful? Give feedback.
-
Alternative solution: |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I'm missed how putting steps in a composite action would allow you to parallelize steps such as multiple logins to different registries? Could you please elaborate further with regards to how this solves step parallelization? |
Beta Was this translation helpful? Give feedback.
-
|
Parallel steps or sub jobs would save us 50% of time waiting for the CI. |
Beta Was this translation helpful? Give feedback.
-
|
GH would lose a lot of money if they let people halve the time they charge for. |
Beta Was this translation helpful? Give feedback.
-
|
Logging into docker in parallel is probably going to result in things exploding. Each docker login command tries to mutate Today, you could probably do something like: (To make these work, you'd want to have them rely on env vars and set the env vars using your workflow based on secrets)
HOME=$HOME/.docker-docker
mkdir -p $HOME
docker login ...
HOME=$HOME/.docker-gcr
mkdir -p $HOME
docker login ...
HOME=$HOME/.docker-ghcr
mkdir -p $HOME
docker login ...
for task in ./log-into-*.sh; do
$task &
done
wait
jq '. * input' ~/.docker*/config.js > my.config.js && mv my.config.js ~/.docker/config.js
jq '. * input' ~/.docker*/plaintext-credentials.config.json > my.plaintext.js && mv my.plaintext.js ~/.docker/plaintext-credentials.config.json |
Beta Was this translation helpful? Give feedback.
-
|
If the |
Beta Was this translation helpful? Give feedback.
-
|
Talking to the docker folks is an exercise for the brave. I wouldn't recommend it to my worst enemy. |
Beta Was this translation helpful? Give feedback.
-
|
@jsoref I think instead of for loop with
About docker I totally agree. Talking to docker devs is PITA and nothing that improve dev work is actually done e.g. multiple dockerignore files or mounting volumes as non root to container and so on. |
Beta Was this translation helpful? Give feedback.
-
|
@piotrekkr: thanks on both points. |
Beta Was this translation helpful? Give feedback.
-
|
can we have the parallel steps please |
Beta Was this translation helpful? Give feedback.
-
|
YES, this is the most useful option and it's obvious. Downvote must come from some GitHub dev who does not like to have some work or what? (nah, downvote was about something else, sorry, it's valid) |
Beta Was this translation helpful? Give feedback.
-
Nope, it's coming from me, because this answer doesn't bring any value. Basically a spam. |
Beta Was this translation helpful? Give feedback.
-
|
Because here no-one can request for anything because GitHub action devs don't see it? Okay, I'm not aware where requests can be made. It seems to be anyways useless because there are some many open requests at various pages and I also saw GitHub action devs saying, they will probably not do it |
Beta Was this translation helpful? Give feedback.
-
|
The main post is the by
Unfortunately yes :(
Link please? |
Beta Was this translation helpful? Give feedback.
-
|
It's from memory, so cannot tell, sorry. I think it was either about better caching or about their unwillingness to make "include yml files for steps" or some other of the obvious shortcomings. But then again, not sure if Github even claims to be a CI/CD solution or just code repo with benefits, so maybe we should not be so hard |
Beta Was this translation helpful? Give feedback.
-
|
The BitBucket has parallel step options. Why not add this option to GitHub actions? |
Beta Was this translation helpful? Give feedback.
-
|
This is the biggest pain point we've run into coming from Jenkins. You can approximate this feature with |
Beta Was this translation helpful? Give feedback.
-
|
How come it's error prone? |
Beta Was this translation helpful? Give feedback.
-
|
Every project has different needs about what artifacts they need available, how they're shaped, fs permissions, size costs, etc... |
Beta Was this translation helpful? Give feedback.
-
|
Need parallel steps support for our scenario: deploy multi clusters concurrently within the same VPN environment. So it is convinient to achive in steps rather than actions or jobs |
Beta Was this translation helpful? Give feedback.
-
|
Using a yarn monorepo I'd like to:
yarn install is slow, and I currently have to repeat it for every workflow like |
Beta Was this translation helpful? Give feedback.
-
|
We've been iterating over an action that does some of the work to make this better: https://github.com/GarnerCorp/build-actions/blob/main/yarn/action.yml You can use actions/upload-artifact and actions/download-artifact to share a build product (e.g. the output tarball from the above action) between jobs (especially a matrix of jobs to run tests). |
Beta Was this translation helpful? Give feedback.
-
|
Are you using yarn workspaces? If you are, you should be able to run |
Beta Was this translation helpful? Give feedback.
-
|
I was looking for a solution for exactly the same problem @raphaelbadia. I did get caching to work but I'm using PNPM, which I would highly recommend as an easy time saver compared to Yarn. It uses symlinks to a system level package cache, instead of installing dependencies in all your monorepo projects. |
Beta Was this translation helpful? Give feedback.
-
|
Check out this approach: Making pnpm run jobs in parallel instead of finding a way to run GitHub steps in parallel: https://stackoverflow.com/questions/68427503/how-can-i-run-pnpm-workspace-projects-as-parallel-jobs-on-github-actions |
Beta Was this translation helpful? Give feedback.
-
|
@michielvanderros-tomtom speaking of PNPM :) Yarn supports pnpm linker out of the box, in addition to pnp and regular node_modules; yet it's developed by a team vs. single person behind pnpm, and has great design with plugin system. |
Beta Was this translation helpful? Give feedback.
-
|
If its easier, GitHub could also allow sharing jobs workdirs |
Beta Was this translation helpful? Give feedback.
-
|
You can use |
Beta Was this translation helpful? Give feedback.
-
|
I'm aware of that. |
Beta Was this translation helpful? Give feedback.
-
|
Have a look here: https://circleci.com/docs/caching/ |
Beta Was this translation helpful? Give feedback.
-
It's super slow. Impractical for anything larger. |
Beta Was this translation helpful? Give feedback.
-
|
I'm in the minority, but I'd actually like this way better too— the handling of logs is more natural because each job has its own, and you already have the dag visualization of how the jobs fit together. The obvious syntax would be instead of Main question mark would be what happens to those post-action steps, if you've got a bunch of torn down auth state, that's mildly annoying, though re-authing is not nearly as bad as starting from scratch. This doesn't cover the "multiple preparatory steps in parallel" ask that some others have been making, but I'm honestly not sure if that should be the responsibility of GitHub Actions anyway. Yes, you can do that kind of thing in your Jenkinsfile, but really it belongs in a proper build tool. Let |
Beta Was this translation helpful? Give feedback.
-
|
I just want to start a resource in a background, which needs some time to start, eg a docker container or a simulator. This resource should start during the job start. My test step should wait until the test resource is available, or fail after an (optional) timeout. - jobs:
- build
- resources:
- test-db
run: docker start someSlowImage
- steps:
- run: assemble
- run: test
resources:
- test-db
timeout: 5min |
Beta Was this translation helpful? Give feedback.
-
|
I think you can do this using service containers: https://docs.github.com/en/actions/using-containerized-services/about-service-containers |
Beta Was this translation helpful? Give feedback.
-
|
GitLab CI does a similar thing: https://docs.gitlab.com/ci/services/ |
Beta Was this translation helpful? Give feedback.
-
Just add a
|
Beta Was this translation helpful? Give feedback.
-
|
They might not, but each Google Cloudbuild step runs a separate disposable container and you can't persist state from one to another. Thus anyone making a comparison to them is making a very uniformed comparison. I'm well aware of both. We're in the process of migrating some builds from Cloudbuild to GitHub Workflows and I've been using GitHub Workflows/Actions for years. |
Beta Was this translation helpful? Give feedback.
-
|
@jsoref you can pass state between steps in Cloud Build just as easily as in GitHub Actions — via the shared A common workflow like:
works naturally in Cloud Build because:
steps:
# Checkout source
- name: gcr.io/cloud-builders/git
args: ["clone", "https://github.com/org/repo.git"]
# Setup Node + install deps
- name: node:18
entrypoint: yarn
args: ["install"]
# Build API and App in parallel
- id: build-api
name: node:18
entrypoint: yarn
args: ["workspace", "api", "build"]
- id: build-app
name: node:18
entrypoint: yarn
args: ["workspace", "app", "build"]
# Deploy in parallel, but only after *both* builds succeed
- id: deploy-api
name: node:18
entrypoint: yarn
args: ["workspace", "api", "deploy"]
waitFor: ["build-api", "build-app"]
- id: deploy-app
name: node:18
entrypoint: yarn
args: ["workspace", "app", "deploy"]
waitFor: ["build-api", "build-app"]Here, deployment of From developer feedback (e.g. Reddit, issues, discussions), a recurring criticism of GitHub Actions is not lack of capability per se, but that core concepts like parallelism and step dependencies require significantly more ceremony (jobs, matrices, A simple Docs: https://docs.cloud.google.com/build/docs/configuring-builds/pass-data-between-steps |
Beta Was this translation helpful? Give feedback.
-
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
# Install dependencies
- run: yarn install
# Build the API and web front-end (app)
- id: build-api-app
run: |
yarn workspace api build &
yarn workspace app build
fg %1 || true
# Deploy
- id: deploy-api-app
run: |
yarn workspace api deploy &
yarn workspace app deploy
fg %1 || true |
Beta Was this translation helpful? Give feedback.
-
|
@jsoref In your example, the build is effectively treated as a single step. There are no independent build steps per package/project, each with its own lifecycle, success/failure status, and logs — and that distinction is the core of this issue. What’s missing in GitHub Actions is true step-level parallelism with explicit dependencies. Without it, workflows quickly devolve into large monolithic steps or awkward job graphs with duplicated setup, artifact passing, and extra indirection. I can’t overstate how painful this is (at least for me) when working with multi-package repos. A simple, declarative way to run independent steps in parallel — with clear dependencies and isolated logs — would make a huge difference in both readability and maintainability. 😫 😖 😣 😡 💔 😭 😩 💢 🤬 💥 💢 😤 |
Beta Was this translation helpful? Give feedback.
-
|
You can make that if you want, just create logs for each thing in the main step and then pass the various job outputs to individual steps for reporting. It really isn't terribly hard |
Beta Was this translation helpful? Give feedback.
-
|
As a workaround I was able to use this ` name: Distributed Tasks This way I was able to run our tests parallel |
Beta Was this translation helpful? Give feedback.
-
|
Are you able to see the logs separately for different parallel commands in GitHub Actions? |
Beta Was this translation helpful? Give feedback.
-
|
@koistya : no, can not see logs separately , but if one of the cmd fails the can see logs |
Beta Was this translation helpful? Give feedback.
-
|
That’s an interesting workaround, @malindajudo. Running parallel steps in GitHub Actions can definitely simplify workflows, especially when dealing with tasks like multiple deployments or integrating third-party APIs. For example, in a web app development company, managing Docker containers and parallelizing tasks can help eliminate the complexity of sequential executions. This capability would allow teams to scale their processes faster, reduce bottlenecks, and enhance overall efficiency in their CI/CD pipelines. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Hi. Are there any updates on this topic from a Github team member ? |
Beta Was this translation helpful? Give feedback.
-
|
Don't know much about Actions as I'm new to this (trying to find alternatives to Drone CI), but a simple feature like this took so long time to implement, maybe the design of Actions has some fundamental flaws on create DAG of steps.
Drone steps are by default parallel, unless you add a 'depends_on' list to steps, which would transform them to a DAG. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I was considering Drone CI to use parallel steps, any reasons why you're trying to replace it? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Would love this feature as well. We already have quite long running pipelines and saving some time would be great. We build our .NET backend then pack it as docker image. While creating the image takes significant time we also need to upload our symbols to Sentry for better error analysis. While docker images are being created we could parallelize the upload to Sentry. A new job is no option as we lose the symbol files we require. Right now we run a separate job which fetches git, compiles, etc. all just to upload already built symbols. Please implement this! :) |
Beta Was this translation helpful? Give feedback.
-
|
I have 5 small steps (10-30sec), each runs a docker container with different arguments. Ideally I would have first step to build docker image and then run those five steps in parallel. Doing this with workflows seems cumbersome and would possibly prolong the total run time with artifact upload and artifact downloads added to share the image between jobs. |
Beta Was this translation helpful? Give feedback.
-
|
if you can fit more than one docker container in a single github runner, then it definitely makes more sense to use a single step to create the image and then run the steps in parallel. We use |
Beta Was this translation helpful? Give feedback.
-
correct me if I'm wrong, but in a case with docker compose that would be a single step and you have to dig into the logs to figure out which of the containers failed? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. We did it. It isn't difficult. You could probably use |
Beta Was this translation helpful? Give feedback.
-
|
try https://github.com/qoomon/actions--parallel-steps |
Beta Was this translation helpful? Give feedback.
-
|
Hey y'all! We are still gathering all the high engagement feedback items and triaging them for our work this year. Sorry I haven't commented on this one already. Thank you for all engaging and staying engaged for such a long period of time - please keep voting up on the top request and sharing your feedback with us |
Beta Was this translation helpful? Give feedback.
-
|
💬 Your Product Feedback Has Been Submitted 🎉 Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users. Here's what you can expect moving forward ⏩
Where to look to see what's shipping 👀
What you can do in the meantime 💻
As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities. Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐ |
Beta Was this translation helpful? Give feedback.
-
|
I just released https://github.com/qoomon/actions--parallel-steps This action supports basically all types of steps. |
Beta Was this translation helpful? Give feedback.
-
|
the latest release now supports pre and post steps and command files (except step summary) |
Beta Was this translation helpful? Give feedback.
-
|
With the latest release of act https://github.com/qoomon/actions--parallel-steps is now fully functional. Looking forward for some feedback |
Beta Was this translation helpful? Give feedback.
-
|
@HariSekhon does this work for your initial post/request? |
Beta Was this translation helpful? Give feedback.
-
|
Just adding my voice so maybe by the time my grandkids are coding, GitHub Actions will support parallel steps. |
Beta Was this translation helpful? Give feedback.
-
|
Implementing parallel steps in GitHub Actions would significantly enhance CI/CD workflows, especially for tasks like deploying to multiple environments or managing dependencies across various services. For instance, in a mobile app development company, managing Docker containers and parallelizing tasks can help eliminate the complexity of sequential executions. This approach not only speeds up the development cycle but also ensures more efficient use of resources. Incorporating a feature that allows for parallel execution of steps within a single job, while maintaining access to the same environment and cache, would be a valuable addition. It would streamline processes and reduce the overhead associated with managing multiple jobs. Looking forward to seeing this feature implemented, as it would greatly improve the flexibility and efficiency of workflows in GitHub Actions. |
Beta Was this translation helpful? Give feedback.
-
|
try my action https://github.com/qoomon/actions--parallel-steps |
Beta Was this translation helpful? Give feedback.
-
|
Kinda reminder |
Beta Was this translation helpful? Give feedback.
-
|
https://github.blog/news-insights/product-news/lets-talk-about-github-actions/
|
Beta Was this translation helpful? Give feedback.
-
|
Come on... it's already 2026 and there is still no way to run parallel works sharing the same file cache... |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone 👋 This discussion has received over 2,400 upvotes and 70+ comments, making it one of the most requested features for GitHub Actions. We've reviewed the feedback carefully and want to share a proposal that we believe addresses the core scenarios you've described and get community feedback on this proposal. What We're ProposingWe're exploring three new primitives for running steps concurrently, plus syntactic sugar for the most common pattern:
We plan to release the first three primitives initially and have the The Core PrimitivesNOTE: These are the names we are currently thinking about using, we're certainly open to reconsidering the specific names or syntax used.
|
| Pattern | Example | Solution |
|---|---|---|
| True parallel | Build 3 packages simultaneously | parallel: or background: + wait-all: |
| Background server | Start server, run tests, stop server | background: + cancel: |
| Fire-and-forget | Upload telemetry while build continues | background: (no wait needed) |
A simple parallel block only handles the first pattern. The background/wait/cancel primitives handle all three.
Mirrors Shell Behavior
This design follows how shells work: & to background a process, wait to synchronize. Workflow authors already understand this mental model.
Better Logging Than '&' Workarounds
Many of you are already using & and nohup workarounds:
- run: |
./start-server.sh &
SERVER_PID=$!
npm test
kill $SERVER_PIDThe problem: no log visibility. Output from backgrounded processes is interleaved or lost. In addition, it is difficult to gracefully terminate these long running processes.
With native background: true, each step gets its own log stream in the Actions UI: collapsible, searchable, and tied to the step that produced it. It can also be terminated more gracefully with cancel
Why aren't you just building a parallel block?
We started investigating a parallel block and we realized two things:
- The concept of a background step was more powerful, solving more challenges than just parallelism
- Building
parallelprobably would have used something very similar to a background step as part of the implementation.
We believe the better option is to start with more powerful primitive to give users a way to solve the parallelism problem (and others). At the same time, we recognize the user experience value of a parallel block, so we want to built it on top of these new primitives. We believe this will help us keep with the philosophy of "make the simple things easy and the complex things possible".
Will I be able to have background steps within composite actions?
Initially, no. You would be able to run a composite action as a background step within a workflow, but you would not be able to author composite actions that use background steps. Supporting these cases will need UI/logging work to visualize properly. We wouldn't want to block an initial release which would help a large portion of the community on this, so we'd probably defer it as a next step.
Examples from Your Comments
E2E Testing with Background Server
From @hfhbd: "I just want to start a resource in a background, which needs some time to start"
NOTE: The specific example @hfhbd gave was about starting Docker containers which can currently be done with service containers. However, service containers don't work if you wanted to do something that isn't containerized, requires mid-job startup, or need to run something that isn't compatible with services like a Mac or Windows runner. Background steps would enable those use cases.
steps:
- run: npm install
- name: Start dev server
id: server
run: npm run dev
background: true
# Health check to make sure server is up and ready before running tests
- name: Wait for server ready
run: curl --retry 10 --retry-delay 1 http://localhost:3000/health
- name: Run E2E tests
run: npm run test:e2e
- cancel: server # this Parallel set of sequential steps
From @lukasz-mitka: parallel branches example (100+ reactions)
Handling sets of sequential steps in parallel would be done with composite actions which could be run in parallel
With parallel: block:
#
# Composite action 1
#
name: 'First Set Steps'
description: 'Handle first set of steps'
runs:
using: "composite"
steps:
- run: echo "step 1"
shell: bash
- run: echo "step 2"
shell: bash#
# Composite action 2
#
name: 'Second Set Steps'
description: 'Handle second set of steps'
runs:
using: "composite"
steps:
- run: echo "step A"
shell: bash
- run: echo "step B"
shell: bash#
# Workflow.yml
#
...
steps:
- uses: actions/checkout@v3
- run: echo "Sequential step"
- parallel:
- uses: octo-org/octo-repo/composite-step-1@v1
- uses: octo-org/octo-repo/composite-step-2@v1
- run: echo "More sequential steps after the whole parallel part is done."
Equivalent with background:
#
# Workflow.yml
#
...
steps:
- uses: actions/checkout@v3
- run: echo "Sequential step"
# Parallel steps
- uses: ocot-org/octo-repo/composite-step-1@v1
background: true
- uses: ocot-org/octo-repo/composite-step-2@v1
background: true
- wait-all:
- run: echo "More sequential steps after the whole parallel part is done."Parallel Uploads During Build
From @Skyppid: "While creating the image takes significant time we also need to upload our symbols to Sentry for better error analysis. While docker images are being created we could parallelize the upload to Sentry. A new job is no option as we lose the symbol files we require. Right now we run a separate job which fetches git, compiles, etc. all just to upload already built symbols."
steps:
- run: npm run build
- name: Upload source maps to Sentry
run: sentry-cli sourcemaps upload ./dist
background: true # Fire-and-forget
- run: docker build -t myimage . # Continues immediately
- run: docker push myimageKey Behaviors
| Aspect | Behavior |
|---|---|
| Logging | Each background step gets its own log stream in the UI |
| Timeouts | timeout-minutes works normally on background steps |
| Outputs | $GITHUB_OUTPUT from background steps available after wait |
| Failure | By default, background step failure fails the job (at the implicit wait point) |
| Job end | Implicit wait-all before post hooks run |
| Concurrency | Up to 10 concurrent background steps per job |
Below is a mockup of what the UI might look like for a job with background steps:
We think our initial propsal would resolve most of the use cases highlighted in the discussion but we want your feedback. What concerns do you have? What did we miss or get wrong? Are there scenarios you are concerned about?
The GitHub Actions Team
Beta Was this translation helpful? Give feedback.
-
|
I agree with all the UI comments above. Also, supporting parallel steps in composite actions would be a welcome addition to all this! |
Beta Was this translation helpful? Give feedback.
-
Possibly. The purpose of the limit is safety (we don't want to overwhelm the runners) and we have considered setting an optional user-defined limit. 10 was our best estimate at this point, as we dig deeper into implementation I'll have a better idea of what limits are reasonable. We want to provide the largest one we feel is safe.
I like this, specifically having
I'm going to flip this question a bit: How would you want it to look? I don't have a specific design in mind so if there's something that would work well (or something that works poorly so that we can avoid it), I'd love that feedback.
Will keep this in mind, we are working on some improvements to logging within composite actions which we might be applicable here. More to come there.
My instinct is that we probably wouldn't support it initially, but that is something that could be looked into. I think it falls within the same category as composite actions using background steps since I think the logging of nested parallels might be a little more complicated (especially if someone were to try and nest multiple |
Beta Was this translation helpful? Give feedback.
-
|
I half imagine someone wanting to see columns of output for different parallel streams, but when you get past 2 or 3 that does not work remotely well. ArgoCD (and Docker) stick labels to the left of lines to indicate which container is responsible for a line of output. I think I'd probably prefer this model. (Probably the form where I only see the label when it changes or at the top of the view if it's continuing.) I personally can't imagine how indenting would help anyone, but I'm quite opinionated about most things. (I do absolutely favor name+id, and I think I asked about them earlier.) To anyone wanting more than 10 parallel things, I'd really love to hear a description of what you're imagining putting in those slots (I don't work for GitHub, but I do work on UX and have worked on resource constrained systems). -- People love asking for unlimited things, but generally the UX for them is terrible and system resources disintegrate rapidly when things aren't bounded (browsers have to work to cheat quite a lot to let everyone use lots of tabs without understanding that tabs really should be very expensive). |
Beta Was this translation helpful? Give feedback.
-
We have some repos that build a bunch of things and have to push up 20 different docker images to our ECR registry. It'd be great if we could do these in parallel, but that would be both above the limit of 10 and we would be bandwidth-limited in this case so we probably wouldn't want to do more than 4-5 concurrent pushes at a time. It would be nice if we could specify a concurrency there so we could just list the 20 jobs and have it run through them however fast it can. Right now we have to package these up into a single archive and store them elsewhere and then we start up a matrix step which each downloads the giant archive, unzips, and pushes up the single package, which ends up not being that much faster than doing it serially. |
Beta Was this translation helpful? Give feedback.
-
|
@clayton-sayer For the specific case of building a bunch of docker images at once, I would say define it all in a bakefile and then get the parallelism that way, maybe just call out to depot.dev for your bake since they'll also have way better caching than GHA. But point taken for the more general case where you really do want to do disparate action-level things (like service logins or tool installs) in parallel, on the same worker. |
Beta Was this translation helpful? Give feedback.
-
|
This is an excellent proposal! The approach of introducing A few thoughts on the design:
The examples provided really help illustrate the use cases this would solve. Looking forward to seeing this feature ship before mid-2026! |
Beta Was this translation helpful? Give feedback.
-
|
WRT to composite actions: composite actions generally is an area we are looking at. There's some other highly requested issues we are working on as well and as mentioned in my other comment, some of that work will probably overlap. With error handling, I want to clarify a little bit. Let's use the following example: steps:
- run: echo "Setup complete"
- name: Build frontend
run: npm run build:frontend
background: true
- name: Build backend
run: npm run build:backend
background: true
- name: Build docs
run: npm run build:docs
background: true
continue-on-error: true
- wait-all:
- run: echo "All builds complete"In this we would first kick off the frontend build, then the backend, then the docs, all of which would run in the background until we hit the wait. Let's say the frontend build failed immediately, I would still expect the backend and docs builds to continue and the job to still be in a pending state. Once we hit the |
Beta Was this translation helpful? Give feedback.
-
|
This looks great, I would add a (more niche) use case: having a background step that can wait for another step without blocking the "main thread" of foreground steps. steps:
- name: Starts A in background
id: a
run: ...
background: true
- name: Does not wait for A
id: b
run: ...
background: true
- name: Waits for A but not B
run: |
wait-step a
...
background: true
- name: Wait for A and B but no C
run: |
wait-step a b
... |
Beta Was this translation helpful? Give feedback.
-
|
Do you have a real world example for this? (Obviously the idea exists, but it's much more interesting with real flows) |
Beta Was this translation helpful? Give feedback.
-
|
Yes I create 2 docker images, the 2nd after the first one to reuse most of the cached steps, and have different tests for each one so I get this graph for ideal parallelism: |
Beta Was this translation helpful? Give feedback.
-
|
That feels like: - name: Checkout
uses: actions/checkout@latest
- name: Build A
run: ...
- name: Build B
id: build-b
run: ...
background: true
- name: Test A
run: ...
- name: Test B
wait: build-b
background: true
run: ...
- name: Report
run: ...
wait-all: ...Which I think should technically work based on the described primitives |
Beta Was this translation helpful? Give feedback.
-
|
tl;dr small clarification on how So - name: Build A
id: build-a
run: ...
background: true
- name: Build B
id: build-b
run: ...
background: true
- name: Build C
id: build-c
run: ...
background: true
- wait: [build-a, build-b, build-c]
- name: Test
run:...We start Build A, then Build B, then Build C. Once we get to the I think there are multiple ways to handle the scenario described with background steps, probably depending on what you want to do. This is how you would do it with a - name: Build A
run: ...
- parallel
- name: Test A
run: ...
- uses: myorg/b-action@v1 # this is a composite action that does Build B, then Test B
- name: Report
run: ...Alternatively, if you wanted to just use background steps, you could do something like this (@jsoref's example would have had Test B wait until Test A finished, regardless of whether Build B had finished, which I don't think is the desired behavior): - name: Build A
run: ...
id: build-a
- name: Test A
run: ...
id: test-a
background: true
- name: Build B
id: build-b
run: ...
- name: Test B
run: ...
- wait-all: # Could also be wait: test-a, since we know test-b would have to be finished before we got here
- name: Report
run: ...I think that if you are trying to run multiple parallel work streams which may eventually converge, using - name: Build A
run: ...
id: build-a
- name: Build B
id: build-b
run: ...
background: true
- name: Test A
run: ...
id: test-a
background: true
- wait: build-b
- name: Test B
run: ...
- wait-all: # Could also be wait: test-a, since we know test-b would have to be finished before we got here
- name: Report
run: ...I think |
Beta Was this translation helpful? Give feedback.
-
|
@salilsub Thank you for your guidance. I prepared the main workflow in my repository to use the new features (Adding |
Beta Was this translation helpful? Give feedback.
-
|
What I really miss in the parallel step design is to have the matrix strategy on step level like this (I create sometimes ca. 30 matrix jobs that run only < 30s) - run: echo Process the following in parallel / generate the matrix in this step dynamically
- strategy:
max-parallel: 2
matrix:
artifacts:
- name: pkg1
path: pkg1/out
- name: pkg2
path: pkg2/out
steps:
# Here you could also add the build command to run sequentially before the upload artifact
# upload-artifact usually only uploads all files into a single artifacts this is bad if you want to download only a single package
- uses: actions/upload-artifacts@v6
with:
name: ${{ stepMatrix.artifacts.name }}
path: ${{ stepMatrix.artifacts.path }}Yes I can do this example with javascript and the @actions/artifact package, but then I need much more code (On the other hand the runner needs more code for this) In my opinion Parallel Syntactic Sugar weakness is that the count of parallel steps is constant, with a matrix strategy you can adjust the called actions count and parallelism depending on the reusable workflow input parameters. Just like in job.strategy. |
Beta Was this translation helpful? Give feedback.
-
|
+1 |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
Can you please raise a request to engineering for parallel steps support?
Eg. for logging in to multiple docker registries in parallel, or other actions which cannot be done in separate jobs because you require the local cache / side effects.
Uploading/downloading artifacts from adjacent jobs is not suitable for this use case.
Thanks
Hari
Beta Was this translation helpful? Give feedback.
All reactions