Concurrency
Cloud Server 3 Server 2 This page summarizes concurrency, and suggests ways to use concurrency to your advantage.
In CircleCI, concurrency refers to utilizing multiple containers to run multiple jobs at the same time. This is different from CircleCI parallelism, which is test-splitting across multiple containers. If you would like information on parallelism, visit the Running Tests in Parallel page.
Every plan with CircleCI includes the ability to run jobs concurrently. However, the number of jobs you can run concurrently differs between plans. Please refer to the pricing page for up-to-date information on the number of concurrent job runs included in your plan.
Concurrency
Within CircleCI, concurrent jobs can be classified as one of two situations:
-
Multiple running jobs configured in a single workflow
-
Multiple workflows running by multiple members within the organization
This page discusses the first point, and will provide examples of how to configure concurrent jobs in a single workflow. Configuring concurrency in a workflow allows multiple jobs to be run at the same time, in a single workflow, through the use of multiple containers.
To keep the system stable for all CircleCI customers, we implement different soft concurrency limits on each of the resource classes for different executors.
If jobs are queued, it is possible you are hitting these limits. Hitting your limit may depend on a few factors, such as how many people in your organization are running jobs at once, or if you have overlapping workflows.
Customers on a Performance or Scale plan can request an increase to those limits at no extra charge by contacting CircleCI support.
Concurrency in workflows
To run a set of concurrent jobs, you will need to add a workflows section to your existing .circleci/config.yml file.
The simple example below shows the default workflow orchestration with two concurrent jobs. The workflows key needs to have a unique name. In this example, the unique name is build_and_test. The jobs key is nested under the uniquely named workflow, and contains the list of job names. Since the jobs have no dependencies, they will run concurrently.
version: 2.1
jobs:
build:
docker:
- image: cimg/<language>:<version TAG>
auth:
username: mydockerhub-user
password: $DOCKERHUB_PASSWORD # context / project UI env-var reference
steps:
- checkout
- run: <command>
test:
docker:
- image: cimg/<language>:<version TAG>
auth:
username: mydockerhub-user
password: $DOCKERHUB_PASSWORD # context / project UI env-var reference
steps:
- checkout
- run: <command>
workflows:
build_and_test:
jobs:
- build
- testPlease note, if you are still using Server 2 and version: 2, your workflows section will need to specify the version again, as in the below snippet:
...
workflows:
version: 2
build_and_test:
jobs:
- build
- test
...Fan-out/fan-in workflow example
A more complex example of using concurrent workflows is running a common build job, fanning-out to run a set of acceptance test jobs concurrently, and then fanning-in to run a common deploy job. This flow is illustrated below.
The .circleci/config.yml snippet below is an example of a workflows section configured for fan-out/fan-in job execution. In this example, as soon as the build job finishes successfully, all four acceptance test jobs start. The deploy job must wait for all four acceptance test jobs to complete successfully before it starts.
...
workflows:
build_accept_deploy:
jobs:
- build
- acceptance_test_1:
requires:
- build
- acceptance_test_2:
requires:
- build
- acceptance_test_3:
requires:
- build
- acceptance_test_4:
requires:
- build
- deploy:
requires:
- acceptance_test_1
- acceptance_test_2
- acceptance_test_3
- acceptance_test_4
...Please note, if you are still using CircleCI Server 2 with version: 2, you will need to specify the version in the workflows section, as mentioned in the Concurrency in workflows section.
See also
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guide first).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Need support?
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
