Advanced pipeline orchestration with the circleback pattern
Developer Advocate
Learn how to manage complex development projects by using the circleback pattern to trigger pipelines from other pipelines. An alternative approach would be to keep the first pipeline running, checking periodically whether the second pipeline has finished. The main drawback of this approach is the increased cost for the running job’s continuous polling.
Depending on the status of the second pipeline, the first continues executing until it either terminates successfully or fails.
Why have pipelines wait for other projects to complete?
As mentioned in the introduction, this is an advanced technique aimed at taming the complexity that stems from working with multiple projects. It can be used by teams working on multiple interdependent projects that they do not want to put in a single repository like a monorepo. Another use would be to have a centralized repository for tests, for example in a hardware company. This technique could also be useful for integration testing of a microservices application, or for orchestrating complex deployment scenarios. There are many possibilities.
When to use this pattern
The circleback pattern is powerful but complex, so it’s worth considering whether your use case truly requires it. This pattern shines when you have genuine cross-project dependencies where Project B must complete successfully before Project A can continue, especially when you need detailed status information beyond a simple pass/fail. It’s particularly valuable for long-running validations that would be expensive to monitor through continuous polling, such as security scans or performance benchmarks that take 30 minutes or more.
However, CircleCI offers several simpler alternatives that may better suit your needs:
- Dynamic configuration with path filtering for monorepo setups
- Custom webhooks for event-driven triggers from external systems
- Serial groups to control job execution order across your organization
- Setup workflows for computing parameters before continuing a pipeline
If you’re just triggering another pipeline without needing to wait for completion, a simple webhook or API call will suffice. For validation checks that complete in under 5 minutes, basic polling might be acceptable despite the cost. The circleback pattern remains the best solution when you need true bidirectional communication between projects, but evaluate these simpler alternatives first. The added complexity is only justified when you genuinely need to orchestrate multiple interdependent projects that live in separate repositories and require detailed status reporting between them.
Prerequisites
Important: This pattern requires GitHub OAuth integration, not GitHub App integration. If your CircleCI projects are connected via GitHub App, you’ll need to switch to OAuth for API-triggered pipelines to work. The GitHub App integration does not support triggering pipelines via the CircleCI API, which is essential for this pattern.
To check your integration type:
- Go to your CircleCI project settings
- If you see project slugs starting with
circleci/followed by UUIDs, you’re using GitHub App - If you see project slugs like
gh/username/project, you’re using GitHub OAuth
To enable GitHub OAuth:
- Go to CircleCI Organization Settings → VCS
- Enable GitHub OAuth alongside your GitHub App integration
- Re-authorize your projects with OAuth
Implementing pipeline triggers
We have 2 pipelines we want to orchestrate
- Pipeline A, which does the triggering
- Pipeline B is triggered, and circles back to pipeline A
Pipeline B is dependent on A, and can be used to validate A.
Both pipelines need to have API keys set up and available. You can use the API key set as an environment variable (CIRCLECI_API_KEY) in the job in pipeline A, and also in pipeline B when it calls back. You can either set it in both projects, or at the organization level as a context. For this tutorial, I set it at the organization level as the circleci_api context, so that both projects can use the same API key.
Important Setup: The CIRCLECI_API_KEY must be properly configured:
- Option 1 (Recommended): Set as project environment variables in both Project A and Project B
- Option 2: Create an organization context called
circleci_apiwith theCIRCLECI_API_KEYvariable
Without the API key properly configured, Pipeline B will not be able to approve Pipeline A’s waiting job.
Note: As a security best practice, rotate your API tokens regularly and limit their scope when possible. Use CircleCI’s context restrictions to control which teams can access sensitive tokens.
Trigger the pipeline
The triggering process is explained in depth in the first part of this tutorial, Triggering pipelines from other pipelines. In this follow-up tutorial, I will cover just the important differences.
- To circle back from the pipeline, pass the original pipeline’s ID to it. Then it can be retrieved and reached with the API.
- You also need to store the triggered pipeline’s ID. You will need to get its result later on.
In the sample code, the parameter is called triggering-pipeline-id:
curl --request POST \
--url https://circleci.com/api/v2/project/gh/your-org/your-project-b/pipeline \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
--data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}'
Note: As of October 2024, CircleCI recommends using the new pipeline trigger endpoint format: /api/v2/project/{provider}/{organization}/{project}/pipeline/run. The endpoint shown above remains functional for backward compatibility.
To store the pipeline ID, wrap your curl call in $() and assign it to the variable CREATED_PIPELINE. To extract the ID from the response body, use the jq tool, and write it to the file pipeline.txt:
CREATED_PIPELINE=$(curl --request POST \
--url https://circleci.com/api/v2/project/gh/your-org/your-project-b/pipeline \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
--data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
| jq -r '.id'
)
echo "my created pipeline"
echo $CREATED_PIPELINE
mkdir -p ~/workspace
echo $CREATED_PIPELINE > ~/workspace/pipeline.txt
Now that you have the file pipeline.txt created, use persist_to_workspace to store it and use it in a subsequent job. Note that workspace storage defaults to 15 days retention, but this can be customized in your CircleCI Plan Usage Controls:
- persist_to_workspace:
root: ~/workspace
paths:
- pipeline.txt
The whole job configuration is here:
...
jobs:
trigger-project-b-pipeline:
docker:
- image: cimg/base:stable
resource_class: small
steps:
- run:
name: Ping another pipeline
command: |
CREATED_PIPELINE=$(curl --request POST \
--url https://circleci.com/api/v2/project/gh/your-org/your-project-b/pipeline \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
--data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
| jq -r '.id'
)
echo "my created pipeline"
echo $CREATED_PIPELINE
mkdir -p ~/workspace
echo $CREATED_PIPELINE > ~/workspace/pipeline.txt
- persist_to_workspace:
root: ~/workspace
paths:
- pipeline.txt
...
Orchestrating the waits
The previous job will trigger a pipeline B, which needs to complete before it can circle back to pipeline A. You can use the approval job in CircleCI like this:
...
workflows:
node-test-and-deploy:
jobs:
...
- trigger-project-b-pipeline:
context:
- circleci-api
requires:
- build-and-test
filters:
branches:
only: main
- wait-for-triggered-pipeline:
type: approval
requires:
- trigger-project-b-pipeline
- check-status-of-triggered-pipeline:
requires:
- wait-for-triggered-pipeline
context:
- circleci-api
...
After the job trigger-project-b-pipeline, enter the wait-for-triggered-pipeline. Because that job type is approval it will wait until someone (in this case, the API) manually approves it. (More details in the next section.) After it is approved, add a requires stanza so it continues to a subsequent job.
Both jobs that use the CircleCI API have the context specified, so the API token is available to both as an environment variable.
Circle back to pipeline A
For now we are done with pipeline A, and it is pipeline B’s time to shine. CircleCI’s approval job is a special kind of job that waits until accepted. It is commonly used to hold a pipeline in a pending state until it is approved by a human delivery lead or infosec engineer.
At this point, pipeline B knows the ID of pipeline A, so you can use the approval API to get it. You only have the ID of the pipeline being run, but not the actual job that needs approval, so you will need more than one API call:
- Get all jobs in the pipeline
- Find the approval job by name
- Send a request to approve the job
Approving the job allows pipeline A to continue.
If the tests fail in pipeline B, then that job automatically fails. A workflow with required jobs will not continue in this case. You can get around by using post-steps in the pipeline, which always executes. The whole workflow is shown in the next sample code block.
Parameters:
parameters:
triggering-pipeline-id:
type: string
default: ""
...
workflows:
node-test-and-deploy:
jobs:
- build-and-test:
post-steps:
- approve-job-in-triggering-pipeline
context:
- circleci-api
Script to perform the approval API call can be implemented like this. For this tutorial, I used a command.
...
commands:
approve-job-in-triggering-pipeline:
steps:
- run:
name: Ping CircleCI API and approve the pending job
command: |
echo << pipeline.parameters.triggering-pipeline-id >>
if ! [ -z "<< pipeline.parameters.triggering-pipeline-id >>" ]
then
workflow_id=$(curl --request GET \
--url https://circleci.com/api/v2/pipeline/<< pipeline.parameters.triggering-pipeline-id >>/workflow \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
| jq -r '.items[0].id')
echo $workflow_id
waiting_job_id=$(curl --request GET \
--url https://circleci.com/api/v2/workflow/$workflow_id/job \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
| jq -r '.items[] | select(.name == "wait-for-triggered-pipeline").id')
echo $waiting_job_id
curl --request POST \
--url https://circleci.com/api/v2/workflow/$workflow_id/approve/$waiting_job_id \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json"
fi
when: always
...
The script first checks for the existence of the triggering-pipeline-id pipeline parameter. It proceeds only if that parameter exists. The when: always line in the command makes sure that this executes regardless of termination status.
Then it makes 3 API calls:
- Get workflow ID in a pipeline. There is just one workflow in that pipeline for this sample project.
- Get jobs in that workflow, use
jqto select the one that matches the approval job name (wait-for-triggered-pipeline), and extract the approval job’s ID. - Make the request to the approval endpoint with the waiting job ID.
For this tutorial, we are storing results like workflow ID and job ID in local bash variables, and using them in subsequent calls to the API.
Note: If you have more jobs than can be sent in a single response, you might have to handle pagination as well.
Now that you have made the approval request, pipeline B is complete, and pipeline A should be running again.
Update pipeline A with the result of B
After pipeline A has been approved, the next job in the workflow will begin. If your workflow graph requires it, pipeline A can trigger multiple jobs.
We still do not have any indication of the result of the previous workflow. To get that information, you can use the API again to get B’s status from pipeline A. An example job could look like that: check-status-of-triggered-pipeline.
First, you need to retrieve the ID of the triggered pipeline, which is pipeline B. This is the same ID that was persisted in a workspace in an earlier step. Retrieve it using cat:
- attach_workspace:
at: ~/workspace
- run:
name: Check triggered workflow status
command: |
triggered_pipeline_id=$(cat ~/workspace/pipeline.txt)
Then use the API to retrieve the workflow. Use jq to get just the status of the first item in the returned array of workflows:
created_workflow_status=$(curl --request GET \
--url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
| jq -r '.items[0].status'
)
Check that status is not success. If it is not, use exit to terminate the job with exit code -1. If the workflow is successful, it will terminate:
if [[ "$created_workflow_status" != "success" ]]; then
echo "Workflow not successful - ${created_workflow_status}"
(exit -1)
fi
echo "Created workflow successful"
Important notes
- Replace
gh/your-org/your-project-bwith your actual GitHub organization and project name - Ensure
CIRCLECI_API_KEYis set either as:- Project environment variable in both projects, OR
- In the
circleci-apicontext at the organization level
- The API key must have full permissions (not read-only)
- Projects must use GitHub OAuth, not GitHub App integration
Verifying success
We can see that both pipelines were successful by looking in the CircleCI interface:
This screenshot demonstrates the successful triggering and completion of both pipelines using the circleback pattern, with Pipeline A initiating Pipeline B and waiting for its completion before finishing.
Conclusion
In this article we have reviewed an example of a complex pipeline orchestration pattern that I have named “circleback”. The circleback pattern creates a dependent pipeline and allows you to wait for it to terminate before completing. It involves making several API calls from both projects, the use of an approval job, and the workspace feature of CircleCI to store and pass values such as pipeline ID across jobs in a workflow.
Key requirements for this pattern:
- GitHub OAuth integration (not GitHub App)
- Properly configured API keys in both projects
- Correct workspace paths and persistence
The sample projects are located in separate repositories: project A, and project B.
