Troubleshooting CircleCI webhooks
Fullstack Developer and Tech Author
CircleCI webhooks open up a variety of exciting use cases, from data logging and integrations with third-party monitoring and observability solutions to setting up your own custom dashboards to monitor pipeline health. To ensure that you can properly monitor events, resolve authentication errors, and also access the information contained within them, you need a reliable process to debug any errors you might encounter.
In this tutorial, I will show you how to set up a webhook on CircleCI as well as techniques for troubleshooting common errors with webhook connections so that you can be confident about the work you and your team are doing.
Prerequisites
To begin, there are a few things you need to have or set up, including:
- A CircleCI account and associated project
- Node.js 26 LTS installed on your system to run the sample project (Node.js 22 or later also works)
- A publicly accessible URL to the API endpoint
- A text editor for editing code
With this setup, you have an environment in which you can conveniently troubleshoot your CircleCI webhooks.
Cloning and running a demo API
To begin, you will be cloning a sample Node.js API. This API receives and logs a subset of your CircleCI webhook information in an in-memory database. I have intentionally introduced an authentication error and a not found error into this application. These are the types of errors you will be debugging and fixing as you follow along with the tutorial.
Clone the project repository by running this command:
git clone --single-branch --branch base-project https://github.com/CIRCLECI-GWP/circleci-webhooks-api
Navigate to the root of the project and install the required dependencies by running these commands:
cd circleci-webhooks-api
npm install
When the installation is complete, run the Node.js server with this command:
npm start
This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.
On the first request, the API downloads a MongoDB binary for its in-memory database, which can take a minute. Open http://localhost:1337 in a browser once to start that download before continuing. The page shows Welcome to the Webhooks API when the database is ready.
We are using two endpoints in this project:
/log-circleci-webhookis the endpoint that will be receiving the CircleCI webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.- The
/fetch-webhooks-logsendpoint can be called to retrieve a collection of the logged webhook data.
Troubleshooting CircleCI webhooks with Hookdeck CLI
Troubleshooting webhooks locally requires that you have a publicly accessible URL. However, locally running APIs do not have publicly accessible endpoints. You need a way to make your local endpoints publicly accessible and that can be done by using an HTTP request tunneling system.
An HTTP request tunneling system helps tunnel your webhooks into your local development environment to target an endpoint on your local API.
There are many open-source tools that can help you achieve this. The one that I have found most convenient is the Hookdeck CLI. Hookdeck CLI not only provides you with a publicly accessible URL that points to your local API endpoint, but it also comes with reporting, an event page to view your headers and payload, and other tools that make debugging webhooks less frustrating.
You can run this command to install the CLI tool for macOS:
brew install hookdeck
If you are using the Windows operating system, use this command to install the CLI tool:
scoop bucket add hookdeck https://github.com/hookdeck/scoop-hookdeck-cli.git
scoop install hookdeck
On Linux, or on any system with Node.js, install the CLI with npm:
npm install hookdeck-cli -g
A Linux binary is also available from the Hookdeck CLI releases page. Find more installation options in the Hookdeck CLI documentation.
Getting a CircleCI webhook URL
The next step is to use the CLI to generate a webhook URL that points to the running API application. Run this command:
hookdeck listen 1337 CircleCI --path /log-circleci-webhook
This command forwards webhooks to port 1337 on the local machine. CircleCI is the name of the Hookdeck source that receives the webhooks, and --path sets the endpoint they’re forwarded to.
Without a Hookdeck login, the CLI creates a temporary guest account. Because the CircleCI source doesn’t exist yet, the CLI asks whether to create it. Type y and press Enter.
The CLI creates the source, prints its webhook URL, and connects. When it shows Connected. Waiting for events..., it’s ready to receive requests.
$ hookdeck listen 1337 CircleCI --path /log-circleci-webhook
🚩 You are using the CLI for the first time without a permanent account. Creating a guest account...
Source "CircleCI" not found.
? Do you want to create a new source named "CircleCI"? Yes
A source represents where requests originate from (ie. Github, Stripe, Shopify, etc.). Each source has it's own unique URL that you can use to send requests to.
There's no CLI destination connected to CircleCI, creating one named cli-CircleCI
●── HOOKDECK CLI ──●
Listening on 1 source • 1 connection • [i] Collapse
CircleCI
│ Requests to → https://console.hookdeck.com/e/6a2tpz9bsvghfl
└─ Forwards to → http://localhost:1337/log-circleci-webhook (cli-CircleCI)
💡 Sign up to make your webhook URL permanent: https://api.hookdeck.com/signin/guest?token=<your-guest-token>
Events • [↑↓] Navigate ──────────────────────────────────────────────
● Connected. Waiting for events...
The Requests to URL is the webhook URL. Keep this terminal open: the CLI forwards webhooks only while it’s running.
Note: To open the Hookdeck dashboard as a guest, use the sign-up link that the CLI prints. Copy and paste it into a browser to begin a guest session.
Setting up the CircleCI webhook
With your webhook URL, you can now set up a webhook on CircleCI. Go to any of your CircleCI projects and navigate to Project Settings > Webhooks. On the Webhooks page, click Add Webhook.
Fill out the fields on the form like this:
- Webhook name: Enter a descriptive name for your webhook. Use something simple like
Log API Webhook. - Receiver URL: Paste the webhook URL from the Hookdeck CLI output here.
- Secret token: This is a security feature that allows you to verify your webhook source (more on this later). Enter the value
ABCD123here. - Certificate verification: When Verify TLS certificate is checked, CircleCI checks that the receiver URL has a valid TLS certificate before it sends an event. This box is checked by default. Leave it checked.
- Events: Check the Workflow Completed event.
Click Add Webhook to create your webhook. It will be displayed in the list of available webhooks.
Next, confirm that you can now receive webhooks from your CircleCI project. To run the project build either:
- Make a new commit
- Click Rerun workflow from start
Once the build is complete, go to the terminal where your Hookdeck CLI session is running. There are two new entries. The first one arrived as soon as the webhook was created: CircleCI sends a short test event that contains only the webhook’s name. The second one is the Workflow Completed event from the build. That’s the event to troubleshoot in the rest of this tutorial.
The entry on the CLI confirms that you are successfully receiving the webhook. But there is a 500 server error.
Each webhook entry on the CLI shows these elements in order:
- Date and time the event arrived
- HTTP status code (
500) - Request method (
POST) - Local URL the CLI forwarded the webhook to (
http://localhost:1337/log-circleci-webhook), followed by the response time - Link to the event in the Hookdeck Console, where you can view details about your webhooks
CircleCI retries webhook deliveries that don’t get a successful (2xx) response, so the same event may arrive more than once during troubleshooting.
Go on to the next section to fix the 500 error.
Troubleshooting the authentication error
Before making any assumptions, inspect the event in the Hookdeck Console. If the guest session isn’t open yet, open the guest sign-up link from the CLI output in a browser. The Console lists the events that the source received. Click the newer entry, the Workflow Completed event, to open its details.
The Request tab shows all the details about the webhook you just received. Take a minute to review the Headers section. CircleCI sends the event type in the circleci-event-type header and the payload signature in the circleci-signature header.
Review the webhook payload in the Body section.
Click the Response tab to review the server’s response.
The response message reveals that we are failing the security check on the webhook payload. Because we defined an API secret, CircleCI sends the circleci-signature header. This header contains a signature: an HMAC-SHA256 hash of the payload, computed with the secret key. The API must compute the same hash from the payload it received and check that the two match.
This is a security check to prevent attackers from replacing the actual payload with a malicious one that could corrupt your API.
We know that we are receiving the actual payload from CircleCI because we set everything up and triggered the webhook. We know it is not an attacker’s payload causing the authentication failure. Something else must be wrong with our set up.
Our next step is to review the validation logic on our API. This can be found in the server.js file.
//Validate payload
function validatePayload(req, res, next) {
if (req.method == "POST") {
if (!req.rawBody) {
return next("Request body empty");
}
// The header is a comma-separated list of versioned signatures (v1=...,v2=...). Check v1.
const signatures = (req.get(sigHeaderName) || "").split(",");
const sig = Buffer.from(
signatures.find((s) => s.startsWith("v1=")) || "",
"utf8"
);
const hmac = crypto.createHmac(sigHashAlg, secret);
const digest = Buffer.from(
"v1=" + hmac.update(req.rawBody).digest("hex"),
"utf8"
);
if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
return next(
`Request body digest (${digest}) did not match ${sigHeaderName} (${sig})`
);
}
}
return next();
}
app.use(validatePayload);
Three variables being used here in the code are worth checking out to confirm that they are referencing the correct values. These are:
sigHeaderName, which represents the signature header name sent by CircleCI.sigHashAlg, which is the hash algorithm used for the signature.secret, which is the API secret set on our CircleCI webhook.
These values are set on lines 10-12 in the server.js file. Here is the code for these definitions:
const sigHeaderName = "circleci-signature";
const sigHashAlg = "sha256";
const secret = "XXX-XXX";
Did you catch the error? If the secret is not the same as the one we set in our webhook form, the validation will always fail.
CircleCI does not allow you to view the API secret you set in your webhooks a second time; you can only reset it. This is why you need to make sure that you remember the value or store it somewhere secure.
For this tutorial, we know we set the secret as a simple ABCD123 string. In real-world applications, you will want to set a more complicated secret and reference it from an environment variable in your code.
Change the value of the secret to the correct value, save the file, and restart the server.
To test the fix, you need to trigger a new webhook. You can trigger another build on your CircleCI project. If you do not want to wait for a new build, you can use the Retry button to rerun the webhook. The Retry button is next to the status on the Response tab.
After you click Retry, the event’s status changes to 404 in the event list and on the Response tab.
There will also be a new webhook entry on the CLI.
Good work! We have cleared the 500 error, but now we have a new error. This 404 not found error indicates that our specified endpoint (/log-circleci-webhook) cannot be found.
Troubleshooting the “not found” error
Our second webhook attempt resulted in a 404 error on the destination endpoint. This tells us that even though we are successfully receiving our webhooks, the endpoint is not at the specified location, or it does not exist. 404 errors can be fixed, most of the time, by checking for typos or misspellings in the specified route name. Worst-case scenario, the route truly does not exist and needs to be created.
Review the server response for this error on the Response tab.
This shows the actual response from the running server, and it confirms that the specified endpoint cannot be found.
The endpoint specified for the webhook to hit is /log-circleci-webhook, which can be found in the routes.js file at the root of the project directory.
router.post("/log-circleci-hook", async function (req, res) {
//console.log(req.body);
const payload = req.body;
let webhook_info = {
name: payload.webhook.name,
project: payload.project.name,
workflow: payload.workflow.name,
status: payload.workflow.status
};
const save_webhook = await req.db
.collection("webhooks")
.insertOne(webhook_info);
res.status(201).send({
message: "Webhook Event successfully logged"
});
});
Closer review of the route handler reveals a typo. On line 10 webhook is incorrectly spelled as hook. Correct this misspelling and save the file.
You will need to restart the Node.js server to allow the changes to take effect. Shut down the server using Ctrl + C and restart it with the npm start command.
Click the Retry button again. The successful webhook is indicated by the 201 status code.
There will also be a webhook entry on your Hookdeck CLI session.
We have a successful operation on our API.
The Response tab shows the server response: a success message.
With a successful operation on our API, our webhook should be logged. Visit the endpoint /fetch-webhooks-logs to review the collection.
Conclusion
Just like any other architectural component of an application, webhooks can run into errors. Debugging is a part of the workflow when developing applications or setting up application infrastructure. Errors are inevitable, so knowing what to look out for when a certain error occurs and having the right tools to debug them makes troubleshooting faster and much easier.
In this article, we have demonstrated how to debug CircleCI webhooks in a way that gives clear insight into the errors taking place.
Happy coding!