TutorialsLast Updated Nov 16, 202611 min read

Troubleshooting CircleCI webhooks

Fikayo Adepoju

Fullstack Developer and Tech Author

CircleCI webhooks open up a variety of exciting use cases, from data logging and integrations with third-party monitoring and observability solutions to setting up your own custom dashboards to monitor pipeline health. To ensure that you can properly monitor events, resolve authentication errors, and also access the information contained within them, you need a reliable process to debug any errors you might encounter.

In this tutorial, I will show you how to set up a webhook on CircleCI as well as techniques for troubleshooting common errors with webhook connections so that you can be confident about the work you and your team are doing.

Prerequisites

To begin, there are a few things you need to have or set up, including:

  • A CircleCI account and associated project
  • Node.js 26 LTS installed on your system to run the sample project (Node.js 22 or later also works)
  • A publicly accessible URL to the API endpoint
  • A text editor for editing code

With this setup, you have an environment in which you can conveniently troubleshoot your CircleCI webhooks.

Cloning and running a demo API

To begin, you will be cloning a sample Node.js API. This API receives and logs a subset of your CircleCI webhook information in an in-memory database. I have intentionally introduced an authentication error and a not found error into this application. These are the types of errors you will be debugging and fixing as you follow along with the tutorial.

Clone the project repository by running this command:

git clone --single-branch --branch base-project https://github.com/CIRCLECI-GWP/circleci-webhooks-api

Navigate to the root of the project and install the required dependencies by running these commands:

cd circleci-webhooks-api
npm install

When the installation is complete, run the Node.js server with this command:

npm start

This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.

On the first request, the API downloads a MongoDB binary for its in-memory database, which can take a minute. Open http://localhost:1337 in a browser once to start that download before continuing. The page shows Welcome to the Webhooks API when the database is ready.

We are using two endpoints in this project:

  • /log-circleci-webhook is the endpoint that will be receiving the CircleCI webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.
  • The /fetch-webhooks-logs endpoint can be called to retrieve a collection of the logged webhook data.

Troubleshooting CircleCI webhooks with Hookdeck CLI

Troubleshooting webhooks locally requires that you have a publicly accessible URL. However, locally running APIs do not have publicly accessible endpoints. You need a way to make your local endpoints publicly accessible and that can be done by using an HTTP request tunneling system.

An HTTP request tunneling system helps tunnel your webhooks into your local development environment to target an endpoint on your local API.

There are many open-source tools that can help you achieve this. The one that I have found most convenient is the Hookdeck CLI. Hookdeck CLI not only provides you with a publicly accessible URL that points to your local API endpoint, but it also comes with reporting, an event page to view your headers and payload, and other tools that make debugging webhooks less frustrating.

You can run this command to install the CLI tool for macOS:

brew install hookdeck

If you are using the Windows operating system, use this command to install the CLI tool:

scoop bucket add hookdeck https://github.com/hookdeck/scoop-hookdeck-cli.git
scoop install hookdeck

On Linux, or on any system with Node.js, install the CLI with npm:

npm install hookdeck-cli -g

A Linux binary is also available from the Hookdeck CLI releases page. Find more installation options in the Hookdeck CLI documentation.

Getting a CircleCI webhook URL

The next step is to use the CLI to generate a webhook URL that points to the running API application. Run this command:

hookdeck listen 1337 CircleCI --path /log-circleci-webhook

This command forwards webhooks to port 1337 on the local machine. CircleCI is the name of the Hookdeck source that receives the webhooks, and --path sets the endpoint they’re forwarded to.

Without a Hookdeck login, the CLI creates a temporary guest account. Because the CircleCI source doesn’t exist yet, the CLI asks whether to create it. Type y and press Enter.

The CLI creates the source, prints its webhook URL, and connects. When it shows Connected. Waiting for events..., it’s ready to receive requests.

$ hookdeck listen 1337 CircleCI --path /log-circleci-webhook
🚩 You are using the CLI for the first time without a permanent account. Creating a guest account...
Source "CircleCI" not found.
? Do you want to create a new source named "CircleCI"? Yes
A source represents where requests originate from (ie. Github, Stripe, Shopify, etc.). Each source has it's own unique URL that you can use to send requests to.
There's no CLI destination connected to CircleCI, creating one named cli-CircleCI

●── HOOKDECK CLI ──●

Listening on 1 source • 1 connection • [i] Collapse

CircleCI
│  Requests to → https://console.hookdeck.com/e/6a2tpz9bsvghfl
└─ Forwards to → http://localhost:1337/log-circleci-webhook (cli-CircleCI)

💡 Sign up to make your webhook URL permanent: https://api.hookdeck.com/signin/guest?token=<your-guest-token>

Events • [↑↓] Navigate ──────────────────────────────────────────────

● Connected. Waiting for events...

The Requests to URL is the webhook URL. Keep this terminal open: the CLI forwards webhooks only while it’s running.

Note: To open the Hookdeck dashboard as a guest, use the sign-up link that the CLI prints. Copy and paste it into a browser to begin a guest session.

Setting up the CircleCI webhook

With your webhook URL, you can now set up a webhook on CircleCI. Go to any of your CircleCI projects and navigate to Project Settings > Webhooks. On the Webhooks page, click Add Webhook.

Fill out the fields on the form like this:

  • Webhook name: Enter a descriptive name for your webhook. Use something simple like Log API Webhook.
  • Receiver URL: Paste the webhook URL from the Hookdeck CLI output here.
  • Secret token: This is a security feature that allows you to verify your webhook source (more on this later). Enter the value ABCD123 here.
  • Certificate verification: When Verify TLS certificate is checked, CircleCI checks that the receiver URL has a valid TLS certificate before it sends an event. This box is checked by default. Leave it checked.
  • Events: Check the Workflow Completed event.

Add Webhook form in CircleCI project settings with the webhook name, Hookdeck receiver URL, secret token, and Workflow Completed event filled in

Click Add Webhook to create your webhook. It will be displayed in the list of available webhooks.

CircleCI project settings Webhooks page listing the enabled Log API Webhook for the Workflow Completed event

Next, confirm that you can now receive webhooks from your CircleCI project. To run the project build either:

  • Make a new commit
  • Click Rerun workflow from start

Once the build is complete, go to the terminal where your Hookdeck CLI session is running. There are two new entries. The first one arrived as soon as the webhook was created: CircleCI sends a short test event that contains only the webhook’s name. The second one is the Workflow Completed event from the build. That’s the event to troubleshoot in the rest of this tutorial.

Hookdeck CLI showing two webhook events forwarded to localhost:1337/log-circleci-webhook, both with a 500 status

The entry on the CLI confirms that you are successfully receiving the webhook. But there is a 500 server error.

Each webhook entry on the CLI shows these elements in order:

  1. Date and time the event arrived
  2. HTTP status code (500)
  3. Request method (POST)
  4. Local URL the CLI forwarded the webhook to (http://localhost:1337/log-circleci-webhook), followed by the response time
  5. Link to the event in the Hookdeck Console, where you can view details about your webhooks

CircleCI retries webhook deliveries that don’t get a successful (2xx) response, so the same event may arrive more than once during troubleshooting.

Go on to the next section to fix the 500 error.

Troubleshooting the authentication error

Before making any assumptions, inspect the event in the Hookdeck Console. If the guest session isn’t open yet, open the guest sign-up link from the CLI output in a browser. The Console lists the events that the source received. Click the newer entry, the Workflow Completed event, to open its details.

Hookdeck Console listing two CircleCI events with a 500 status, with the Request tab showing the circleci-event-type and circleci-signature headers

The Request tab shows all the details about the webhook you just received. Take a minute to review the Headers section. CircleCI sends the event type in the circleci-event-type header and the payload signature in the circleci-signature header.

Review the webhook payload in the Body section.

Hookdeck Console Request tab showing the workflow-completed payload body with its project, webhook, and workflow fields

Click the Response tab to review the server’s response.

Hookdeck Console Response tab showing a 500 status and the request body digest error message from the API

The response message reveals that we are failing the security check on the webhook payload. Because we defined an API secret, CircleCI sends the circleci-signature header. This header contains a signature: an HMAC-SHA256 hash of the payload, computed with the secret key. The API must compute the same hash from the payload it received and check that the two match.

This is a security check to prevent attackers from replacing the actual payload with a malicious one that could corrupt your API.

We know that we are receiving the actual payload from CircleCI because we set everything up and triggered the webhook. We know it is not an attacker’s payload causing the authentication failure. Something else must be wrong with our set up.

Our next step is to review the validation logic on our API. This can be found in the server.js file.

//Validate payload
function validatePayload(req, res, next) {
  if (req.method == "POST") {
    if (!req.rawBody) {
      return next("Request body empty");
    }

    // The header is a comma-separated list of versioned signatures (v1=...,v2=...). Check v1.
    const signatures = (req.get(sigHeaderName) || "").split(",");
    const sig = Buffer.from(
      signatures.find((s) => s.startsWith("v1=")) || "",
      "utf8"
    );
    const hmac = crypto.createHmac(sigHashAlg, secret);
    const digest = Buffer.from(
      "v1=" + hmac.update(req.rawBody).digest("hex"),
      "utf8"
    );

    if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
      return next(
        `Request body digest (${digest}) did not match ${sigHeaderName} (${sig})`
      );
    }
  }

  return next();
}
app.use(validatePayload);

Three variables being used here in the code are worth checking out to confirm that they are referencing the correct values. These are:

  • sigHeaderName, which represents the signature header name sent by CircleCI.
  • sigHashAlg, which is the hash algorithm used for the signature.
  • secret, which is the API secret set on our CircleCI webhook.

These values are set on lines 10-12 in the server.js file. Here is the code for these definitions:

const sigHeaderName = "circleci-signature";
const sigHashAlg = "sha256";
const secret = "XXX-XXX";

Did you catch the error? If the secret is not the same as the one we set in our webhook form, the validation will always fail.

CircleCI does not allow you to view the API secret you set in your webhooks a second time; you can only reset it. This is why you need to make sure that you remember the value or store it somewhere secure.

For this tutorial, we know we set the secret as a simple ABCD123 string. In real-world applications, you will want to set a more complicated secret and reference it from an environment variable in your code.

Change the value of the secret to the correct value, save the file, and restart the server.

To test the fix, you need to trigger a new webhook. You can trigger another build on your CircleCI project. If you do not want to wait for a new build, you can use the Retry button to rerun the webhook. The Retry button is next to the status on the Response tab.

After you click Retry, the event’s status changes to 404 in the event list and on the Response tab.

There will also be a new webhook entry on the CLI.

Hookdeck CLI showing the retried Workflow Completed event returning a 404 status

Good work! We have cleared the 500 error, but now we have a new error. This 404 not found error indicates that our specified endpoint (/log-circleci-webhook) cannot be found.

Troubleshooting the “not found” error

Our second webhook attempt resulted in a 404 error on the destination endpoint. This tells us that even though we are successfully receiving our webhooks, the endpoint is not at the specified location, or it does not exist. 404 errors can be fixed, most of the time, by checking for typos or misspellings in the specified route name. Worst-case scenario, the route truly does not exist and needs to be created.

Review the server response for this error on the Response tab.

Hookdeck Console Response tab showing a 404 status and the Cannot POST /log-circleci-webhook message

This shows the actual response from the running server, and it confirms that the specified endpoint cannot be found.

The endpoint specified for the webhook to hit is /log-circleci-webhook, which can be found in the routes.js file at the root of the project directory.

router.post("/log-circleci-hook", async function (req, res) {
  //console.log(req.body);

  const payload = req.body;

  let webhook_info = {
    name: payload.webhook.name,
    project: payload.project.name,
    workflow: payload.workflow.name,
    status: payload.workflow.status
  };

  const save_webhook = await req.db
    .collection("webhooks")
    .insertOne(webhook_info);

  res.status(201).send({
    message: "Webhook Event successfully logged"
  });
});

Closer review of the route handler reveals a typo. On line 10 webhook is incorrectly spelled as hook. Correct this misspelling and save the file.

You will need to restart the Node.js server to allow the changes to take effect. Shut down the server using Ctrl + C and restart it with the npm start command.

Click the Retry button again. The successful webhook is indicated by the 201 status code.

There will also be a webhook entry on your Hookdeck CLI session.

Hookdeck CLI showing the retried Workflow Completed event returning a 201 status

We have a successful operation on our API.

The Response tab shows the server response: a success message.

Hookdeck Console Response tab showing a 201 status and the Webhook Event successfully logged message

With a successful operation on our API, our webhook should be logged. Visit the endpoint /fetch-webhooks-logs to review the collection.

Browser showing the JSON response from localhost:1337/fetch-webhooks-logs with the logged Log API Webhook entry

Conclusion

Just like any other architectural component of an application, webhooks can run into errors. Debugging is a part of the workflow when developing applications or setting up application infrastructure. Errors are inevitable, so knowing what to look out for when a certain error occurs and having the right tools to debug them makes troubleshooting faster and much easier.

In this article, we have demonstrated how to debug CircleCI webhooks in a way that gives clear insight into the errors taking place.

Happy coding!


Fikayo Adepoju is a LinkedIn Learning (Lynda.com) Author, Full-stack developer, technical writer, and tech content creator proficient in Web and Mobile technologies and DevOps with over 10 years experience developing scalable distributed applications. With over 40 articles written for CircleCI, Twilio, Auth0, and The New Stack blogs, and also on his personal Medium page, he loves to share his knowledge to as many developers as would benefit from it. You can also check out his video courses on Udemy.