Using workflows to deploy an API to multiple environments

Deploying an applications to a staging environment for testing before deploying it to production is a common practice. Testing to make sure features are working properly is a best practice, and many developers, startups, and organizations deploy their applications this way.

I have found that it quickly becomes time consuming and inefficient to go through a manual process every time updates are made to my applications. How great would it be to automate the process? In this tutorial, you will deploy an API to both a staging and production environment. This process is also known as multi-deployment.

In a multi-deployment process, you will create a workflow that first deploys an API to a staging environment, and runs automated tests against the staged API using Postman. If the tests pass, your workflow will deploy the API to the production environment without intervention. If the tests do not pass, the API will not be deployed to production.

Prerequisites

To follow this tutorial, a few things are required:

1. Node.js installed on your system
2. Postman for Desktop installed on your system (you can download it here){: target=”_blank”}
3. Heroku or another application hosting platform
4. A CircleCI account
5. A GitHub account

With all these installed and set up, it is time to begin the tutorial.

Cloning the API project

To begin, you will need to clone the API project. You will be working with a simple Node.js API application with a root endpoint and two other endpoints for creating and fetching users. Clone the project by running this command:

git clone --single-branch --branch base-project https://github.com/CIRCLECI-GWP/deploy-api-to-multiple-env.git

When the cloning process is complete, go to the root of the project and install the dependencies:

cd deploy-api-to-multiple-env
npm install

Next, run the application:

npm start

The application will start listening on a default port of 3000.

Open Postman and make a GET request to the http://localhost:3000/users/get endpoint. This will return an array of users.

The API is ready to deploy.

Setting up staging and deploy environments on Heroku

The next task is to create the deployment environments. You will be creating one for staging (where the application’s updates will be deployed, for testing) and deploy (the production environment). You will use Heroku for this tutorial, but you can use the hosting platform you prefer for your other projects.

Go to your Heroku account dashboard and click New, then Create new app. On the app creation page, create the staging application environment.

Now, repeat the same process to create the production app.

The environments are now set up for deploying the application. Finally, on Heroku, get your API Key from the Account tab on the Account Settings page.

Setting up API tests with Postman Collections

After it is deployed to the staging environment, we want the API to be tested. For this tutorial, we will use Postman to set up API tests that can be automated.

The first step is to create a dedicated environment for the API requests. From the Postman desktop, click the Manage Environments cog icon (it is at the top right corner).

Open Postman desktop and select Environments from the left side menu bar of the Postman interface. Existing enviroments will be displayed here if you have any, otherwise, you will have a link to create a new one. Click Create Environment or use the plus icon.

Note: Your Postman UI may look slightly different than the screenshots in this tutorial, but the tools you need will be in the places specified.

Enter a name for the environment, then enter the environment variable, (api_url). Fill in the API base URL with https://users-api-staging-env-096353097de9.herokuapp.com (without a trailing slash). The value in INITIAL VALUE is duplicated in CURRENT VALUE. Keep them identical for this tutorial.

Click Save to create the new environment.

Select the new environment from the dropdown at the top right of the interface.

Creating a collection

The next step is to create a Postman Collection for the user endpoints of the API you will be testing.

Click the Collections tab on the left sidebar of the interface. Then click Create Collection.

Fill in the name (Users) of your collecion. You can also add a description for the collection to provide more insight. Your new collection is listed on the left sidebar on the Collections tab.

Adding requests to the collection

Now it is time to add requests to your collection. The API consists of two endpoints:

• {{api_url}}/users/get (GET): Fetches a list of user profiles
• {{api_url}}/users/create (POST): Creates a new user profile

To add a new request, click the “meatball” menu beside the collection.

Click Add requests. Create a request for the {{api_url}}/users/get endpoint and click Save. Users is the collection you created earlier.

A new request tab is listed. Enter the endpoint for the request in the address bar {{api_url}}/users/get using the api_url variable you created for the current environment.

Next, create another request for the {{api_url}}/users/create endpoint. You need random values so that the POST request can create test users. For that, you will need a Pre-request script.

Open the Pre-request Script tab and add this script:

let random = +new Date();

pm.globals.set("name", `Test-User-${random}`);

This script uses the current timestamp to randomly create names for each fired request. The random name variable is set as a global variable for the request instance.

Now write the request body with the dynamic name variable.

A dynamic name parameter will be used for each request to the /users/create endpoint.

Adding tests

It is time to add some tests to the requests you just created.

Click the {{api_url}}/users/get request (in the left sidebar) to make sure it is loaded. Click the Tests tab. In the window, add:

pm.test("Request is successful with a status code of 200", function () {
  pm.response.to.have.status(200);
});

pm.test("Check that it returns an array", function () {
  var jsonData = pm.response.json();
  pm.expect(jsonData).to.be.an("array");
});

The first test checks that the request returns successfully with a status code of 200. The second test makes sure that it also returns an array.

Click the user creation request Test tab. Then add:

pm.test("User creation was successful", function () {
  pm.expect(pm.response.code).to.be.oneOf([200, 201, 202]);
});

pm.test("Confirm response message", function () {
  var jsonData = pm.response.json();
  pm.expect(jsonData.message).to.eql("User successfully registered");
});

The first test here checks that user creation is successful by asserting a status code of either 200, 201 or 202. The second test makes sure that the correct response message is returned.

Note: Be sure to click Save whenever you make a change to either of the requests in this collection.

Configuring automated testing with Newman

Next, you will use Postman’s CLI tool, Newman, to automate how the tests run. Newman is a command-line collection runner for Postman. It allows you to run and test a Postman collection directly from the command line.

You will need to export both the collection and the environment files. Start by clicking the Export button on the collection flyout menu. Select the format as Collection v2 and click Export. Save the file to the root of your project.

Next, export the Postman environment you created for this tutorial. Click the environment tab from the sidebar and ensure that the staging is selected. Click the “meatballs” menu. Export your environment from the pop-up dialog.

Next, install the Newman CLI tool at the root of your project by running:

npm install --save-dev newman

Replace the test script in package.json with:

"scripts": {
    ...
    "test": "npx newman run [YOUR_COLLECTION_FILENAME] -e ./[YOUR_ENVIRONMENT_FILENAME].json"
}

This test script uses npx to run newman against the collection, using the environment’s filename to specify where the tests will run. Replace the values [YOUR_COLLECTION_FILENAME] and [YOUR_ENVIRONMENT_FILENAME] with your collection filename and environment filename, respectively. Newman runs the tests defined in the collection against the requests and picks up any pre-request scripts that have been defined.

Now your tests are set up for automation.

Connecting your project to CircleCI

As promised at the beginning of this tutorial, your goal is to create an automated workflow in which an application is:

1. Deployed to a staging environment
2. Tested on the staging address and (if tests pass)
3. Deployed to the production environment

Your next task is to write a deployment pipeline script that performs those steps. Start by creating a folder named .circleci at the root of the project. Add a configuration file named config.yml inside the folder you just created. Enter the following code:

version: 2.1
orbs:
  heroku: circleci/heroku@2.0.0
  node: circleci/node@6.3.0
jobs:
  stage:
    executor: heroku/default
    steps:
      - checkout
      - heroku/install
      - heroku/deploy-via-git:
          app-name: $HEROKU_STAGING_APP_NAME

  test:
    working_directory: ~/repo
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - node/install
      - run:
          name: Install Dependencies
          command: npm install
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - ./node_modules
      - run:
          name: Run tests
          command: npm run test

  deploy:
    executor: heroku/default
    steps:
      - checkout
      - heroku/install
      - heroku/deploy-via-git:
          app-name: $HEROKU_PRODUCTION_APP_NAME

workflows:
  stage_test_deploy:
    jobs:
      - stage
      - test:
          requires:
            - stage
      - deploy:
          requires:
            - test

That is quite a script, right? Let me break it down and add some detail. First, you pulled in both heroku and node orb:

orbs:
  heroku: circleci/heroku@2.0.0
  node: circleci/node@6.3.0

Then you create three jobs:

• stage: Checks out the project code from the repo and uses the Heroku CircleCI orb to deploy the application to the staging environment.
• test: Installs the project dependencies needed to have the newman CLI installed. It then runs the test script in package.json, invoking newman to run the tests in the collection against the staging application that was just deployed.
• deploy: Deploys the application to the production environment on Heroku.

Orbs are reusable pipeline packages that abstract common tasks. The Heroku orb that you are using abstracts the process of installing and setting up the heroku CLI tool to deploy the application.

Finally, the script defines the stage_test_deploy workflow. This workflow runs the stage job. Once that job is run successfully, the test job runs tests against the application that was just deployed to staging. When all tests on the staging application have passed, the script runs the deploy job to deploy the application to the production environment. It is a smooth, simple process.

It is time to test your workflow. Commit all changes and push the code to the remote repository to run the pipeline script.

Begin by pushing your project to GitHub.

Next, log into your CircleCI account dashboard and search for the project you just pushed. In this case, the project is named deploy-api-to-multiple-env.

Click Set Up Project.

CircleCI detects the config.yml file for the project. Click Set Up Project. Your first workflow will start running, but it will fail!

The reason for the failure is that you have not set up the environment variables for the project. To do that now, go to the Pipelines page, select your application, then click Project Settings.

From the Project Settings side-menu, click Environment Variables, then click Add Environment Variable.

Add these variables:

• HEROKU_STAGING_APP_NAME: The Heroku app name for the staging environment (in this case users-api-staging-env)
• HEROKU_PRODUCTION_APP_NAME: The Heroku app name for the production environment (in this case users-api-production-env)
• HEROKU_API_KEY: Your Heroku API key

Re-run your workflow from the start, and this time it will run successfully.

Click the workflow (stage_test_deploy) link to review the steps in the process and their duration.

You can also click each build for details about how it ran.

Great work!

Conclusion

Automating real-life manual tasks can help you save both time and effort, boosting your productivity. In this tutorial, you automated the process of staging, testing, and deploying an application to a production environment. You can apply what you have learned here to even more complex scenarios and further increase efficiency in your workflows.

Happy coding!