Deploy applications using CircleCI, Docker, HashiCorp Terraform, and Google Cloud

At the conferences and tech meetups I speak at, I regularly field questions about the continuous delivery of applications to cloud platforms, such as Google Cloud, using HashiCorp Terraform. In this post, I will show you how to deploy an application using CI/CD pipelines, Docker, and Terraform into a Google Cloud instance.

You’ll set up a fully automated pipeline that builds and pushes a Docker image to Docker Hub using CircleCI, then provisions infrastructure on Google Cloud using Terraform. The target environment will be a Google Container-Optimized OS virtual machine, specifically designed for running Docker containers efficiently and securely.

Terraform’s state will be stored remotely in Google Cloud Storage, enabling stateless infrastructure deployment directly from your CI pipeline. Once deployed, your VM will pull the container image from Docker Hub and run it, completing a fully automated, infrastructure-as-code deployment workflow.

Prerequisites

Before you get started, make sure you have the following:

A GitHub account
A CircleCI account
A Docker Hub account
A Google Cloud account
The Google Cloud SDK (gcloud) installed locally for one-time setup
A GCP project with billing enabled and permission to create and manage resources (e.g. Compute Engine, IAM, and Cloud Storage)

Note: You do not need to install Terraform locally. All Terraform steps are handled automatically in the CircleCI pipeline using the remote state stored in Google Cloud Storage.

Getting started

To follow along with this tutorial, begin by cloning the starter branch of the project repository. This branch contains a basic Flask application that you’ll use to build and push a Docker image. The CircleCI configuration to automate this process will be added in the next section.

git clone -b starter https://github.com/CIRCLECI-GWP/python-terraform-project.git

This will clone the project into a python-terraform-project directory.

Your first goal is to trigger a successful CI pipeline that builds the application, packages it into a Docker image, and pushes that image to Docker Hub. Infrastructure deployment using Terraform will follow in the later stages of the workflow.

Add the CircleCI config file

Next, create the .circleci/config.yml file in the root of your project. This file defines the steps CircleCI will run to build, test, and deploy your application.

Paste this code into the file:

version: 2.1
jobs:
  build_test:
    docker:
      - image: cimg/python:3.10.13
    steps:
      - checkout
      - run:
          name: Install Python Dependencies
          command: |
            pip install --user --no-cache-dir -r requirements.txt
      - run:
          name: Run Tests
          command: |
            python test_hello_world.py
  deploy:
    docker:
      - image: cimg/python:3.10.13
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Upgrade pip and install dependencies
          command: |
            python -m pip install --upgrade pip setuptools wheel
            pip install --user --no-cache-dir -r requirements.txt
      - run:
          name: Set Python Path
          command: |
            echo "export PYTHONPATH=$(python -c 'import site; print(site.USER_SITE)')" >> $BASH_ENV
            source $BASH_ENV
      - run:
          name: Install PyInstaller dependencies
          command: |
            pip install pyinstaller-hooks-contrib
      - run:
          name: Build and push Docker image
          command: |
            echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
            echo 'export IMAGE_NAME=python-cicd-terraform' >> $BASH_ENV
            source $BASH_ENV

            docker build --no-cache -t $DOCKER_LOGIN/$IMAGE_NAME:latest -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME:latest
            docker push $DOCKER_LOGIN/$IMAGE_NAME:$TAG
      - run:
          name: Write Docker tag to workspace file
          command: |
            echo $TAG > docker_tag.txt
      - persist_to_workspace:
          root: .
          paths:
            - docker_tag.txt

workflows:
  build_test_deploy:
    jobs:
      - build_test
      - deploy:
          requires:
            - build_test

The configuration below sets up two jobs:

build_test installs Python dependencies and runs unit tests
deploy builds a standalone executable using PyInstaller, packages the app into a Docker image, and pushes it to Docker Hub

Save your changes and push the code to your Github repository.

Build project on CircleCI and deploy to Docker Hub

Once you’ve pushed your project to GitHub, go to the Projects tab in the CircleCI dashboard and find your repository.

![CircleCI dashboard]

Click Set Up, enter the branch that contains your config.yml, then click Set Up Project again to trigger the first pipeline run.

This initial build may fail if you have not yet set the required environment variables.

CircleCI pipeline failed

Configure environment variables

To allow CircleCI to authenticate with external services like Docker Hub you’ll need to add the following [project-level environment variables][14]{: target=”_blank”}:

$DOCKER_LOGIN - your Docker Hub username
$DOCKER_PWD - your Docker Hub password

Go to Project Settings → Environment Variables. Add the two variables.

Return to the CircleCI dashboard and re-run the failed pipeline. If everything is configured correctly, you will get a successful build and a Docker image pushed to your Docker Hub account.

CircleCI pipeline success

Infrastructure as Code

Infrastructure as code (IaC) is the process of managing and provisioning cloud and IT resources using machine-readable configuration files. IaC empowers teams to automate the setup, scaling, and teardown of infrastructure using tools like Terraform.

In this tutorial, you’ll apply IaC principles by using Terraform to automatically provision infrastructure on Google Cloud Platform directly from your CircleCI pipeline.

Set up Google Cloud for Terraform

To provision infrastructure via CI/CD, you’ll need to prepare your Google Cloud project and configure Terraform to authenticate using a service account. Follow the steps below to set up the necessary project, service account, and permissions using the gcloud CLI.

Set the active account

First, authenticate using the gcloud CLI:

gcloud auth login

If this is your first time authenticating, click Console. Accept the “Terms of Service” to continue.

Google Cloud terms

Next, make sure that billing is enabled on your account. You can list your billing accounts with:

gcloud billing accounts list

You should see a list of billing accounts associated with your GCP project listed with the account ID.

If you don’t have an enabled/ open billing account, create one using theseCloud Billing account creation steps. Take note of your billing account ID; you will need it later. You can also reopen a closed billing account and use it for this project.

Note: While following the rest of the tutorial, be sure to replace YOUR_BILLING_ACCOUNT_ID with your billing account ID.

Creating a GCP project

Here, we’ll create a dedicated Google Cloud project to host our function and ensure it’s properly configured with billing and the necessary permissions.

Create a new project via the Google Cloud Console or run the following command.

Note: While following the rest of the tutorial, be sure to replace YOUR_PROJECT_ID with the unique ID for your project. For example, my-circleci-terraform-project

gcloud projects create YOUR_PROJECT_ID \
  --name="My Terraform Project" \
  --set-as-default

The --set-as-default flag sets the newly created or specified Google Cloud project as the default project for your gcloud CLI environment. This means that future gcloud commands will use this project unless you specify a different one with the --project flag. It helps avoid having to repeatedly specify the project ID in subsequent commands.

List your projects using:

gcloud projects list

Note: If your project is not listed, run the command a second time. It might take a moment to show on the list.

The output returned is in this format below.

PROJECT_ID                     NAME                        PROJECT_NUMBER
my-circleci-terraform-project  My Terraform Project        747150897727

Take note of the PROJECT_NUMBER you’ve just created; you will need it later.

Note: While following the rest of the tutorial, be sure to replace YOUR_PROJECT_NUMBER with the correct information.

After creating the project, link it to the enabled billing account with this command:

gcloud billing projects link YOUR_PROJECT_ID --billing-account YOUR_BILLING_ACCOUNT_ID

You should see a message confirming that billing is enabled, similar to:

billingAccountName: billingAccounts/YOUR_BILLING_ACCOUNT_ID
billingEnabled: true
name: projects/YOUR_PROJECT_ID/billingInfo
projectId: YOUR_PROJECT_ID

You can list the projects linked to your billing account with:

gcloud billing projects list --billing-account=YOUR_BILLING_ACCOUNT_ID

Creating the service account and generating credentials

A service account is a special kind of Google account intended for non-human users like applications or virtual machines. It is used by these non-humans to authenticate and interact with Google Cloud services securely. In this section, you will create a dedicated service account to handle deployments and then assign roles to it.

Create the service account using the following command.

Note: While following the rest of the tutorial, be sure to replace SERVICE_ACCOUNT with your unique name for the service account. For example, terraform-deployer.

gcloud iam service-accounts create SERVICE_ACCOUNT \
  --display-name "Terraform Deployer Service Account"

This creates a service account with this ID: terraform-deployer@<your-project-id>.iam.gserviceaccount.com.

Use this command to verify:

gcloud iam service-accounts list

Generate service account credentials

After creating the service account, generate a key for it within your working directory by running the following command.

Run this command in a secure directory, as it will generate a JSON credentials file used for authentication.

gcloud iam service-accounts keys create cicd_tutorial_gcp_creds.json \
  --iam-account="terraform-deployer@$PROJECT_ID.iam.gserviceaccount.com"

This command creates a JSON key file named cicd_tutorial_gcp_creds.json in your current directory. Later on, we’ll use the contents of this file as the value for the GOOGLE_CREDENTIALS environment variable in CircleCI.

Grant IAM roles

Next, grant the service account permissions to manage compute and Terraform resources.

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:SERVICE_ACCOUNT@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/editor"

For production scenarios, you may want more granular roles like these:

roles/compute.admin
roles/iam.serviceAccountUser
roles/storage.admin

Enable services

gcloud services enable \
  compute.googleapis.com \
  iam.googleapis.com \
  cloudresourcemanager.googleapis.com

gcloud services enable compute.googleapis.com \
  --project=YOUR_PROJECT_ID

Creating a GCS Bucket for remote Terraform state

To enable Terraform to track and manage infrastructure across CI/CD runs, you’ll use remote state. This means Terraform will store its state file in a Google Cloud Storage (GCS) bucket, rather than relying on local files.

This setup is essential for team workflows and continuous deployment environments like CircleCI, where builds run in ephemeral containers without persistent storage.

Follow the steps below to create a dedicated bucket for your Terraform state.

Create the Cloud storage bucket

Run the following command to create a GCS bucket:

Note: While following the rest of the tutorial, be sure to replace CLOUD_BUCKET with your unique bucket name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization). For example, my-terraform-state-bucket-circleci. Or, replace us-central1 if you specified a different region when creating the App engine.

gsutil mb -l us-central1 gs://CLOUD_BUCKET

Then, give your service account permission to read and write Terraform state in the bucket:

gsutil iam ch \
  serviceAccount:SERVICE_ACCOUNT@YOUR_PROJECT_ID.iam.gserviceaccount.com:roles/storage.admin \
  gs://CLOUD_BUCKET

This bucket will later be referenced in your backend.tf file to configure remote state. With this setup, your CircleCI pipeline will be able to run terraform init, plan, and apply consistently across deployments.

Defining your Terraform configuration

With your Google Cloud project and service account ready, the next step is to define your infrastructure using Terraform configuration files.

You’ll create two key files in the terraform/google_cloud/ directory:

main.tf defines the infrastructure resources Terraform should create
backend.tf configures where Terraform stores its state file (in this case, Google Cloud Storage)

Create or update `main.tf`

Inside the terraform/google_cloud/ directory, create or update a file named main.tf. Add this content:


variable "project_name" {
  type = string
  default = "my-circleci-terraform-project"
}

variable "port_number" {
  type = string
  default = "5000"
}

variable "docker_image_tag" {
  description = "Docker image tag to deploy"
  type        = string
}

variable "boot_image_name" {
  type = string
  default = "projects/cos-cloud/global/images/cos-stable-69-10895-62-0"
}

data "google_compute_network" "default" {
  name = "default"
}

# Specify the provider (GCP, AWS, Azure)
provider "google"{
  project = var.project_name
  region = "us-central1"
}

resource "google_compute_firewall" "http-5000" {
  name    = "http-5000"
  network = data.google_compute_network.default.name

  allow {
    protocol = "icmp"
  }

  allow {
    protocol = "tcp"
    ports    = [var.port_number]
  }

  source_ranges = ["0.0.0.0/0"]
}

resource "google_compute_firewall" "allow_ssh" {
  name    = "allow-ssh"
  network = data.google_compute_network.default.name

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  source_ranges = ["0.0.0.0/0"]
}

resource "google_compute_instance" "default" {
  name = "default"
  machine_type = "e2-small"
  zone = "us-central1-a"
  tags =[
      "name","default"
  ]

  boot_disk {
    auto_delete = true
    initialize_params {
      image = var.boot_image_name
      type = "pd-standard"
    }
  }

  metadata = {
    gce-container-declaration = <<EOT
spec:
  containers:
    - name: web
      image: "yemiwebby/python-cicd-terraform:${var.docker_image_tag}"
      stdin: false
      tty: false
      ports:
        - containerPort: 5000
          hostPort: 5000
  restartPolicy: Always
EOT
  }

  labels = {
    container-vm = "cos-stable-69-10895-62-0"
    deploy_id    = formatdate("20060102150405", timestamp())
  }

  network_interface {
    network = "default"
    access_config {
      // Ephemeral IP
    }
  }
}

output "Public_IP_Address" {
  value = google_compute_instance.default.network_interface[0].access_config[0].nat_ip
}

This configuration file begins by declaring variables for project-specific values such as the GCP project name, the Docker image tag, and the exposed port. The provider block specifies that Terraform should use the Google Cloud provider and sets the region and project context.

Two firewall rules are defined:

One allows HTTP traffic on port 5000
The other rule enables SSH access on port 22.

The compute instance block provisions a virtual machine running Google’s Container-Optimized OS, configures it to run your Docker image, and maps the application to port 5000.

Lastly, an output block retrieves and displays the public IP address of the deployed VM once the infrastructure is successfully created.

Two firewall rules are defined:

One allows HTTP traffic on port 5000
The other enables SSH access on port 22

The compute instance block provisions a virtual machine running Google’s Container-Optimized OS, configures it to run your Docker image, and maps the application to port 5000. If you pushed your Docker image using a different repository or tag, be sure to update the image reference in the metadata.gce-container-declaration section accordingly.

Lastly, an output block retrieves and displays the public IP address of the deployed VM once the infrastructure is successfully created.

Create `backend.tf`

In the same directory, create a file named backend.tf and add this content:

terraform {
  backend "gcs" {
    bucket = "my-terraform-state-bucket-circleci"
    prefix = "circleci/my-circleci-terraform"
  }
}

Note: While following the rest of the tutorial, be sure to replace my-terraform-state-bucket-circleci with the actual name of your GCS bucket.

This file:

Configures Terraform to store its state file remotely in Google Cloud Storage, rather than locally.
Uses bucket to refer to the name of the GCS bucket you created earlier.
Uses prefix as the folder path inside that bucket where the state will be stored.

This setup allows Terraform to track infrastructure across CI/CD runs, making it ideal for automation workflows in CircleCI.

Update the CircleCI configuration to deploy infrastructure

Now that your Docker image is being built and pushed successfully, it’s time to extend your CircleCI pipeline to deploy infrastructure using Terraform. This updated configuration adds a third job named terraform_deploy, which provisions your Google Cloud VM using the previously defined .tf files.

Update the .circleci/config.yml file with this content:

version: 2.1
jobs:
  build_test:
    docker:
      - image: cimg/python:3.10.13
    steps:
      - checkout
      - run:
          name: Install Python Dependencies
          command: |
            pip install --user --no-cache-dir -r requirements.txt
      - run:
          name: Run Tests
          command: |
            python test_hello_world.py
  deploy:
    docker:
      - image: cimg/python:3.10.13
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Upgrade pip and install dependencies
          command: |
            python -m pip install --upgrade pip setuptools wheel
            pip install --user --no-cache-dir -r requirements.txt
      - run:
          name: Set Python Path
          command: |
            echo "export PYTHONPATH=$(python -c 'import site; print(site.USER_SITE)')" >> $BASH_ENV
            source $BASH_ENV
      - run:
          name: Install PyInstaller dependencies
          command: |
            pip install pyinstaller-hooks-contrib
      - run:
          name: Build and push Docker image
          command: |
            echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
            echo 'export IMAGE_NAME=python-cicd-terraform' >> $BASH_ENV
            source $BASH_ENV

            docker build --no-cache -t $DOCKER_LOGIN/$IMAGE_NAME:latest -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME:latest
            docker push $DOCKER_LOGIN/$IMAGE_NAME:$TAG
      - run:
          name: Write Docker tag to workspace file
          command: |
            echo $TAG > docker_tag.txt
      - persist_to_workspace:
          root: .
          paths:
            - docker_tag.txt

  terraform_deploy:
    docker:
      - image: hashicorp/terraform:light
    steps:
      - checkout
      - attach_workspace:
          at: /tmp/workspace
      - run:
          name: Verify TAG file exists
          command: |
            echo "Contents of /tmp/workspace:"
            ls -l /tmp/workspace
            cat /tmp/workspace/docker_tag.txt
      - run:
          name: Set up GCP credentials
          command: |
            echo "$GOOGLE_CREDENTIALS" > /tmp/account.json
            export GOOGLE_APPLICATION_CREDENTIALS=/tmp/account.json
      - run:
          name: Terraform Init
          command: |
            cd terraform/google_cloud
            terraform init
      - run:
          name: Terraform Plan
          command: |
            TAG=$(cat /tmp/workspace/docker_tag.txt)
            cd terraform/google_cloud
            terraform plan -out=tfplan -var="docker_image_tag=$TAG"
      - run:
          name: Terraform Apply
          command: |
            TAG=$(cat /tmp/workspace/docker_tag.txt)
            cd terraform/google_cloud
            terraform apply -auto-approve tfplan

workflows:
  build_test_deploy:
    jobs:
      - build_test
      - deploy:
          requires:
            - build_test
      - terraform_deploy:
          requires:
            - deploy

The configuration has been updated to include the terraform_deploy job. This job is responsible for provisioning infrastructure on Google Cloud using Terraform. It begins by retrieving the Docker image tag—previously generated and saved by the deploy job, from the shared workspace. It then sets up authentication by writing the GOOGLE_CREDENTIALS environment variable to a temporary credentials file, which Terraform uses to authenticate with GCP.

After authentication, the job initializes the Terraform working directory with terraform init, ensuring all required providers and backend configurations are loaded. It proceeds to generate an execution plan with terraform plan, injecting the image tag as a variable to ensure the correct version of your Docker image is deployed. Finally, it applies the changes with terraform apply, launching or updating the Google Cloud resources defined in your main.tf file and bringing your application live on a VM instance.

Once added, save your changes and push the code to your Github repository.

Before pushing your updated configuration to GitHub, make sure CircleCI can authenticate with Google Cloud by securely passing your service account credentials as an environment variable.

Open the cicd_tutorial_gcp_creds.json file you created earlier and copy its contents. Then, go to Project Settings → Environment Variables and add a new variable named GOOGLE_CREDENTIALS. Paste the contents of the JSON file into the value field.

The terraform_deploy job will use the GOOGLE_CREDENTIALS variable to authenticate with your Google Cloud project.

Now you’re ready to push your changes to GitHub and trigger a full CI/CD pipeline that includes infrastructure provisioning.

CircleCI Terraform pipeline success

Using the public IP address output from the Terraform job, you can access your application in a web browser. The URL will be in the format http://<PUBLIC_IP>:5000, where <PUBLIC_IP> is the public IP address of your Google Cloud VM instance.

GCP instance live

Summary

In this post, you built a fully automated CI/CD pipeline that deploys a Python web application to a live Google Cloud instance. You used CircleCI to run tests, build a Docker image, push it to Docker Hub, and then provision infrastructure using Terraform — all without running Terraform locally.

This workflow demonstrates how to integrate Terraform into your CI/CD process, enabling reliable, reproducible deployments triggered by code changes. With remote state stored in Google Cloud Storage and service account authentication handled via CircleCI environment variables, your infrastructure remains declarative, portable, and secure.

To explore more advanced workflows like gated deployments, approval steps, or multi-environment pipelines, check out the CircleCI documentation and join discussions in the CircleCI community forum. Want to learn about deploying infrastructure with an approval job using Terraform? Visit this post.

Site

Blog

Deploy applications using CircleCI, Docker, HashiCorp Terraform, and Google Cloud

Prerequisites

Getting started

Add the CircleCI config file

Build project on CircleCI and deploy to Docker Hub

Configure environment variables

Infrastructure as Code

Set up Google Cloud for Terraform

Set the active account

Creating a GCP project

Creating the service account and generating credentials

Generate service account credentials

Grant IAM roles

Enable services

Creating a GCS Bucket for remote Terraform state

Create the Cloud storage bucket

Defining your Terraform configuration

Create or update `main.tf`

Create `backend.tf`

Update the CircleCI configuration to deploy infrastructure

Summary

Similar posts you may enjoy

Validating OS-compatibility for locally-run LLMs using Ollama with CI/CD matrix workflows

Understanding Playwright test hooks in the CI context (JavaScript) – A complete tutorial

Kubernetes sidecar deployment using CircleCI

Site

Blog

Prerequisites

Getting started

Add the CircleCI config file

Build project on CircleCI and deploy to Docker Hub

Configure environment variables

Infrastructure as Code

Set up Google Cloud for Terraform

Set the active account

Creating a GCP project

Creating the service account and generating credentials

Generate service account credentials

Grant IAM roles

Enable services

Creating a GCS Bucket for remote Terraform state

Create the Cloud storage bucket

Defining your Terraform configuration

Create or update main.tf

Create backend.tf

Update the CircleCI configuration to deploy infrastructure

Summary

Similar posts you may enjoy

Validating OS-compatibility for locally-run LLMs using Ollama with CI/CD matrix workflows

Understanding Playwright test hooks in the CI context (JavaScript) – A complete tutorial

Kubernetes sidecar deployment using CircleCI

Create or update `main.tf`

Create `backend.tf`