Using CircleCI workflows to replicate Docker Hub automated builds
Full-Stack Developer and DevOps Specialist at Croz.io
Note from the publisher: This content was updated by Support Engineer Nick Bialostosky on 10/9/2020 to reflect changes to Docker Hub support for access tokens.
CircleCI workflows can be used to make your deployment process simple and intuitive. In this tutorial, we will learn how to use them to automatically push images to the Docker registry, just like Docker Hub’s own automated build process, but with all of the customization that your own build process offers.
CircleCI workflows
A workflow is a set of rules for defining a collection of jobs and their run order. Workflows support complex job orchestration using a simple set of configuration keys to help you resolve failures sooner. - CircleCI Workflows Documentation
In our workflow, pushing a commit to the main branch will run a specific job that publishes an image with the latest tag on Docker Hub. Each commit will also add a Git tag to the image as an image tag and publish the image with that tag, too. Finally, we will look into how to use CircleCI’s environment variables to automatically publish incremental versions of our images instead of using Git tags.
Building a Docker image
Let’s start with a basic Docker image that just prints the arguments passed when it is run. The Dockerfile will look as simple as this:
FROM alpine:3.20.3
ENTRYPOINT [ "echo" ]
Building and running the image locally should show us that it is working correctly:
$ docker build -t building-on-ci .
Sending build context to Docker daemon 66.56kB
Step 1/2 : FROM alpine:3.20.3
3.20.3: Pulling from library/alpine
cf04c63912e1: Already exists
Digest: sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
Status: Downloaded newer image for alpine:3.20.3
---> c157a85ed455
Step 2/2 : ENTRYPOINT [ "echo" ]
---> Running in 085bd0db6bda
Removing intermediate container 085bd0db6bda
---> c981d86e3b7c
Successfully built c981d86e3b7c
Successfully tagged building-on-ci:latest
$ docker run building-on-ci "Hello!"
Hello!
Now that we have our image, we need to push it to our GitHub repository.
To start building our image on each commit, we are going to bootstrap our project with the following config file. In your project’s root directory, create a .circle directory and save the following lines as config.yml:
version: 2
jobs:
build:
environment:
IMAGE_NAME: yemiwebby/building-on-ci
docker:
- image: cimg/base:stable
steps:
- checkout
- setup_remote_docker
- run:
name: Build Docker image
command: docker build --no-cache -t $IMAGE_NAME:latest .
workflows:
version: 2
build-main:
jobs:
- build:
filters:
branches:
only: main
We are using a pre-built Docker image available from CircleCI called cimg/base:stable to run our build. CircleCI has many custom images on Docker Hub that extend the official images with extra tools that are useful in a CI/CD environment. Check the CircleCI images docs for more information.
There is also an environment variable called IMAGE_NAME which we are going to use to set the name of our Docker image. It is imperative that you change it to your username/orgname instead of using mine (yemiwebby). Otherwise, you are going to get an error when trying to publish that image to Docker Hub.
For now, our config has a single workflow called build-main that runs a single build job when we push a commit to the master branch. This job simply builds our Docker image and does nothing more.
One of our steps is called setup_remote_docker and is one of the most important ones when building Docker images on CircleCI. Since our job steps are run inside of a Docker image, they cannot build other Docker images. The setup_remote_docker command tells CircleCI to allocate a new Docker Engine, separate from the environment that is running our job, specifically to execute the Docker commands. Check the building Docker images docs for more information.
Commit the changes and push your project to GitHub. Log in to CircleCI and search for your project. Click on the “Set Up Project” button to start building your project.
Example of execution: https://circleci.com/gh/CIRCLECI-GWP/docker-building-on-ci/1
Publishing an image to Docker Hub with the :latest tag on every commit to master
Now that we are building our image on every commit to main, we need to create a new job to also publish that image to Docker Hub.
Docker Hub does not have access tokens that you can use to access your account, so we are going to need to use our own username/password to log in on the Docker Hub Registry. This is generally done by running:
docker login -u "username" -p "password"
Note: Docker Hub now supports access tokens so you can utilize a token instead of your password for the following instructions if you prefer.
However, having your credentials in clear text like that is a bad practice because it can expose your password. We need to create new CircleCI environment variables with our username and password. There are multiple ways to do that in CircleCI, but if multiple projects are going to push images to Docker Hub, the recommended way is to use Contexts. Otherwise, we can set them directly in the project settings. For this article we are going with the latter method.
From the CircleCI dashboard, click on the project settings, then on the left sidebar, click on the Environment Variables tab. Add two new environment variables called DOCKERHUB_USERNAME and DOCKERHUB_PASS with your Docker Hub username and password, respectively.
After setting the environment variables DOCKERHUB_USERNAME and DOCKERHUB_PASS, let’s create our job:
version: 2
jobs:
build:
environment:
IMAGE_NAME: yemiwebby/building-on-ci
docker:
- image: cimg/base:stable
steps:
- checkout
- setup_remote_docker
- run:
name: Build Docker image
command: docker build --no-cache -t $IMAGE_NAME:latest .
publish-latest:
environment:
IMAGE_NAME: building-on-ci
docker:
- image: cimg/base:stable
steps:
- setup_remote_docker
- run:
name: Publish Docker Image to Docker Hub
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
docker push $IMAGE_NAME:latest
workflows:
version: 2
build-main:
jobs:
- build:
filters:
branches:
only: main
- publish-latest:
requires:
- build
filters:
branches:
only: main
We added a new publish-latest job to the build-main workflow. This job tries to push the image we just built to Docker Hub. However, if we commit that change and check the build log, we will see that it fails because Docker cannot find the image yemiwebby/building-on-ci.
This happens because the jobs in a workflow are isolated from each other, the image we built on the build job is not available to publish-latest.
Note the duplication we have in our configuration file. We are repeating ourselves two times with the environment variable IMAGE_NAME definition and the Docker image that is going to be used to run our jobs.
One way we could use to solve the duplication issue is by using YAML anchors. However, CircleCI has evolved, and with version 2.1, we can now reuse some configurations directly by specifying a reusable executor. At the top of the file we are going to change the version from 2 to 2.1 and add a new executors key:
version: 2.1
executors:
docker-publisher:
environment:
IMAGE_NAME: building-on-ci
docker:
- image: cimg/base:stable
Now on each job, instead of declaring docker and environment keys, we are going to specify a single executor key:
# ...
jobs:
build:
executor: docker-publisher
# ...
publish-latest:
executor: docker-publisher
# ...
Duplication resolved. Time to fix the issue with our Docker image not being available on the publish-latest job. We are going to use Workflows’ workspaces to share the image between the two jobs.
Let’s add a new run step at the end of the build job. This run step is going to save the image as a tar archive:
# ...
- run:
name: Archive Docker image
command: docker save -o image.tar $IMAGE_NAME:latest
# ...
Now, we can persist that file on the workflow workspace by adding another step called persist_to_workspace:
# ...
- run:
name: Archive Docker image
command: docker save -o image.tar $IMAGE_NAME:latest
- persist_to_workspace:
root: .
paths:
- image.tar
# ...
To retrieve the image.tar file on the publish-latest job, we are going to add a new step, attach_workspace, before the others:
# ...
steps:
- attach_workspace:
at: /tmp/workspace
# ...
And load our archived image with docker load just before we try to push it to the Docker Hub registry:
# ...
- run:
name: Load archived Docker image
command: docker load -i /tmp/workspace/image.tar
# ...
Our full configuration file looks like this:
version: 2.1
executors:
docker-publisher:
environment:
IMAGE_NAME: yemiwebby/building-on-ci
docker:
- image: cimg/base:stable
jobs:
build:
executor: docker-publisher
steps:
- checkout
- setup_remote_docker
- run:
name: Build Docker image
command: docker build --no-cache -t $IMAGE_NAME:latest .
- run:
name: Archive Docker image
command: docker save -o image.tar $IMAGE_NAME:latest
- persist_to_workspace:
root: .
paths:
- image.tar
publish-latest:
executor: docker-publisher
steps:
- attach_workspace:
at: /tmp/workspace
- setup_remote_docker
- run:
name: Load archived Docker image
command: docker load -i /tmp/workspace/image.tar
- run:
name: Publish Docker Image to Docker Hub
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
docker push $IMAGE_NAME:latest
workflows:
version: 2
build-main:
jobs:
- build:
filters:
branches:
only: main
- publish-latest:
requires:
- build
filters:
branches:
only: main
Everything should be green now!
Now, we can retrieve and run our image from the Docker Hub registry:
$ docker run yemiwebby/building-on-ci "Workflows are awesome!""
Unable to find image 'yemiwebby/building-on-ci:latest' locally
latest: Pulling from yemiwebby/building-on-ci
43c4264eed91: Pull complete
Digest: sha256:862d8ef9bdcac13179d089fab0972db0a0e272e0763a3ce6dd9a6ab42eb4fadf
Status: Downloaded newer image for yemiwebby/building-on-ci:latest
Workflows are awesome
Publishing Git tags as Docker image tags on Docker Hub
Depending on an image with latest tag can easily cause confusion since we don’t have control over which version of the image we are running. The recommended way is to use a specific tag.
Suppose we are going to use Git tags to release new versions of our Docker image, and we want CircleCI to publish our image with each Git tag as an image tag on Docker Hub. Here is how we can do it.
First, we are going to add a new workflow, build-tags:
# ...
build-tags:
jobs:
- build:
filters:
tags:
only: /^v.*/
branches:
ignore: /.*/
- publish-tag:
requires:
- build
filters:
tags:
only: /^v.*/
branches:
ignore: /.*/
Both jobs share the same filters, which basically say that they must run for every tag that starts with v and should not execute for branches.
The build job is the same (we can reuse jobs between workflows), but the publish-tag job will look like the following:
# ...
publish-tag:
executor: docker-publisher
steps:
- attach_workspace:
at: /tmp/workspace
- setup_remote_docker
- run:
name: Load archived Docker image
command: docker load -i /tmp/workspace/image.tar
- run:
name: Publish Docker Image to Docker Hub
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
IMAGE_TAG=${CIRCLE_TAG/v/''}
docker tag $IMAGE_NAME:latest $IMAGE_NAME:$IMAGE_TAG
docker push $IMAGE_NAME:latest
docker push $IMAGE_NAME:$IMAGE_TAG
# ...
The difference between this job and publish-latest is that this job uses an environment variable set automatically by CircleCI called CIRCLE_TAG. We retrieve the value and remove the v prefix, e.g., v0.0.1 becomes 0.0.1. We then proceed to tag our image with that version and push it to the Docker Hub Registry. We also make sure to push the latest tag, too, so that both are kept in sync. If we commit a tag we can see it working:
$ git tag v0.0.1 && git push origin v0.0.1
And if we access the Docker Hub page for the image, we can see the tag there:
Now, we can run our image using our tag:
docker run yemiwebby/building-on-ci:0.0.1 "Tags are working!"
Unable to find image 'yemiwebby/building-on-ci:0.0.1' locally
0.0.1: Pulling from yemiwebby/building-on-ci
Digest: sha256:862d8ef9bdcac13179d089fab0972db0a0e272e0763a3ce6dd9a6ab42eb4fadf
Status: Downloaded newer image for yemiwebby/building-on-ci:0.0.1
Tags are working!
Automatic image tags for each build
What if we have an image that we don’t use frequently or one that we don’t want to use git tags for? An image that we want to have strict control over which version we are using, so we can’t rely on the latest tag?
CircleCI allow us to do exactly that: it exposes an environment variable called CIRCLE_BUILD_NUM that is an ever-increasing number. Using it, we can let CircleCI publish a new version of the image for every build. Let’s modify our publish-latest job to do exactly that:
# ...
- run:
name: Publish Docker Image to Docker Hub
command: |
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
IMAGE_TAG="0.0.${CIRCLE_BUILD_NUM}"
docker tag $IMAGE_NAME:latest $IMAGE_NAME:$IMAGE_TAG
docker push $IMAGE_NAME:latest
docker push $IMAGE_NAME:$IMAGE_TAG
# ...
We are using CIRCLE_BUILD_NUM to create the image tag and publishing it alongside the latest tag to Docker Hub Registry. If we commit that to main and wait for the workflow to finish, we can see that it was published:
Conclusion
These versioning techniques are just building blocks. You can make tagging the versions of your image as complex or as simple as you want. You could even use this knowledge to upload an image to your own private registry, instead of Docker Hub, for specific tags.
The repository used while writing this article is available on GitHub: https://github.com/CIRCLECI-GWP/docker-building-on-ci
