Building Docker images for multiple operating system architectures
Developer Advocate, CircleCI
Often, software is compiled and packaged into artifacts that must function on multiple operating systems (OS) and processor architectures. It is almost impossible to execute an application on an OS/architecture platform other than the one it was designed for. That’s why it’s a common practice to build releases for many different platforms. This can be difficult when the platform you are using to build artifacts is different from the platforms you want to target for deployment. For example, to develop an application on Windows and deploy it to Linux and macOS machines, you must provision and configure build machines for each of the operating systems and architecture platforms you’re targeting. You can create multi-OS builds within pipelines using a variety of techniques, but because of the stringent characteristics of processor architectures, artifacts must be compiled and produced on the same hardware that they are targeting.
Docker is a way to package applications into immutable and deployable artifacts as Docker images and containers. Docker images have the same processor architecture build constraints as traditional artifact packaging. Docker images must be built on the hardware architectures they’re intended to run on. In this post I’ll discuss how to build Docker images using CI pipelines that target multiple processor architectures such as linux/amd64, linux/arm64, linux/riscv64, etc.
Getting started
Take a look at an example code repository that demonstrates how to package an application into multi-architecture Docker images. We’re only going to focus on the continuous integration aspects of building these multi-architecture Docker images. The CircleCI config.yml file defines the CI pipeline build instructions. It is found in the .circleci/config.yml directory. In this post I will focus on the .circleci/config.yml file and the Makefile file from this repo.
Makefiles can be viewed as build/compile directives that are required by the make utility that automates the build processes. The Makefile in this project contains the directives and commands that are executed from the CI pipeline.
Using Docker Buildx
Before I go deeper into the Makefile and config.yml file breakdowns, I want to take a moment to discuss Buildx, a CLI plugin that extends the Docker CLI with the full set of features provided by the Moby BuildKit builder toolkit. It provides the same user experience as docker build with many new features like creating scoped builder instances and building against multiple nodes concurrently.
At the time of this writing, the Buildx feature is still in experimental status, and requires a few environment configurations on the machine where Docker images will be built. The complete Buildx installation instructions can be found here.
Here are Buildx install directions for Docker version 19.03 and higher. These commands compile and build the Buildx binary from source and install it into the Docker plugin directory:
export DOCKER_BUILDKIT=1
docker build --platform=local -o . git://github.com/docker/buildx
mkdir -p ~/.docker/cli-plugins
mv buildx ~/.docker/cli-plugins/docker-buildx
You can also download the latest Buildx binaries for your OS here, and install it using these Buildx release binary directions.
After installing Buildx on your Docker builder machine, you can take advantage of all the Buildx capabilities:
I suggest you take the time to get better familiar with Buildx features. It is an essential technology when building multi-architecture Docker images, and it is heavily used in the examples below.
Configuring your CI pipeline
The config.yml file in the example project leverages the Makefile file and its functionality to execute the appropriate commands to complete the multi-architecture builds. This config.yml demonstrates how to leverage a single build job using a machine executor. This may seem a bit out of the norm since CircleCI provides the ability to build Docker images using the Docker executor.
The Docker platform leverages sharing and managing its host operating system kernels vs. the kernel emulation found in virtual machines (VMs). Since running Docker containers share the host OS kernel, they are architecturally very different from VMs. VMs are not based on container technology. They are made up of the user-spaces and kernel-spaces of an operating system. VM server hardware is virtualized, and each VM has its own isolated OS and apps. It shares hardware resources from the host, and can emulate various processor architectures/kernels within the VM. The kernel and hardware emulation capabilities of VMs are the main reasons the machine executor is the best choice for building multi-architecture Docker images.
Take a look at the config.yml in the example project:
version: 2.1
jobs:
build:
machine:
image: ubuntu-2204:current
environment:
DOCKER_BUILDKIT: 1
BUILDX_PLATFORMS: linux/amd64,linux/arm64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6
steps:
- checkout
- run:
name: Unit Tests
command: make test
- run:
name: Log in to docker hub
command: |
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
- run:
name: Build from dockerfile
command: |
TAG=edge make build
- run:
name: Push to docker hub
command: |
TAG=edge make push
- run:
name: Compose Up
command: |
TAG=edge make run
- run:
name: Check running containers
command: |
docker ps -a
- run:
name: Check logs
command: |
TAG=edge make logs
- run:
name: Compose down
command: |
TAG=edge make down
- run:
name: Install buildx
command: |
BUILDX_BINARY_URL="https://github.com/docker/buildx/releases/download/v0.4.2/buildx-v0.4.2.linux-amd64"
curl --output docker-buildx \
--silent --show-error --location --fail --retry 3 \
"$BUILDX_BINARY_URL"
mkdir -p ~/.docker/cli-plugins
mv docker-buildx ~/.docker/cli-plugins/
chmod a+x ~/.docker/cli-plugins/docker-buildx
docker buildx install
# Run binfmt
docker run --rm --privileged tonistiigi/binfmt:latest --install "$BUILDX_PLATFORMS"
- run:
name: Tag golden
command: |
BUILDX_PLATFORMS="$BUILDX_PLATFORMS" make cross-build
As you may have noticed, most of the command: keys in this config file execute the functions defined in the Makefile. This pattern produces much less YAML syntax in the config file, but does complicate what’s actually being executed in the Makefile.
Next, I’m going to focus on explaining the critical command: keys in this config file.
version: 2.1
jobs:
build:
machine:
image: ubuntu-2204:current
environment:
DOCKER_BUILDKIT: 1
BUILDX_PLATFORMS: linux/amd64,linux/arm64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6
steps:
- checkout
- run:
name: Unit Tests
command: make test
- run:
name: Log in to docker hub
command: |
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
- run:
name: Build from dockerfile
command: |
TAG=edge make build
- run:
name: Push to docker hub
command: |
TAG=edge make push
In this code, the build is using a machine executor and assigning values to the DOCKER_BUILDKIT variable that enables Docker access to the experimental features and Buildx. The BUILDX_PLATFORMS variable is the list of OS and processor architectures that will produce Docker images. This list is targeting the Linux OS and a variety of processor architectures.
The remaining run: and command: keys demonstrate how to execute the application’s unit tests, authenticate to Docker Hub in order to pull and push images, build a Docker image using the Dockerfile found in the /app directory, and push that image to Docker Hub.
Note: The docker login step above ensures that your requests to Docker Hub are authenticated. Whenever you pull images from, or push images to, Docker Hub with CircleCI, we recommend logging in to your Docker Hub account for both docker pull and docker push steps in your CircleCI config. Logging in will make sure that your jobs have access to a higher Docker Hub rate limit.
- run:
name: Install buildx
command: |
BUILDX_BINARY_URL="https://github.com/docker/buildx/releases/download/v0.4.2/buildx-v0.4.2.linux-amd64"
curl --output docker-buildx \
--silent --show-error --location --fail --retry 3 \
"$BUILDX_BINARY_URL"
mkdir -p ~/.docker/cli-plugins
mv docker-buildx ~/.docker/cli-plugins/
chmod a+x ~/.docker/cli-plugins/docker-buildx
docker buildx install
# Run binfmt
docker run --rm --privileged tonistiigi/binfmt:latest --install "$BUILDX_PLATFORMS"
In the code snippet above, the Buildx feature is being used to install the Buildx binary and configure it for usage in the executor. The Buildx tool can build multi-architecture images using a variety of strategies but the easiest method is to use Qemu emulation. It is a generic, open source machine emulator and virtualizer. When BuildKit needs to run a binary for a different architecture, it will automatically load it through a binary registered in the binfmt_misc handler. For QEMU binaries registered with binfmt_misc on the host OS to work transparently inside containers, they must be registed with the fix_binary flag.
The docker run --rm --privileged tonistiigi/binfmt:latest --install "$BUILDX_PLATFORMS" command pulls and spawns a binfmt container for every platform listed in the $BUILD_PLATFORMS variable defined earlier in the file.
- run:
name: Tag golden
command: |
BUILDX_PLATFORMS="$BUILDX_PLATFORMS" make cross-build
The above code snippet specifies the last command to execute in the pipeline. It builds the multi-architecture Docker images we want to target. The command: key is making a call to the cross-build function defined inside the Makefile, so let’s take a look at the underlying commands associated with this function.
# Makefile cross-build function
.PHONY: cross-build
cross-build:
@docker buildx create --name mybuilder --use
@docker buildx build --platform ${BUILDX_PLATFORMS} -t ${PROD_IMAGE} --push ./app
The code snippet above is the actual cross-build make command, which creates a new Buildx builder instance. It follows that with the docker buildx build command that triggers the process to build an individual Docker image for every platform listed in the ${BUILDX_PLATFORMS} environment variable. This is fed into the --platform flag of the command. The -tflag tags/names the Docker images and the --push flag will automatically push the build result to a Docker registry. In this case, it is Docker Hub.
If you want to see the full Makefile and config.yml files, you can find them in the example project. Once deployed to CircleCI, the pipeline will build and push multi-architecture Docker images to Docker Hub.
Conclusion
This post demonstrated how to build various Docker images for multiple operating systems and processor architectures from within a CI pipeline. This post also briefly introduced the Docker Buildx feature, which is currently an experimental utility that is expected to become the defacto build utility in future releases of Docker. I consider Buildx to be the next-gen Docker image building tool that will enable expansive, advanced, and optimized capabilities to enhance the current image building experience.
I also briefly discussed some of the intricacies of building Docker images that target multiple operating systems and platform architectures, which highlight the technical differences between Docker containers and VMs. Though seemingly similar at an abstract view, they are fundamentally different at their cores. Finally, I’ll reiterate that Docker has implemented new Docker Hub rate limits which requires all calls to Docker Hub to be authenticated. Whenever you pull images from, or push images to, Docker Hub on CircleCI, we recommend logging in to your Docker Hub account for both docker pull and docker push steps in your CircleCI config.
Thank you for following this post and I hope you found it useful.
