Skip to content
Docs
Cancel

Setting Up Docker Hub Pull Through Mirror

Last updated
Tags Server v2.x Server Admin

Overview

Starting November 1, 2020, Docker Hub will impose rate limits for anonymous pulls based on the originating IP. To avoid service disruption, it is recommended to authenticate Docker pulls with Docker Hub. You can authenticate using build config (see Using Docker Authenticated Pulls).

Alternatively, you can set up a Docker Hub pull through registry mirror pre-configured with Docker Hub account credentials. Using a pull through registry mirror is potentially simpler than making many build config modifications. It may also bring additional performance improvements since network round-trips to Docker Hub are reduced.

These setup instructions are divided into three parts: Setting up a pull through cache registry, configuring Nomad clients to use the registry as a mirror, and configuring VM Service to use the registry as a mirror in machine executors and remote Docker jobs.

The Services machine does not need to be configured to use the mirror. The Services machine does not pull Docker images regularly (it pulls images only upon initial setup and upgrade processes), and most images are pulled from Replicated. Hence it has few chances to hit the rate limit.

Set up a pull through cache registry

This section covers setting up a pull through cache registry, which works as a mirror and reverse proxy for Docker Hub. Two types of pull through cache registry are presented: The elementary and easier-to-setup version using HTTP, and the more secure option using HTTPS. Either option is functional for most cases, but you might need the more secure option in certain cases.

Prerequisites: Set up Docker Hub account

Before setting up a pull through cache registry, you need to set up a Docker Hub account. The account will be used by the pull through cache registry to authenticate itself with Docker Hub.

We recommend that you set up a dedicated account. Resources that the account has access to will also become accessible by all end users of your CircleCI server installation. This is especially important if the account has private images registered on Docker Hub, as these private images will become accessible by all users of your CircleCI server installation.
While a free account can be used, setting up a paid account would be a good option to ease the rate limiting further: Paid accounts have an unlimited allowance of image pulls. See also: https://www.docker.com/pricing.

Set up an elementary pull through cache registry (HTTP Proxy for Docker Hub)

  1. Set up an independent Linux server that has Docker installed.

    We set up this registry as an independent server (i.e. outside of CircleCI boxes) to avoid load on this cache registry affecting other CircleCI server services.

    Assuming that the IP address for this server is 192.0.2.1, the URL for the registry to set up is http://192.0.2.1. This URL will be needed later to arm Nomad clients and the VM Service.

  2. Run the command below to start the registry.

    Replace DOCKER_HUB_USERNAME and DOCKER_HUB_ACCESS_TOKEN with the username for the Docker Hub account and the access token you obtained through https://hub.docker.com/settings/security, respectively. 
    sudo docker run \
  -d \
  -p 80:5000 \
  --restart=always \
  --name=through-cache \
  -e REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io" \
  -e REGISTRY_PROXY_USERNAME=DOCKER_HUB_USERNAME \
  -e REGISTRY_PROXY_PASSWORD=DOCKER_HUB_ACCESS_TOKEN \
  registry

  3. Finally, make sure that the TCP port 80 (i.e. HTTP) is open and accessible. For better security, we recommend that you only open the port to Nomad clients and VMs for machine executors and remote docker engines.

    On AWS, for example, you will need to make sure that both Security Groups and the firewall at the OS level are configured correctly to accept connections from Nomad clients over HTTP.

Set up a secure pull through cache registry (HTTPS Proxy for Docker Hub)

Under certain circumstances (especially when you are enabling BuildKit on machine executors and remote Docker engines), your pull through cache registry needs to accept connections over HTTPS. These instructions cover setting up a pull through cache that listens for HTTPS connections.

  1. Set up an independent Linux server that has Docker installed.

    We set up this registry as an independent server (i.e. outside of CircleCI boxes) to avoid load on this cache registry affecting other CircleCI server services.

    The server needs to be accessible by its hostname. Namely, you must configure DNS accordingly so that the hostname (assume your-mirror.example.com) resolves to the IP address of the server correctly. In this guide we assume that the URL for the registry we are setting up is https://your-mirror.example.com (be aware that the URL scheme is https, not http). The URL will be needed later to arm Nomad clients and the VM Service.

  2. Obtain a TLS certificate for your-mirror.example.com and put the certificate and the private key into /root/tls of the server.

    This guide assumes you are obtaining a certificate issued by a well-known CA, not a self-signed certificate. Use of self-signed certificates has NOT been tested.

  3. Run the command below to start the registry.

    Replace DOCKER_HUB_USERNAME and DOCKER_HUB_ACCESS_TOKEN with the username for the Docker Hub account and the access token you obtained through https://hub.docker.com/settings/security, respectively.

    Replace fullchain.pem and privkey.pem with the filenames of your certificate and private key, respectively. The certificate specified there must be correctly chained enough for tracing the certification path from the leaf to the root.

    		 
    sudo docker run \
  -d \
  -p 443:5000 \
  --restart=always \
  --name=through-cache-secure \
  -v /root/tls:/data/tls \
  -e REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io" \
  -e REGISTRY_PROXY_USERNAME=DOCKER_HUB_USERNAME \
  -e REGISTRY_PROXY_PASSWORD=DOCKER_HUB_ACCESS_TOKEN \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/data/tls/fullchain.pem \
  -e REGISTRY_HTTP_TLS_KEY=/data/tls/privkey.pem \
  registry

  4. Finally, make sure that the TCP port 443 (i.e. HTTPS) is open and accessible. For better security, we recommend you to open the port only to Nomad clients and VMs for machine executors and remote docker engines.

    On AWS, for example, you will need to make sure that both Security Groups and the firewall at the OS level are configured correctly to accept connections from Nomad clients and VMs for machine/setup_remote_docker jobs over HTTPS.

Plan for renewal of TLS certificates

You will need to renew TLS certificates periodically. These are the steps required to renew certificates.

  1. Update the certificate and the private key in /root/tls.

  2. Run docker restart through-cache-secure.

Technically, this can be automated. For example, if you are using Let’s Encrypt for your certificates, you can setup a cron task that executes certbot renew and the steps above.

Configure Nomad clients to use the pull through cache registry (run for each Nomad client)

  1. Run the command below to configure the registry-mirrors option for the Docker daemon.

    Replace http://192.0.2.1.or.https.your-mirror.example.com with the URL of your pull through cache registry accordingly. 
    sudo bash -c 'cat <<< $(jq ".\"registry-mirrors\" = [\"http://192.0.2.1.or.https.your-mirror.example.com\"]" /etc/docker/daemon.json) > /etc/docker/daemon.json'

  2. Reload Docker daemon to apply the configuration.

    sudo systemctl restart docker.service

Configure VM service to let machine/remote Docker VMs use the pull through cache registry

Follow the steps below on your services machine.

  1. Run the command below to create a directory for your customization files.

    sudo mkdir -p /etc/circleconfig/vm-service

  2. Populate a customization script to be loaded by vm-service. Add the script below to /etc/circleconfig/vm-service/customizations.

    Replace http://192.0.2.1.or.https.your-mirror.example.com in DOCKER_MIRROR_HOSTNAME variable with the URL of your pull through cache registry accordingly.
    This customization is only available in 2.19.0 version and later. 
    export JAVA_OPTS='-cp /resources:/service/app.jar'
export DOCKER_MIRROR_HOSTNAME="http://192.0.2.1.or.https.your-mirror.example.com"

mkdir -p /resources/ec2
cat > /resources/ec2/linux_cloud_init.yaml << EOD
#cloud-config
system_info:
  default_user:
    name: "%1\$s"
ssh_authorized_keys:
  - "%2\$s"
runcmd:
  - bash -c 'if [ ! -f /etc/docker/daemon.json ]; then mkdir -p /etc/docker; echo "{}" > /etc/docker/daemon.json; fi'
  - bash -c 'cat <<< \$(jq ".\"registry-mirrors\" = [\"$DOCKER_MIRROR_HOSTNAME\"]" /etc/docker/daemon.json) > /etc/docker/daemon.json'
  - systemctl restart docker.service
EOD

  3. Restart VM Service to apply the customization.

    sudo docker restart vm-service

Testing your setup

Use private images without explicit authentication

If the Docker ID for the cache registry has a private image, the private image should be accessible without explicit end-user authentication.

Below is a sample config to test the access (assume that the cache registry uses Docker ID yourmachineaccount, and there is a private image yourmachineaccount/private-image-with-docker-client):

version: 2

jobs:
  remote-docker:
    docker:
      - image: yourmachineaccount/private-image-with-docker-client # A copy of library/docker
    steps:
      - setup_remote_docker
      - run: docker pull yourmachineaccount/private-image-with-docker-client

  machine:
    machine: true
    steps:
      - run: docker pull yourmachineaccount/private-image-with-docker-client

workflows:
  version: 2

  run:
    jobs:
      - remote-docker
      - machine

Check logs on the cache registry

By running sudo docker logs through-cache (or sudo docker logs through-cache-secure if you have set up a secure registry) you can see log outputs from your cache registry. If it is operational, there should be messages that the registry is responding to the requests for manifests and blobs with HTTP status code 200.

Reverting the setup

Disarm Nomad clients

Follow the steps below on each Nomad client.

  1. Remove registry-mirrors option in /etc/docker/daemon.json by running the command below.

    sudo bash -c 'cat <<< $(jq "del(.\"registry-mirrors\")" /etc/docker/daemon.json) > /etc/docker/daemon.json'

  2. Run sudo systemctl restart docker.service to apply the change.

Disarm VM service

Follow the steps below on your services machine.

  1. Void the JAVA_OPTS environment variable by running the command below.

    echo 'unset JAVA_OPTS' | sudo tee -a /etc/circleconfig/vm-service/customizations

  2. Run sudo docker restart vm-service to apply the change.

Next steps


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

Creative Commons License
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

6,000 Minutes Free

CircleCI, now with more build minutes
than anyone else for FREE 🎉

Sign up for Free

These teams get it

CCI customer teams