Skip to content
Docs
Cancel

CircleCI Server v3.x Installation Phase 1

Last updated
Tags Server v3.x Server Admin

Phase 1: Prerequisites

CircleCI server v3.x is installed in 4 phases. There is a validation step at the end of each phase, allowing you to confirm success before moving to the next phase. Depending on your requirements, phases 3 and 4 may include multiple steps. This installation guide assumes you have already read the CircleCI Server v3.x Overview.

Flow chart showing the installation flow for server 3.x with phase 1 highlighted
Figure 1. Installation Experience Flow Chart Phase 1
In the following sections, replace any items or credentials displayed between < > with your details.

Install required software

Download and install the following software before continuing:

Tool Version Used for Notes

Terraform

0.15.4 or greater

Infrastructure Management

kubectl

1.19 or greater

Kubernetes CLI

Helm

3.4.0 or greater

Kubernetes Package Management

KOTS: Mac or Linux.

1.65.0 *

Replicated Kubernetes Application Management. KOTS is a kubectl plugin.

Once you have extracted kots from the tar.gz (tar zxvf kots_linux_amd64.tar.gz), run sudo mv kots /usr/local/bin/kubectl-kots to install it. Mac users will need to grant a security exception.

Velero CLI

Latest

Backup and restore capability

See Velero’s supported providers documentation for further information.

* Please take note of the supported KOTS versions for your Kubernetes cluster. KOTS version compatibility

AWS required software

GCP required software

  • gcloud and gsutil. You can install and set-up these tools up by installing Google Cloud SDK. For further information refer to the Google Cloud SDK docs.

S3 compatible storage required software

  • Install and configure MinIO CLI for your storage provider.

Create a Kubernetes cluster

CircleCI server installs into an existing Kubernetes cluster. The application uses a large number of resources. Depending on your usage, your Kubernetes cluster should meet the following requirements:

Number of daily active CircleCI users Minimum Nodes Total CPU Total RAM NIC speed

< 500

3

12 cores

32 GB

1 Gbps

500+

3

48 cores

240 GB

10 Gbps

Supported Kubernetes versions:

CircleCI Version Kubernetes Version

3.0.0 - 3.2.1

< 1.21

3.2.2 - 3.3.0

1.16 - 1.21

3.4.0 - 3.4.1

1.16 - 1.23

Creating a Kubernetes cluster is your responsibility. Please note:

  • Your cluster must have outbound access to pull Docker containers and verify your license. If you do not want to provide open outbound access, see our list of ports that need access.

  • You must have appropriate permissions to list, create, edit, and delete pods in your cluster. Run this command to verify your permissions:

    kubectl auth can-i <list|create|edit|delete> pods

  • There are no requirements regarding VPC setup or disk size for your cluster. It is recommended that you set up a new VPC rather than use an existing one.

EKS

You can learn more about creating an Amazon EKS cluster here. We recommend using eksctl to create your cluster, which creates a VPC and selects the proper security groups for you.

  1. Install and configure the AWS CLI for your AWS account.

  2. Install eksctl.

  3. Create your cluster by running the following (Cloud formation with eksctl and EKS can take more than 20 minutes to complete):

    eksctl create cluster --name=circleci-server --nodes 4 --node-type m5.xlarge

  4. Once the cluster has been created, you can use the following command to configure kubectl access:

    eksctl utils write-kubeconfig --cluster circleci-server
You may see the following error AWS STS Access - cannot get role ARN for current session: InvalidClientTokenID. This means your AWS credentials are invalid, or your IAM user does not have permission to create an EKS cluster. Proper IAM permissions are necessary to use eksctl. See the AWS documentation on IAM permissions.

GKE

You can learn more about creating a GKE cluster here.

Do not use Autopilot cluster. CircleCI requires functionality that is not supported by GKE Autopilot.

  1. Install and configure the GCP CLI for your GCP account. This includes creating a Google Project, which will be required to create a cluster within your project.

    When you create your project, make sure you also enable API access. If you do not enable API access, the command we will run next (to create your cluster) will fail.

  2. Create your cluster by running the following command:

    gcloud container clusters create circleci-server --project <YOUR_GOOGLE_CLOUD_PROJECT_ID> --region europe-west1 --num-nodes 3 --machine-type n1-standard-4

  3. Configure kubectl with your your gcloud credentials:

    gcloud container clusters get-credentials circleci-server --region europe-west1

  4. Verify your cluster:

    kubectl cluster-info

  5. Create a service account for this cluster:

    gcloud iam service-accounts create <YOUR_SERVICE_ACCOUNT_ID> --description="<YOUR_SERVICE_ACCOUNT_DESCRIPTION>"  --display-name="<YOUR_SERVICE_ACCOUNT_DISPLAY_NAME>"

  6. Get the credentials for the service account:

    gcloud iam service-accounts keys create <PATH_TO_STORE_CREDENTIALS> --iam-account <SERVICE_ACCOUNT_ID>@<YOUR_GOOGLE_CLOUD_PROJECT_ID>.iam.gserviceaccount.com
Enable Workload Identities in GKE (optional)

Workload Identities for GKE allow workloads/pods in your GKE cluster to impersonate IAM service accounts to access Google Cloud services without using static service account credentials. In order to use Workload Identities you must enable them on your GKE cluster.

  1. Enable Workload Identity on existing cluster

      gcloud container clusters update "<CLUSTER_NAME>" \
    --region="<REGION>" \
    --workload-pool="<PROJECT_ID>.svc.id.goog"

  2. Get node pools of existing GKE cluster

      gcloud container node-pools list --cluster "<CLUSTER_NAME>" --region "<REGION>"

  3. Update existing node pools

      gcloud container node-pools update "<NODEPOOL_NAME>" \
    --cluster="<CLUSTER_NAME>" \
    --workload-metadata="GKE_METADATA" \
    --region="<REGION>"

You must repeat Step 3 for all the existing node pools. Follow these links for steps to enable Workload Identity for your Kubernetes service accounts: Nomad Autoscaler, VM and Object-Storage

Create a new GitHub OAuth app

If GitHub Enterprise and CircleCI server are not on the same domain, then images and icons from GHE will fail to load in the CircleCI web app.

Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information.

  1. In your browser, navigate to your GitHub instance > Settings > Developer Settings > OAuth Apps and click the New OAuth App button.

    Screenshot showing setting up a new OAuth app
    Figure 2. New GitHub OAuth App

  2. Complete the following fields, based on your planned installation:

    • Homepage URL: The URL of your planned CircleCI installation.

    • Authorization callback URL: The authorization callback URL is the URL of your planned CircleCI installation followed by /auth/github

  3. Once completed, you will be shown the Client ID. Select Generate a new Client Secret to generate a Client Secret for your new OAuth App. You need these values when you configure CircleCI server.

    Screenshot showing GitHub Client ID
    Figure 3. Client ID and Secret
If using GitHub Enterprise, you also need a personal access token and the domain name of your GitHub Enterprise instance.

Frontend TLS certificates

By default, CircleCI server creates self-signed certificates to get you started. In production, you should supply a certificate from a trusted certificate authority. The Let’s Encrypt certificate authority, for example, can issue a free certificate using their certbot tool. The sections below cover using Google Cloud DNS and AWS Route 53.

AWS Route 53

  1. If you are using AWS Route 53 for DNS, you need the certbot-route53 plugin installed. You can install the plugin with the following command:

    pip3 install certbot-dns-route53

  2. Then execute this example to create a private key and certificate (including intermediate certificates) locally in /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>:

    certbot certonly --dns-route53 -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"
It is important that your certificate contains both your domain and the app.* subdomain as subjects. For example, if you host your installation at server.example.com, your certificate must cover app.server.example.com and server.example.com.

Google Cloud DNS

  1. If you host your DNS on Google Cloud, you need the certbot-dns-google plugin installed. You can install the plugin with the following command:

    pip3 install certbot-dns-google

  2. Then, the following commands will provision a certification for your installation:

    certbot certonly --dns-google --dns-google-credentials <PATH_TO_CREDENTIALS> -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"
It is important that your certificate contains both your domain and the app.* subdomain as subjects. For example, if you host your installation at server.example.com, your certificate must cover app.server.example.com and server.example.com.

You will need these certificates later, and they can be retrieved locally with the following commands:

ls -l /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>
cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/fullchain.pem
cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/privkey.pem

Encryption/signing keys

These keysets are used to encrypt and sign artifacts generated by CircleCI. You need these values to configure server.

Store these values securely. If they are lost, job history and artifacts will not be recoverable.

Artifact signing key

To generate an artifact signing key, run the following command:

docker run circleci/server-keysets:latest generate signing -a stdout

Encryption signing key

To generate an encryption signing key, run the following command:

docker run circleci/server-keysets:latest generate encryption -a stdout

Object storage and permissions

Server 3.x hosts build artifacts, test results, and other state object storage. We support the following:

While any S3 compatible object storage may work, we test and support AWS S3 and MinIO. For object storage providers that do not support S3 API, such as Azure blob storage, we recommend using MinIO Gateway.

Please choose the option that best suits your needs. A Storage Bucket Name is required, in addition to the fields listed below, depending on whether you are using AWS or GCP. Before proceeding, ensure the bucket name you provide exists in your chosen object storage provider.

If you are installing behind a proxy, object storage should be behind this proxy also. Otherwise proxy details will need to be supplied at the job level within every project .circleci/config.yml to allow artifacts, test results, cache save and restore, and workspaces to work. For more information see the Configuring a Proxy guide.

Create an S3 storage bucket

You will need the following details when you configure CircleCI server.

  • Storage Bucket Name - The bucket name to be used for server.

  • Access Key ID - Access Key ID for S3 bucket access.

  • Secret Key - Secret Key for S3 bucket access.

  • AWS S3 Region - AWS region of bucket, if your provider is AWS. You will either have an AWS region or S3 Endpoint, depending on your specific setup.

  • S3 Endpoint - API endpoint of S3 storage provider, when your storage provider is not Amazon S3.

Steps to create your S3 bucket:

  1. Create AWS S3 Bucket

    aws s3api create-bucket \
    --bucket <YOUR_BUCKET_NAME> \
    --region <YOUR_REGION> \
    --create-bucket-configuration LocationConstraint=<YOUR_REGION>
    us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit the bucket configuration

  2. Create an IAM user for CircleCI server

    aws iam create-user --user-name circleci-server

  3. Create a policy document policy.json

    If using IAM Roles for Service Accounts (IRSA) for authentication, use the following content

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutAnalyticsConfiguration",
        "s3:GetObjectVersionTagging",
        "s3:CreateBucket",
        "s3:GetObjectAcl",
        "s3:GetBucketObjectLockConfiguration",
        "s3:DeleteBucketWebsite",
        "s3:PutLifecycleConfiguration",
        "s3:GetObjectVersionAcl",
        "s3:PutObjectTagging",
        "s3:DeleteObject",
        "s3:DeleteObjectTagging",
        "s3:GetBucketPolicyStatus",
        "s3:GetObjectRetention",
        "s3:GetBucketWebsite",
        "s3:GetJobTagging",
        "s3:DeleteObjectVersionTagging",
        "s3:PutObjectLegalHold",
        "s3:GetObjectLegalHold",
        "s3:GetBucketNotification",
        "s3:PutBucketCORS",
        "s3:GetReplicationConfiguration",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:GetObject",
        "s3:PutBucketNotification",
        "s3:DescribeJob",
        "s3:PutBucketLogging",
        "s3:GetAnalyticsConfiguration",
        "s3:PutBucketObjectLockConfiguration",
        "s3:GetObjectVersionForReplication",
        "s3:GetLifecycleConfiguration",
        "s3:GetInventoryConfiguration",
        "s3:GetBucketTagging",
        "s3:PutAccelerateConfiguration",
        "s3:DeleteObjectVersion",
        "s3:GetBucketLogging",
        "s3:ListBucketVersions",
        "s3:ReplicateTags",
        "s3:RestoreObject",
        "s3:ListBucket",
        "s3:GetAccelerateConfiguration",
        "s3:GetBucketPolicy",
        "s3:PutEncryptionConfiguration",
        "s3:GetEncryptionConfiguration",
        "s3:GetObjectVersionTorrent",
        "s3:AbortMultipartUpload",
        "s3:PutBucketTagging",
        "s3:GetBucketRequestPayment",
        "s3:GetAccessPointPolicyStatus",
        "s3:GetObjectTagging",
        "s3:GetMetricsConfiguration",
        "s3:PutBucketVersioning",
        "s3:GetBucketPublicAccessBlock",
        "s3:ListBucketMultipartUploads",
        "s3:PutMetricsConfiguration",
        "s3:PutObjectVersionTagging",
        "s3:GetBucketVersioning",
        "s3:GetBucketAcl",
        "s3:PutInventoryConfiguration",
        "s3:GetObjectTorrent",
        "s3:PutBucketWebsite",
        "s3:PutBucketRequestPayment",
        "s3:PutObjectRetention",
        "s3:GetBucketCORS",
        "s3:GetBucketLocation",
        "s3:GetAccessPointPolicy",
        "s3:GetObjectVersion",
        "s3:GetAccessPoint",
        "s3:GetAccountPublicAccessBlock",
        "s3:ListAllMyBuckets",
        "s3:ListAccessPoints",
        "s3:ListJobs"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_BUCKET_NAME>",
        "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "sts:AssumeRole"
      ],
      "Resource": "<YOUR_OBJECT_STORAGE_ROLE>"
    }
  ]
}

    Otherwise, if using IAM keys for authentication, use the following content

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutAnalyticsConfiguration",
        "s3:GetObjectVersionTagging",
        "s3:CreateBucket",
        "s3:GetObjectAcl",
        "s3:GetBucketObjectLockConfiguration",
        "s3:DeleteBucketWebsite",
        "s3:PutLifecycleConfiguration",
        "s3:GetObjectVersionAcl",
        "s3:PutObjectTagging",
        "s3:DeleteObject",
        "s3:DeleteObjectTagging",
        "s3:GetBucketPolicyStatus",
        "s3:GetObjectRetention",
        "s3:GetBucketWebsite",
        "s3:GetJobTagging",
        "s3:DeleteObjectVersionTagging",
        "s3:PutObjectLegalHold",
        "s3:GetObjectLegalHold",
        "s3:GetBucketNotification",
        "s3:PutBucketCORS",
        "s3:GetReplicationConfiguration",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:GetObject",
        "s3:PutBucketNotification",
        "s3:DescribeJob",
        "s3:PutBucketLogging",
        "s3:GetAnalyticsConfiguration",
        "s3:PutBucketObjectLockConfiguration",
        "s3:GetObjectVersionForReplication",
        "s3:GetLifecycleConfiguration",
        "s3:GetInventoryConfiguration",
        "s3:GetBucketTagging",
        "s3:PutAccelerateConfiguration",
        "s3:DeleteObjectVersion",
        "s3:GetBucketLogging",
        "s3:ListBucketVersions",
        "s3:ReplicateTags",
        "s3:RestoreObject",
        "s3:ListBucket",
        "s3:GetAccelerateConfiguration",
        "s3:GetBucketPolicy",
        "s3:PutEncryptionConfiguration",
        "s3:GetEncryptionConfiguration",
        "s3:GetObjectVersionTorrent",
        "s3:AbortMultipartUpload",
        "s3:PutBucketTagging",
        "s3:GetBucketRequestPayment",
        "s3:GetAccessPointPolicyStatus",
        "s3:GetObjectTagging",
        "s3:GetMetricsConfiguration",
        "s3:PutBucketVersioning",
        "s3:GetBucketPublicAccessBlock",
        "s3:ListBucketMultipartUploads",
        "s3:PutMetricsConfiguration",
        "s3:PutObjectVersionTagging",
        "s3:GetBucketVersioning",
        "s3:GetBucketAcl",
        "s3:PutInventoryConfiguration",
        "s3:GetObjectTorrent",
        "s3:PutBucketWebsite",
        "s3:PutBucketRequestPayment",
        "s3:PutObjectRetention",
        "s3:GetBucketCORS",
        "s3:GetBucketLocation",
        "s3:GetAccessPointPolicy",
        "s3:GetObjectVersion",
        "s3:GetAccessPoint",
        "s3:GetAccountPublicAccessBlock",
        "s3:ListAllMyBuckets",
        "s3:ListAccessPoints",
        "s3:ListJobs"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_BUCKET_NAME>",
        "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
      ]
    }
  ]
}

  4. Attach policy to user

    aws iam put-user-policy \
  --user-name circleci-server \
  --policy-name circleci-server \
  --policy-document file://policy.json

  5. Create Access Key for user circleci-server

    You will need this when you configure your server installation later. 
    aws iam create-access-key --user-name circleci-server

    The result should look like this:

    {
  "AccessKey": {
        "UserName": "circleci-server",
        "Status": "Active",
        "CreateDate": "2017-07-31T22:24:41.576Z",
        "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
        "AccessKeyId": <AWS_ACCESS_KEY_ID>
  }
}

Create a Google Cloud storage bucket

You will need the following details when you configure CircleCI server.

  • Storage Bucket Name - The bucket used for server.

  • You can choose one of the following:

    • Service Account JSON - A JSON format key of the Service Account to use for bucket access.

    • Service Account Email - Service Account Email id if using Google Workload Identity.

A dedicated service account is recommended. Add to it the Storage Object Admin role, with a condition on the resource name limiting access to only the bucket specified above. For example, enter the following into the Google’s Condition Editor in the IAM console:

Use startsWith and prefix the bucket name with projects/_/buckets/. 
resource.name.startsWith("projects/_/buckets/<YOUR_BUCKET_NAME>")

  1. Create a GCP bucket

    If your server installation runs within a GKE cluster, ensure that your current IAM user is a cluster admin for this cluster, as RBAC (role-based access control) objects need to be created. More information can be found in the GKE documentation.

    gsutil mb gs://circleci-server-bucket

  2. Create a Service Account

    gcloud iam service-accounts create circleci-server --display-name "circleci-server service account"

    You will need the email for the service account in the next step. Run the following command to find it:

    gcloud iam service-accounts list \
  --filter="displayName:circleci-server account" \
  --format 'value(email)'

  3. Grant Permissions to Service Account

    gcloud iam roles create circleci_server \
    --project <PROJECT_ID> \
    --title "CircleCI Server"
    gcloud projects add-iam-policy-binding <PROJECT_ID> \
    --member serviceAccount:<SERVICE_ACCOUNT_EMAIL> \
    --role projects/<PROJECT_ID>/roles/circleci_server
    gsutil iam ch serviceAccount:<SERVICE_ACCOUNT_EMAIL>:objectAdmin gs://circleci-server-bucket

  4. JSON Key File

    This step is NOT required if using Workload Identities.

    After running the following command, you should have a file named circleci-server-keyfile in your local working directory. You will need this when you configure your server installation.

    gcloud iam service-accounts keys create circleci-server-keyfile \
    --iam-account <SERVICE_ACCOUNT_EMAIL>

  5. Enable workload Identity

    This step is required only if you are using Workload Identities for GKE. Steps to enable Workload Identities are here

    gcloud iam service-accounts add-iam-policy-binding <YOUR_SERVICE_ACCOUNT_EMAIL> \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:<GCP_PROJECT_ID>.svc.id.goog[circleci-server/object-storage]"
    gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
    --member serviceAccount:<YOUR_SERVICE_ACCOUNT_EMAIL> \
    --role roles/iam.serviceAccountTokenCreator \
    --condition=None
If you are switching from static JSON credentials to Workload Identity, you should delete the keys from GCP as well as from CircleCI KOTS Admin Console.

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

Creative Commons License
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

6,000 Minutes Free

CircleCI, now with more build minutes
than anyone else for FREE 🎉

Sign up for Free

These teams get it

CCI customer teams