Skip to content
Docs
Cancel

Installation on AWS with Terraform

Last updated
Tags Server v2.x
CircleCI Server version 2.x is no longer a supported release. Please consult your account team for help in upgrading to a supported release.

Following is a step by step guide to installing CircleCI server v2.x with Terraform.

Define Variables for Terraform

  1. Clone the Setup repository. If you already have it cloned, make sure it is up-to-date and you are on the main branch by running:

    git checkout main && git pull

  2. Go to the top directory of the enterprise-setup repo on your local machine.

  3. Run terraform init to initialize your working directory.

  4. Run make init to initialize a terraform.tfvars file (your previous terraform.tfvars if any, will be backed up in the same directory).

  5. Open terraform.tfvars in an editor and fill in appropriate AWS values for section 1.

  6. If you plan to use 1.0 builders, specify a circle_secret_passphrase in section 2, replacing …​ with alpha numeric characters, if not, leave it as is. 1.0 builders are disabled by default in section 3.

  7. Specify the instance type to use for your Nomad clients. By default, the value specified in the terraform.tfvars file for Nomad clients is m5.2xlarge (8 vCPUs, 32GB RAM). To increase the number of concurrent CircleCI jobs that each Nomad client can run, modify section 2 of the terraform.tfvars file to specify a larger nomad_client_instance_type. Refer to the AWS Amazon EC2 Instance Types guide for details, and read our guidance in the System Requirements document.

    The builder_instance_type is only used for CircleCI 1.0 and is disabled by default in section 3.

  8. In section 3 you can:

    1. choose to use 1.0 Builders if your project requires it (by changing the count to 1)

    2. enter proxy details, and enter a prefix if there will be multiple installations within your AWS region – the Services and Nomad client instances will be displayed with this prefix in the AWS console.

Example tfvars
#####################################
# 1. Required Cloud Configuration
#####################################

aws_access_key = "..."
aws_secret_key = "..."
aws_region = "eu-central-1"
aws_vpc_id = "..."
aws_subnet_id = "..."
aws_ssh_key_name = "..."

#####################################
# 2. Required CircleCI Configuration
#####################################

circle_secret_passphrase = "..."
services_instance_type = "m4.2xlarge"
builder_instance_type = "r3.4xlarge"
nomad_client_instance_type = "m4.2xlarge"

#####################################
# 3. Optional Cloud Configuration
#####################################

# Set this to `1` or higher to enable CircleCI 1.0 builders
desired_builders_count = "0"

# Provide proxy address if your network configuration requires it
http_proxy = ""
https_proxy = ""
no_proxy = ""

# Use this var if you have multiple installation within one AWS region
prefix = "..."

Above is an example of the terraform.tfvars file you will be editing. The table below shows some of the default settings, and some optional variables that can be used to further customize your cluster. A full list of variables and defaults can be found in the variables.tf file in the root of the enterprise-setup directory.

If you require your installation to work on AWS GovCloud, you can enable this by setting enable_govcloud to true.

Optional vars:

Var Description Default

services_instance_type

Instance type for the centralized services box. We recommend a m4 instance

m4.2xlarge

builder_instance_type

Instance type for the 1.0 builder machines. We recommend a r3 instance

r3.2xlarge

max_builders_count

Max number of 1.0 builders

2

nomad_client_instance_type

Instance type for the nomad clients (2.0 builders). We recommend a XYZ instance

m4.xlarge

max_clients_count

Max number of nomad clients

2

prefix

Prefix for resource names

circleci

enable_nomad

Provisions a nomad cluster for CircleCi Server v2.x

1

enable_route

Enable creating a Route53 route for the Services box

0

enable_govcloud

Allows deployment into AWS GovCloud

false

services_user_data_enabled

Set to 0 to disable automated installation on Services Box

1

force_destroy_s3_bucket

Add/Remove ability to forcefully destroy S3 bucket when your installation is shut down

false

services_disable_api_termination

Protect the services instance from API termination. Set to false if you would like to terminate the Services box automatically when your installation is shut down

true

Provision Instances

  1. Save your changes to the tfvars file and run the following:

    terraform plan

  2. To provision your instances, run the following:

    terraform apply

    You will be asked to confirm if you wish to go ahead by typing yes.

  3. An IP address will be provided at the end of the Terraform output. Visit this IP to carry on the install process.

Access Your Installation

  1. Your browser may prompt you with a SSL/TLS info box. This is just to inform you that on the next screen your browser might tell you the connection to the admin console is unsafe, but you can be confident it is secure. Click Continue to Setup and proceed to your installation IP.

    SSL Security
    Figure 1. SSL Security

  2. Enter your hostname. This can be your domain name or public IP of the Services Machine instance. At this time you can also upload your SSL public key and certificate if you have them. To proceed without providing these click Use Self-Signed Cert – choosing this option prompts security warnings each time you visit the Management Console.

    Hostname
    Figure 2. Hostname

  3. Upload your license.

  4. Decide how to secure the Management Console. You have three options:

    1. Anonymous admin access to the console, anyone on port 8800 can access (not recommended)

    2. Set a password that can be used to securely access the Management Console (recommended)

    3. Use your existing directory-based authentication system (for example, LDAP)

      Secure the Management Console
      Figure 3. Admin Password

  5. Your CircleCI installation will be put through a set of preflight checks, once they have completed, scroll down and click Continue.

    Preflight Checks
    Figure 4. Preflight Checks

Installation Setup

You should now be on the Management Console settings page (your-circleci-hostname.com:8800).

You can make changes to the settings on this page at any time but changes here will require downtime while the service is restarted. Some settings are covered in more detail in out Operations Guide.

  1. Hostname – The Hostname field should be pre-populated from earlier in the install process, but if you skipped that step, enter your domain or public IP of the Services machine instance. You can check this has been entered correctly by clicking Test Hostname Resolution.

  2. Services – The Services section is only used when externalizing services. Externalization is available with a Premium service contract. Contact support@circleci.com if you would like to find out more.

    Hostname and Services Settings
    Figure 5. External Services

  3. Execution Engines – only select 1.0 Builders if you require them for a legacy project – most users will leave this unchecked.

  4. Builders Configuration – select Cluster in the 2.0 section. The Single box option will run jobs on the Services machine, rather than a dedicated instance, so is only suitable for trialling the system, or for some small teams.

    Execution Engine
    Figure 6. 1.0 and 2.0 Builders

  5. GitHub Integration – register CircleCI as a new OAuth application in GitHub.com or GitHub Enterprise by following the instructions provided on the page.

    If you get an "Unknown error authenticating via GitHub. Try again, or contact us." message, try using http: instead of https: for the Homepage URL and callback URL.

    1. Copy the Client ID and Secret from GitHub and paste it into the relevant fields, then click Test Authentication.

    2. If you are using GitHub.com, move on to step 6. If using Github Enterprise, you will also need to follow some supplementary steps and supply an API Token so we can verify your organization. To provide this, complete the following from your GitHub Enterprise dashboard:

      1. Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.

      2. Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes, we only require the default public read-level access so no extra permissions are required. We recommend this token should be shared across your organization rather than being owned by a single user.

      3. Copy the new token and paste it into the GitHub Enterprise Default API Token field.

        Github Integration
        Figure 7. Enter Github Enterprise Token

  6. LDAP – if you wish to use LDAP authentication for your installation, enter the required details in the LDAP section. For a detailed run-through of LDAP settings, read our LDAP authentication guide

  7. Privacy – We recommend using an SSL certificate and key for your install. You can submit these in the Privacy section if this step was missed during the installation.

    Privacy settings
    Figure 8. Privacy Settings

  8. Storage – We recommend using S3 for storage and all required fields for Storage are pre-populated. The IAM user, as referred to in the planning section of this document, is used here.

    Storage options
    Figure 9. Storage Options

  9. Enhanced AWS Integration – Complete this section if you are using 1.0 builders.

  10. Email Complete the Email section if you wish to configure your own email server for sending build update emails. Leave this section is you wish to use our default email server.

    Due to an issue with our third party tooling, Replicated, the Test SMTP Authentication button is not currently working

  11. VM Provider – Configure VM service if you plan to use Remote Docker or machine executor (Linux/Windows) features. We recommend using an IAM instance profile for authentication, as described in the planning section of this document. With this section completed, instances will automatically be provisioned to execute jobs in Remote Docker or use the machine executor. To use the Windows machine executor you will need to build an image. For more information on VM Service and creating custom AMIs for remote Docker and machine executor jobs, read our VM service guide.

    You can preallocate instances to always be up and running, reducing the time taken for Remote Docker and machine executor jobs to start. If preallocation is set, a cron job will cycle through your preallocated instances once per day to prevent them getting into a bad/dead state.

    If Docker Layer Caching (DLC) is to be used, VM preallocation should be set to 0, forcing containers to be spun up on-demand for both machine and Remote Docker. It is worth noting here that if these fields are not set to 0 but all preallocated instances are in use, DLC will work correctly, as if preallocation was set to 0.

  12. AWS Cloudwatch or Datadog Metrics can be configured for your installation. Set either of these up in the relevant sections. For more information read our Monitoring guidance:

    AWS Cloudwatch and Datadog metrics
    Figure 10. Metrics

  13. Custom Metrics are an alternative to Cloudwatch and Datadog metrics, you can also customize the metrics you receive through Telegraf. For more on this read our Custom Metics guide.

  14. Distributed Tracing is used in our support bundles, and settings should remain set to default unless a change is requested by CircleCI Support.

  15. Artifacts persist data after a job is completed, and may be used for longer-term storage of your build process outputs. By default, CircleCI server v2.x only allows approved types to be served. This is to protect users from uploading, and potentially executing malicious content. The Artifacts setting allows you to override this protection. For more information on safe/unsafe types read our Build Artifacts guidance.

  16. After agreeing to the License Agreement and saving your settings, select Restart Now from the popup. You will then be redirected to start CircleCI and view the Management Console Dashboard. It will take a few minutes to download all of the necessary Docker containers.

If the Management Console reports Failure reported from operator: no such image click Start again and it should continue.

Validate Your Installation

  1. When the application is started, select Open to launch CircleCI in your browser, and sign up/log in to your CircleCI installation and start running 2.0 builds! You will become the Administrator at this point as you are the first person to sign in. Have a look at our Getting Started guide to start adding projects.

    CircleCI server dashboard
    Figure 11. Start CircleCI from your Dashboard

  2. After build containers have started and images have been downloaded, the first build should begin immediately. If there are no updates after around 15 minutes, and you have clicked the Refresh button, contact CircleCI support for assistance.

  3. Next, use our realitycheck repo to check basic CircleCI functionality.

  4. If you’re unable to run your first builds successfully please start with our Troubleshooting guide for general troubleshooting topics, and our Introduction to Nomad Cluster Operation for information about how to check the status of Builders in your installation.


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

Creative Commons License
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

6,000 Minutes Free

CircleCI, now with more build minutes
than anyone else for FREE 🎉

Sign up for Free

These teams get it

CCI customer teams