Changelog

Before upgrading

See the CircleCI server 4.5 release notes and upgrade guide for this release.

Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s new in Release 4.5.0

The v4.5 release introduces security and performance improvements.

New features

• You can now restrict access to contexts using expression restrictions. Read more in the docs.
• You can now configure the Nomad Autoscaler evaluation_interval and cooldown.

Changes

• Very large and complex configuration files, notably those using a high number of pipeline parameters, will now have a shorter compile time.
• Optimizations have been made in orb-service for a reduction in both storage space used and timeouts for very large configuration compilation.
• The login page has been replaced with a new one. When logging in to server now you will be redirected to /server-login. The banner from the login page has been removed.
• Selecting items in the navigation bars of the settings pages no longer triggers a full page reload.
• Project and personal access tokens are now hashed rather than being stored as plaintext.
• Obsolete caches have been removed leading to improve Workflows Conductor performance and reduced Redis usage.
• Required domain-service caching variables are now defined by default.
• Various changes have been made in the UI to standardize style and icon usage.
• CircleCI 1.0 notify feature will no longer function, outbound webhooks should be used in the place of the notify feature
• We are introducing parameter limits to improve responsiveness of the trigger pipeline API and limit potential abuse. The new limits will be as follows: 100 max entries, 128 maximum key length, 512 maximum value length.

Bug fixes

• Fixed a bug that was causing duplicate workflows showing in the UI.
• Fixed a bug that was causing config compliation checks to attempt to access an invalid subdomain.
• The self-hosted runners Create Resource Class button in the web app has been fixed.

New services

New services introduced with this release:

• ciam-gateway
• ciam
• web-ui-authentication

Database migrations

The following databases will run migrations when upgrading to this version:

• builds_service
• permissions
• runneradmin
• conductor_production

Known issues

• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• Customers installing Server to GKE should wait to upgrade until further notice. There is an issue preventing a successful upgrade. New installations are not impacted.

To learn more about Server 4.5 installation, migration, or operations please review our documentation.

Before upgrading

See the CircleCI server 4.4 release notes and upgrade guide for this release.

Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s new in Release 4.4.0

The v4.4 release introduces the availability of OIDC tokens in jobs. Improvements to runner support including access to machine runner 3.0, rerun with SSH for container runner (open preview), and the introduction of the self-hosted runner UI in the web app. New configuration options available in values.yml are detailed below.

New features

• CircleCI provides OpenID Connect ID (OIDC) tokens in environment variables. A job can be configured to use these tokens to access compatible cloud services without long-lived credentials being stored in CircleCI.
• Machine runner 3.0 for improved setup, usability, and maintainability.
• The ability to rerun job with SSH option for container runners is in open preview.
• The Self-hosted runner UI is now available in the CircleCI web app. This includes a UI for creating and managing your runner resource classes.

Changes

• You can now create and manage your own secrets for the following values: Nomad mTLS (CACertificate, certificate, and privateKey), CircleCI server license, and Vault token.
• You can now configure CPU and memory limits for all services in your values.yaml.
• nodeSelector, nodeAffinity and toleration are now globally configurable in your values.yaml.
• The runner-admin and context-service user’s access scope has been reduced for improved security. User passwords can now be rotated as required.
• The PostgreSQL image is now configured without LLVM to avoid errors with use of JIT.

New services

New services introduced with this release:

• oidc-service
• oidc-tasks-service
• web-ui-runners

Database migrations

The following databases will run migrations when upgrading to this version:

• branch_service
• domain
• permissions

Known issues

• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.4 installation, migration, or operations please review our documentation.

Fixes

• Fixed a bug that caused the Runners page to appear in the side navigation on the Projects page, which is unsupported on server at this time.
• Fixed a bug that caused the Docker Layer Caching (DLC) settings to appear in Project Settings, which is only supported in server 4.3+.
• Bumped Nomad Autoscaler to version 0.3.7 to resolve an upstream regression that broke the default AWS credential chain for Nomad.

Before upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s new in Release 4.3.0

The v4.3 release introduces upgraded execution architecture. Previously, VM service was responsible for creating virtual machines for machine and remote Docker jobs. This has been replaced with machine provisioner. The new architecture has improved efficiency and stability. This upgrade also brings server inline with CircleCI cloud execution, meaning server will benefit from future updates faster. Some specific benefits include:

• A new UI for step output, which is better at handling larger output and colour.
• New systems for processing job output (step data, caches, artifacts etc.) is much more efficient.
• Machine and remote Docker jobs no longer run through Nomad so compute costs are cut down significantly.
• You do not need to run a Nomad cluster if you just want to run machine (and/or remote Docker jobs) on your server instance.
• DLC is now configured to run through S3, which is cheaper than running on SSD volumes, and there is no longer a limit on the amount of DLC jobs that can be run in parallel. DLC stays more consistent with better pruning and is shared between all jobs in a project instead of each cache being different.
• VM service could fall back to another region/zone, but machine provisioner can handle gaps in availability through:
  • Distributing load across availability zones through round-robining
  • Sending a percentage of traffic to a different zone
  • Instance type fallbacks based on deficits and success

New features

• Server execution architecture has been upgraded from VM service to machine provisioner. For further information, see the Installation guide.
• Server now ships with a custom postgreSQL image which now supports pg_crypto, pg_partman, and pg_cron support extensions.
• Added a REPL toggle to turn on a background which deletes workflow data older than 1 year. This is off by default.
• Config policies are now fully supported on server. _compiled_ is now a reserved word in config policies, so that parameterized configuration is fully available to the config policies engine. For further information see the Config policies overview docs.
• The circleci env subst command is now supported. See the Environment variables docs for more information.
• Added support overriding non-CircleCI-managed images.

Changes

• Nomad Autoscaler has been updated to 0.3.7.
• Server no longer deploys the Prometheus chart.
• The artifact paths have changed to improve download performance. The new path will now be <server-domain>/output/job/<job-id>/artifacts/0/home/circleci/artifacts/<artifact-file>

New services

New services introduced with this release

• feature-flags
• runner-admin
• docker-provisioner
• execution-gateway
• machine-provisioner
• output
• step

Services removed with this release

• output-processing
• picard-dispatcher
• test-results
• vm-service

Database migrations

The following databases will run migrations when upgrading to this version

• distributor
• machineprovisioner - Newly Created
• permissions
• runneradmin - Newly Created
• conductor_production

Fixes

• Fixed a bug that caused the Insights page to crash if a user tried to view job resource usage, which is unsupported on server at this time.
• If a job is retried due to an infrastructure failure, it should no longer cause a workflow to get stuck
• Fixed a bug that would try and pull a project’s config with a suspended user’s token.

Known issues

• All pipelines must re-run from start due to architectural changes.
• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.0

New features

• Air-gapped installation is now supported on server v4.1.
• Self-signed certificates or private certificate authorities are now supported when using GitHub Enterprise.
• For new installations, Tink can be used as the encryption backend instead of Vault.

Changes

• The minimum Telegraf version is now set to 1.8.14.
• Nomad and Vault images are now stored in templates, rather than hardcoded in charts.
• The Nomad Terraform configurations now have the Nomad port hardcoded. Previously this needed to be entered manually.

Database Migrations

The following databases will run migrations when upgrading this version.

• builds_service
• branch_service
• contexts_service_production
• orbs
• orb_analytics
• permissions
• webhook_service
• conductor_production

Fixes

• The Trigger Pipeline button in the web app now works for forked PRs, and an error message will be visible if the trigger fails.
• Fixed a bug that was preventing forked PRs from building if both the fork and original were building on CircleCI.
• Fixed a bug that was causing intermittent status updates for jobs in GitHub.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.0

New Features

• Android images for the machine executor are now available. For more information, see the Using Android Images with the Machine Executor guide.
• The CircleCI Build Agent can now be pulled from a custom Docker registry if required. For more information contact customer support.
• You can now use Amazon Certificate Manager (ACM) to manage your TLS certificates. For more information see the installation docs.
• Kubernetes v1.16 - v1.23 are now supported.
• Workload Identities for installations on GCP are now supported. Workload Identities can be used as an alternative to static credentials. For more informations see the installation docs

Fixes

• Fixed a bug that was causing the jobs page and step output access to run very slow in the CircleCI web app.
• Fixed a bug that was causing the workflows conductor pod to crash and issues with StatsD metrics.

Known Issues

• Before installing v3.4.0 delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Config files will now be stored in the chosen object store (e.g., AWS S3, Google Cloud Storage or Minio). Users can run into errors accessing certain pipelines, if these config files are deleted due to retention policies (e.g., AWS S3 Lifecycle).

To learn more about Server 3.x installation, migration, or operations please see our documentation.

What’s New in Release 2.19.00

Requirements

• Before upgrading to Server v2.19 your Nomad launch configuration must be updated by following this guide. Server v2.19 uses Nomad version 0.9.3, so if you have externalized Nomad, contact your support engineer before upgrading.

New Features

• You can now customize resource classes for your installation to provide developers with CPU/RAM options for the Jobs they configure. For more information see our guide to customizing resource classes in Server v2.19.

• CircleCI Server installations on AWS can now be configured to work on GovCloud. Using AWS GovCloud enables you to meet a variety of regulatory requirements, such as FedRAMP, ITAR, DoD SRG, CJIS, and HIPAA. Read more about how CircleCI helps customers meet security and compliance needs here.

Upgrades and Fixes

• For v2.19 we have updated Nomad clusters from 0.5x to 0.9, reducing the accumulation of dead jobs and improving overall performance. For more information, see the Nomad changelog.

• The image used to run the RabbitMQ server has been updated to fix vulnerabilities.

• The Server-Mongo-Upgrader service has been removed from the Services machine. Server Mongo Upgrader was a migration service written to facilitate the upgrade from Mongo 3.2 to 3.6, which was done as a part of the CircleCI Server v2.15.0 release in October 2018. This means it is not possible to upgrade directly from a Server version pre v2.15.0 to v2.19.0.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.x until further notice. We have known issues around job step logging and docker commands customization that make the current 2.19.x releases incompatible with an HTTP proxy setup of CircleCI Server. This issue will be fixed in a future 2.19.x patch release and the ‘safe’ version will be flagged up clearly for affected customers.

What’s New in Release 2.17

Known Issues

There have been reports of our backpressure settings being too aggressive due to lack of nomad garbage collecting, as well as issues with the Vm Service overwhelming AWS API Limits. If you are on 2.17, you should upgrade to 2.17.1.

Required Changes

IMPORTANT We have upgraded Replicated to version 2.34.1 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

• When using Github Enterprise, operators will now need to provide CircleCI with a Github User Token in the Management Console Settings. This is required due to a change in how the Github Enterprise API allows us to verify Organizations. It is recommended that you use a shared ops account to generate this token. After upgrading to CircleCI Server v2.17, please follow these steps to enact this change:
  • Sign into your GitHub Enterprise instance.
  • Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.
  • Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes.
  • Copy the new token and paste it into the Management Console settings. (Github Integration > GitHub Enterprise Default API Token)
  • Save the settings in Replicated.

Notes and Best Practices

• Fill out this form to receive updates about CircleCI through email.
• It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2.18 it will become required. See docs for our recommendation(s).
• We have updated our software packages to the following versions. This is not a required update for those with externalized environments at this time, but will be when v2.18 is released.
  • Vault
    • 1.1.2
  • Mongo
    • 3.6.12-xenial
  • Redis
    • 4.0.14
• We are removing the 1.0 Single-Box options from CircleCI 2.0. We found a few critical vulnerabilities in our 1.0 build image, and we have long stopped recommending it for trials. If this is absolutely critical to your workflow please reach out to us. This does not impact people who are running 1.0 in clustered mode.

Features/Improvements

• Workflows now has a Slack Integration! Users can choose to receive Slack notifications when their workflows complete.
• Administrators can now restrict what organizations are allowed into their CircleCI installation. For more details on how to enable this feature, please see the User Management Section of the 2.17 Ops Manual.
• We changed the renamed org flow so orgs that have been renamed will no longer result in errors going forward. Existing users who had applied workarounds for this use case will now no longer require said workarounds.
• Workflows now take up less DB space and will be easier to maintain going forward.
• Improved the cache in front of GraphQL API resulting in overall improved performance.
• Added backpressure to avoid overwhelming nomad with requests, this will result in increased performance from existing nomad clusters.

• New machine executor AMIs based on Ubuntu 16.04 for AWS.

• Ubuntu 16.04 with Docker 18.06.3 has apt-daily and apt-daily-upgrade services disabled.
  • It is highly recommended that customers try to experiment with the AMIs below before officially switching:
  • The new images are as follows
    • Ap-northeast-1:ami-0e49af0659db9fc5d
    • Ap-northeast-2:ami-03e485694bc2da249
    • Ap-south-1:ami-050370e57dfc6574a
    • Ap-southeast-1:ami-0a75ff7b28897268c
    • Ap-southeast-2:ami-072b1b45245549586
    • Ca-central-1:ami-0e44086f0f518ad2d
    • Eu-central-1:ami-09cbcfe446101b4ea
    • Eu-west-1:ami-0d1cbc2cc3075510a
    • Eu-west-2:ami-0bd22dcdc30fa260b
    • Sa-east-1:ami-038596d5a4fc9893b
    • Us-east-1:ami-0843ca047684abe87
    • Us-east-2:ami-03d60a35576647f63
    • Us-west-1:ami-06f6efb13d9ccf93d
    • us-west-2:ami-0b5b8ad02f405a909
  • They are replacing:
    • Ap-northeast-1:ami-cbe000ad
    • Ap-northeast-2:ami-629b420c
    • Ap-south-1:ami-97bac2f8
    • Ap-southeast-1:ami-63b22100
    • Ap-southeast-2:ami-dd6c73be
    • Ca-central-1:ami-d02c93b4
    • Eu-central-1:ami-b243eedd
    • Eu-west-1:ami-61de3718
    • Eu-west-2:ami-904e5ff4
    • Sa-east-1:ami-c22057ae
    • Us-east-1:ami-05b6e77e
    • Us-east-2:ami-9b4161fe
    • Us-west-1:ami-efc9e08f
    • Us-west-2:ami-ce8c94b7

Bug fixes

• Fixed some bugs related to GitHub API response handling and webhook handling.
• Fixed issue with Scheduled Workflows when the services machine is restarted.
• Fixed changing the RabbitMQ hostname for Scheduled Workflows when externalizing.
• You can no longer create contexts with no names. If you are using a context with no names, it will no longer be accessible from your execution environment.
• We have optimized our handling of large amounts of build output and test results XML, to avoid out-of-memory errors.
• The CIRCLE_PULL_REQUEST environment variable was not being populated correctly in all cases when building across forks. This has been fixed.
• Fixed a bug where some commits with [ci skip] in the message were still being built.
• Fixed a bug causing Workflows to get stuck when infrastructure_failure happens after a job fails.
• Fixed a bug causing duplicate docker networks on same nomad client (if running build using machine:true AND vm-provider=on_host).
• Improved performance when using local storage. Previously, caching issues had been experienced when local storage was used rather than the default option of using S3 (selecting None under Storage Driver options from the Management Console).
• We have added more error checking and validation around Github’s API so the existing list commit endpoint no longer causes issues.
• Datadog API token field was stored in plaintext, now set as a password field.
• Fixed issue where workflows were constrained from fanning out to large number of jobs.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.16

The following updates and fixes are also of note:

IMPORTANT We have upgraded Replicated to version 2.29.0 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

New Feature We are excited to announce that you can now distribute your data and workload external to the Services Machine. The following services can be externalized; MongoDB, Redis, Nomad Server, RabbitMQ, Postgres and Vault. Please contact your CSM for the latest documentation.

New Feature Custom Metrics can now be accomplished via a Telegraf Output Configuration File. See the Custom Metrics section of the Monitoring section. Note: The Custom Metrics section in the admin UI been disabled in favor of a configuration file.

New Feature Users can now receive email notifications about Workflows

New Feature The PostgreSQL image has been updated to allow modifying the default configuration by creating the following file: /etc/circleconfig/postgres/extra.conf A list of configuration options can be found here.

Updated Removing EOL Banner on build emails

Updated VM Service stability improvements

Bug Fix Fix issue with contexts that would break after 32 days

Bug Fix Vault auth tokens will now renew periodically to prevent expiration

Bug Fix Fix for an issue where workflow listing pages would intermittently fail to render

Bug Fix We have fixed a small number of bugs affecting processing incoming web hooks from GitHub to CircleCI.

Bug Fix For installs that are using an HTTP or HTTPS proxy, the jobs will now ignore that proxy so that setup_remote_docker works.

Bug Fix Reduced the number of connections 1.0 builders make to the PostgreSQL database.

Security Fix Security fixes for potential cross-site scripting vulnerability and HTTP header injection vulnerability

Security Fix Forked PRs can no longer write the caches of parent projects by default for security reasons. It is possible to still write parent project caches from the fork if the “Pass secrets to builds from forked pull requests” (in Advanced settings) is enabled.

Update Improved metrics for VM provisioning for machine executor. This changed metric names, so if you are monitoring VM provisioning in your install already, you will need to reconfigure the monitoring dashboards for these new metrics.

• New metrics:
  • vm-service.gauges.available-vms and vm-service.gauges.running-vms replaced by vm-service.gauges.vms_by_status
  • vm-service.gauges.running-tasks replaced by vm-service.gauges.tasks_by_status
  • vm-service.gauges.oldest-unassigned-task replaced by vm-service.gauges.unassigned_tasks_age

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
In some instances the metrics section becomes hidden in the settings page. Please see this Knowledge article for instructions.

What’s New in Release 2.13*

• Fixed: Due to a certificate not being imported jobs were unable to submit after a container restart.
• Fixed: Stability issues within contexts service.
• Fixed: vm-service issue where AWS ECS and EBS volumes were left used. Added better idempotent client retries, API eventual consistency handling.
• Added: vm-service EBS Volumes cache evictor. Docker Layer Caching EBS volumes get deleted after 14 days of unused. You can control the inactivity period in Installation Admin Settings page.
• Added: Periodic checks to scale down any unused long running instances (e.g. due to reduction in prescaling configurations, or or due to a bug preventing termination of VMs in timely manner).
• Added: Optimizations to speed up provisioning. Resiliency to network/external changes.
• CircleCI is excited to announce that users on AWS can now forward host (Services VM) and docker metrics via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to AWS CloudWatch. This is is the first step into bringing you greater observability into your installation. To enable and configure please go tohttps://example.com/settings#metrics. In this release the following metrics may be enabled: Docker , Networking , Disk , Memory, and CPU. For more information please see the Monitoring documentation.
• CircleCI is enhancing visibility by capturing usage statistics of CircleCI installations. This aggregate data will be used to better support you and ensure a smooth transition off 1.0. This optional feature is automatically available when you upgrade to the latest version of your installation or once a support bundle is downloaded. Below are the metrics being captured. If you would like access to this data in order to better understand your users please contact a CircleCI account representative. Please reference exhibit C within your terms of contract and our standard license agreement for our complete security and privacy disclosures. For more information please see the Usage Statistics documentation.

Weekly Account Usage

Weekly Job Activity

* this release contains migrations and is a required upgrade