Changelog

What’s New in Release 2.19.15

Notes

In March 2022 updates to CircleCI server v2.x will stop.

Fixes

• Removed out-dated truststore for root CAs.
• Patched various security vulnerabilities.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.14

Notes

In March 2022 updates to CircleCI server v2.x will stop. Support for customers using CircleCI server v2.x will continue.

• There has been a change to how you will checkout using tags in jobs. Branch tags will now be referenced by either CIRCLE_TAG or +refs/heads/tag/path. For example, git checkout --force "$CIRCLE_TAG" or git fetch --force origin '+refs/heads/master:refs/remotes/origin/master'

Fixes

• Security patches for the exim image
• Replaced deprecated GitHub API endpoint for teams

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.1

Fixes

• CircleCI runner can now be used when server is installed behind a proxy.
• Fixed a bug that was causing some intermittent broken dependency caches.
• Removed outdated cert trust stores from some images. They were found to be causing access issues to some server installations.
• Fixed a bug that was affecting runner updates.
• SSH reruns now work when server is installed behind a proxy.
• Replaced deprecated GitHub API endoint for teams.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.0

Features

• Nomad Autoscaler can now be used to scale Nomad clients. For more information see the execution environments installation docs.
• Webhooks are now available.
• The Insights dashboard is now available.
• IRSA (AWS) can now be used as an alternative to keys for object storage authentication.
• The email address from which build notifications are sent is now configurable from the KOTS admin console.
• We have replaced Traefik with Kong as our reverse proxy. However, in order to minimize disruption when upgrading, we chose not to rename the service used by Kong. Therefore, you will see a service named circleci-server-traefik, however, this service is actually for Kong.

Fixes

• Upgraded Python to v3 in Vault container.
• Docs improvements on using the shared VPC architecture in GCP.
• The JVM heap size has been updated so that output-processor can use up to 80% of the pod memory limit.

Known Issues

• Updating GitHub Enterprise with a modified Let’s Encrypt certificate is unsupported.
• Let’s Encrypt’s new root certificate isrgrootx1 is not trusted.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

What’s New in Release 2.19.13

Notes

In March 2022 updates to CircleCI server v2.x will stop. Support for customers using CircleCI server v2.x will continue.

Fixes

• Various security and vulnerability patches.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.2

Notes

• The rerun workflow endpoint now returns workflow ID rather than the message accepted.

Fixes

• TLS is terminated outside of frontend so the SSL server has been completely removed from the frontend container.
• Moved the default certificate logic from KOTS to helm.
• Fixed the build agent image version used in server v3.x. The image used previously was causing problems with runner.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.1

New Features

• Private VMs are now supported for installations on GCP.

Fixes

• mTLS is now disabled by default.
• SSH timeout for VMs has been increased to 10 minutes.
• Private VMs now request private IPs.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.0

New Features

• Customers who require a fully private installation can now access a setting in the KOTS admin console to ensure public IPs are not assigned to VMs. Note that with this non-public IP setting enabled, a work-around will be needed if SSH access to running jobs is required, for example, by using a VPN into your VPC.
• Customers that manage outbound traffic through a proxy can now configure proxy settings through the KOTS admin console. Please see our documentation for specifics on proxy support for server.
• We have expanded the machine execution environment options available to include additional resource classes, sizes, and executors. You now have access to Arm (medium, large), Linux (medium, large, X large, and XX large), and Windows (medium, large, XX large) resource classes.
• The insights API is now available to all server customers. Leverage build and other data to better understand the performance of teams and the health of your build and testing efforts.
• We have revamped the admin UI, and updated our installation instructions, making it easier to set up and manage server.
• You can now supply a custom Linux AMI for VM service.
• SSL termination can now be disabled - If you have put server login behind a firewall, this will enable SSL termination at the firewall.
• You can now control the size of persistent volumes. For larger customers, the initial persistent volume size was too small, by default. You can now set this at install time, providing an easier migration for those customers that require it. For further information see the Internal Database Volume Expansion doc.
• We have added an auto-scaling example to the nomad client terraform.
• You can now choose to serve ‘unsafe’ build artifacts. Previously this option was hidden, meaning potentially unsafe artifacts were rendered as plain text. For more information see the Build Artifacts doc.

Fixes

• The default windows executor was not as documented, we have increased the size to align with documentation and cloud.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

IMPORTANT: With this release, the frontend-external load balancer has been removed. The traefik load balancer now handles all incoming traffic. When updating from a previous server 3.x version, you will need to update the DNS record that was pointing to the frontend-external load balancer and have it point to the circleci-server-traefik load balancer instead. Remember, you can retrieve the external IP address or DNS name of your traefik load balancer by typing kubectl get svc/circleci-server-traefik in a terminal that has access to the cluster.

For further information see the What’s New doc.

What’s New in Release 3.1.0

New Features

• Telegraf plugins can now be added to server and customized to use third party monitoring solutions, for example, Datadog. For more information, see the Metrics and Monitoring doc.
• The option to use only private load balancers has been introduced for customers who want a fully private installation. For more information see the Load Balancers guide.
• Server 3.x hosts build artifacts, test results, and other state in object storage. We support any S3-compatible storage and Google Cloud Storage. For more information, see the Installation guide for further information.
• Dynamic config via setup workflows is now available on server installations. For more information see our blog post and the Dynamic Configuration docs page.
• Runner is now available on server. For further information, including installation steps, see the Runner docs. Runner allows the use of the macOS executor in server installations and VM service functionality for customers with server installed in a private data centre.
• The frontend load balancer from v3.0 has been removed and replaced with an Ingress resource and the Traefik Ingress controller. This is a breaking change requiring you to reconfigure your DNS. See the What’s New in server docs for further information and guidance.
• The following services can now be externalized. For setup information, see the server v3.x installation guide:
  • Postgres
  • MongoDB
  • Vault
• Backup and restore functionality is now available. For more information see the Backup and Restore guide.
• Prometheus is now deployed by default with server to monitor your cluster health and usage. Prometheus can be managed and configured from the KOTS admin UI. For further information, see the Metrics and Monitoring doc.
• Server now supports the 2XL resource class. The Nomad cluster needs to be made large enough to account for larger resource classes.
• The lifecycle of build artifacts and test results can now be configured from the KOTS admin console under Storage Object Expiry, including the option to disable the expiration and retain artifacts and test results indefinitely.

Fixes

• Resolved a collection of bugs that were causing sensitive information to be leaked into CircleCI support bundles:
  • Instances of faulty and partial redactions of secrets were detected, in part due to 3rd party bugs.
  • PostgresDB leaking sensitive information to STDOUT.
  • Several CircleCI services were logging secrets.
• Tightened network security in the Nomad terraform module.
• Terraform v0.15.0 and up are now supported.
• Updated installation scripts to use functions supported by most recent versions of Terraform.
• Resolved a bug that was leading to machine large builds being run on the wrong machine type. Machine large builds now correctly use 4 vCPUs and 16GB of RAM.
• Resolved a bug that caused contexts-service to fail on expiration of Vault client tokens.
• Resolved a bug that was causing legacy-notifier to report readiness prematurely.
• The JVM heap size parameter has been removed for all services. The heap size is set to be half of the memory limit.
• Changes to networking config and certs are now picked up automatically by Traefik. Previously, a restart would have been required.
• Minimum requirements for CPU and memory have changed. For the new values, see the Installation Prerequisites doc.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

What’s New in Release 3.0.0

Server 3.0 is now generally available. The newest version of server offers the ability to scale under heavy workloads, all within your own Kubernetes cluster and private network, while still enjoying the full CircleCI cloud experience. Server 3.0 includes the latest CircleCI features, such as orbs, scheduled workflows, matrix jobs, and more. For existing customers interested in migrating from 2.19 to 3.x, contact your customer success manager. Server 3.0 will receive monthly patch releases and quarterly feature releases.

New Features

• New UI
• Orbs for Server
• Pipelines
• Config 2.1
• API 2.0
• Scheduled Workflows
• Matrix Jobs
• And much more…

Known Issues

• No support for external data stores (Postgres, Mongo, Vault). This feature will be implemented in a future release.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior. This will be fixed in a future patch release.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• CircleCI currently assigns a public load balancer for the frontend services. For some customers, their infrastructure or security groups won’t allow this. We will provide an optional internal local balancer for the frontend services in a future release.
• Telegraf metrics collection customization is not yet available.
• Support for GPU builders is not yet available.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

What’s New in Release 2.19.04

Fixes

• Fixed a bug that was leading to support bundle creation timing out before the Services machine logs had been created, leaving only Replicated logs.

• S3 connection pool metrics under circle.s3.connection_pool.* have been added to the test results service, making it easier to debug issues with this service.

• Add in missing environment variables for the workflows service. The absence of these environment variables was causing excessive stack tracing whenever workflows were run. This in turn lead to excessive log rotation.

• Fixed a bug that was causing failure to update GitHub statuses. Some customers experienced this bug when a project had a follower with a broken auth token.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.03

Before Upgrading

If you are using an IAM role scoped to a non-root path, you will need to unset the OUTPUT_PROCESSOR_USE_NAIVE_ROLE_MAPPING environment variable in your output processor customization script. See the Customizations Guide in our documentation for more information on using customization scripts.

Fixes

• Removed the use of the depecated GitHub.com API endpoint GET applications/%s/tokens/%s.

• Distributed tracing is now enabled by default for Server installations. Traces are used in CircleCI support bundles to improve our ability to troubleshoot Server issues. Options for the tracing sampling rate are displayed in the Replicated Management Console, but should only be changed from the default if requested by CircleCI Support.

• Fixed an issue that was preventing restore_cache from working with the storage driver set to “none” - i.e not S3.

• Fixed an issue that was preventing the output_processor service from using AWS AssumeRole when the role was located in a subfolder. This issue affected customers with security policies forcing the use of a subfolder in this case, and the symptoms included the inability to store artifacts or use timings-based test splitting.

• JVM heap size can now be changed using the JVM_HEAP_SIZE environment variable for the following services: vm-service, domain-service, permissions-service and federations-service.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.02

Fixes

• In the LDAP login flow we now use an anonymous form to POST LDAP auth state, rather than sending it as a GET parameter. Previously, when a user authenticated using LDAP, their username and password were sent in plaintext as part of a query parameter in a GET request. As requests are over HTTPS, this left usernames and passwords in request logs, etc. This issue is now fixed.

• Optimizely and Zendesk are now removed from Server release images.

• Fixed an issue in which setting CIRCLE_ADMIN_SERVER_HTTP_THREADS or CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS too high would prevent the frontend container from starting.

• Due to changes in the GitHub API we have removed the use of ?client_id=x&client_secret=y for GitHub, and GHE versions 2.17 and later.

• Fixed an issue that was causing intermittent failures to spin up VMs with DLC in use.

• Fixed an issue that was preventing the customization of proxy settings for Docker containers. See the Nomad Client Proxy and Service Configuration Overrides guides for more infomation.

• Fixed a bug that was preventing job steps for non-failing builds being logged when proxy settings were used for the job container.

• Removed legacy TLS versions 1.0 and 1.1, in addition, enabled 1.2 and 1.3 TLS, and specified the following ciphersuites
  • ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
• Fixed a statsd configuration issue that meant some services were not emitting Telegraf metrics.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client
• Classic Load Balancer is no longer available from this version due to the ciphersuite changes listed above. CircleCI no longer accepts requests from Classic Load Balancer, so you should move to Network Load Balancer (NLB) or Application Load Balancer (ALB).

What’s New in Release 2.19.01

Fixes

• Fixed a bug that was preventing some customers from upgrading due to a schema change in one of our library dependencies.

• Fixed a bug that was preventing some customers from inspecting builds via SSH due to a logic change in our build agent.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.01, instead upgrade to at least v2.19.02. We have known issues around job step logging and docker commands customization that makes the v2.19.01 release incompatible with an HTTP proxy setup of CircleCI Server.

• If any changes have been made to your networking configuration you may need to run the following steps to ensure you can use SSH to inspect your builds:

  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.00

Requirements

• Before upgrading to Server v2.19 your Nomad launch configuration must be updated by following this guide. Server v2.19 uses Nomad version 0.9.3, so if you have externalized Nomad, contact your support engineer before upgrading.

New Features

• You can now customize resource classes for your installation to provide developers with CPU/RAM options for the Jobs they configure. For more information see our guide to customizing resource classes in Server v2.19.

• CircleCI Server installations on AWS can now be configured to work on GovCloud. Using AWS GovCloud enables you to meet a variety of regulatory requirements, such as FedRAMP, ITAR, DoD SRG, CJIS, and HIPAA. Read more about how CircleCI helps customers meet security and compliance needs here.

Upgrades and Fixes

• For v2.19 we have updated Nomad clusters from 0.5x to 0.9, reducing the accumulation of dead jobs and improving overall performance. For more information, see the Nomad changelog.

• The image used to run the RabbitMQ server has been updated to fix vulnerabilities.

• The Server-Mongo-Upgrader service has been removed from the Services machine. Server Mongo Upgrader was a migration service written to facilitate the upgrade from Mongo 3.2 to 3.6, which was done as a part of the CircleCI Server v2.15.0 release in October 2018. This means it is not possible to upgrade directly from a Server version pre v2.15.0 to v2.19.0.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.x until further notice. We have known issues around job step logging and docker commands customization that make the current 2.19.x releases incompatible with an HTTP proxy setup of CircleCI Server. This issue will be fixed in a future 2.19.x patch release and the ‘safe’ version will be flagged up clearly for affected customers.

What’s New in Release 2.18.03

Features

Following our release of Windows support on the hosted offering of CircleCI, running VM-based jobs in Windows environments is now supported on CircleCI Server instances running in AWS.

The version of Windows currently supported is Windows Server 2019. Check out the Customizing VM Service Images doc for the details of how to build your own custom Windows Server 2019 image, and then follow the Server examples in the Getting Started with Windows doc for instructions on how to set up a Windows job on your CircleCI Server installation.

Deprecation

• Server-Mongo-Upgrader will be removed in Server 2.19.0. Only effects customers running < 2.15.0.

What’s New in Release 2.18

Upgrading your installation

Our upgrade guide takes you step by step through the process of upgrading to the latest version of CircleCI Server, including any checks and updates required before starting the upgrade process.

Note: If at any time your organization name has been changed, there is a script that must be run before starting the upgrade process.

Notes and Best Practices

• We now require a minimum 32GB of RAM for the Services Machine.

• We have made some changes to our Redis configuration. If you have externalized Redis then you’ll need to update your configuration. Please contact your Customer Success Manager.

• We have also made changes to our Postgres version and require at least postgreSQL v9.5.16. If you have externalized postgreSQL then please update to at least that version in 2.17.x before upgrading to 2.18.

Known Issues

Nomad garbage collection only runs every 4 hours. This could prevent nomad jobs from running in environments with heavy usage. If you hit this bug please follow this work around:

1. Create /etc/circleconfig/schedulerer/customizations on the services machine
2. Inside /etc/circleconfig/schedulerer/customizations enter the following:

   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_PENDING_JOBS=500 
   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_DEAD_JOBS=100000
   

3. Restart the picard-scheduler container:

   sudo docker restart picard-scheduler
   

Features/Improvements

• It is now possible to restrict environment variables at run time by adding security groups to contexts. Security groups are defined as GitHub teams or LDAP groups. After a security group is added to a context, only CircleCI users who are members of that security group may access or use the environment variables of the context. For more information see our guide to restricting a context.

• Customers running storage drivers external to AWS will see improved routing times when searching for build Artifacts.

• You can now customize the metrics that get output from CircleCI. For steps and options, see our Custom Metrics guide. Below is a short list of metrics that are included by default when enabling Custom Metrics:

• You can now provide individual AMIs for both Remote Docker and machine executor jobs. Previously we provided the option for a single custom AMI to be used across both, but with v2.18, this expanded customization gives you greater control over versioning and dependencies to meet your individual CICD needs. See our VM Service guide for more information.

Bug fixes

• Additional fixes around contexts and org renames.

• Fixed an issue where occasionally volumes would fail to attach to spun up Remote docker/machine instances.

• Fixed an issue where the CircleCI integration could not be installed on JIRA instances with the jira.com subdomain.

• Fixed an issue where the Workflows page would still point to an old repo after renaming an organization.

• Fixed issue where the Workflows UI would fail to refresh data automatically.

• Improved context loading times in cases when they could cause timeouts in the UI.

• Fixed an issue where contexts would cause builds to return CIRCLE_BUG.

What’s New in Release 2.17

Known Issues

There have been reports of our backpressure settings being too aggressive due to lack of nomad garbage collecting, as well as issues with the Vm Service overwhelming AWS API Limits. If you are on 2.17, you should upgrade to 2.17.1.

Required Changes

IMPORTANT We have upgraded Replicated to version 2.34.1 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

• When using Github Enterprise, operators will now need to provide CircleCI with a Github User Token in the Management Console Settings. This is required due to a change in how the Github Enterprise API allows us to verify Organizations. It is recommended that you use a shared ops account to generate this token. After upgrading to CircleCI Server v2.17, please follow these steps to enact this change:
  • Sign into your GitHub Enterprise instance.
  • Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.
  • Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes.
  • Copy the new token and paste it into the Management Console settings. (Github Integration > GitHub Enterprise Default API Token)
  • Save the settings in Replicated.

Notes and Best Practices

• Fill out this form to receive updates about CircleCI through email.
• It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2.18 it will become required. See docs for our recommendation(s).
• We have updated our software packages to the following versions. This is not a required update for those with externalized environments at this time, but will be when v2.18 is released.
  • Vault
    • 1.1.2
  • Mongo
    • 3.6.12-xenial
  • Redis
    • 4.0.14
• We are removing the 1.0 Single-Box options from CircleCI 2.0. We found a few critical vulnerabilities in our 1.0 build image, and we have long stopped recommending it for trials. If this is absolutely critical to your workflow please reach out to us. This does not impact people who are running 1.0 in clustered mode.

Features/Improvements

• Workflows now has a Slack Integration! Users can choose to receive Slack notifications when their workflows complete.
• Administrators can now restrict what organizations are allowed into their CircleCI installation. For more details on how to enable this feature, please see the User Management Section of the 2.17 Ops Manual.
• We changed the renamed org flow so orgs that have been renamed will no longer result in errors going forward. Existing users who had applied workarounds for this use case will now no longer require said workarounds.
• Workflows now take up less DB space and will be easier to maintain going forward.
• Improved the cache in front of GraphQL API resulting in overall improved performance.
• Added backpressure to avoid overwhelming nomad with requests, this will result in increased performance from existing nomad clusters.

• New machine executor AMIs based on Ubuntu 16.04 for AWS.

• Ubuntu 16.04 with Docker 18.06.3 has apt-daily and apt-daily-upgrade services disabled.
  • It is highly recommended that customers try to experiment with the AMIs below before officially switching:
  • The new images are as follows
    • Ap-northeast-1:ami-0e49af0659db9fc5d
    • Ap-northeast-2:ami-03e485694bc2da249
    • Ap-south-1:ami-050370e57dfc6574a
    • Ap-southeast-1:ami-0a75ff7b28897268c
    • Ap-southeast-2:ami-072b1b45245549586
    • Ca-central-1:ami-0e44086f0f518ad2d
    • Eu-central-1:ami-09cbcfe446101b4ea
    • Eu-west-1:ami-0d1cbc2cc3075510a
    • Eu-west-2:ami-0bd22dcdc30fa260b
    • Sa-east-1:ami-038596d5a4fc9893b
    • Us-east-1:ami-0843ca047684abe87
    • Us-east-2:ami-03d60a35576647f63
    • Us-west-1:ami-06f6efb13d9ccf93d
    • us-west-2:ami-0b5b8ad02f405a909
  • They are replacing:
    • Ap-northeast-1:ami-cbe000ad
    • Ap-northeast-2:ami-629b420c
    • Ap-south-1:ami-97bac2f8
    • Ap-southeast-1:ami-63b22100
    • Ap-southeast-2:ami-dd6c73be
    • Ca-central-1:ami-d02c93b4
    • Eu-central-1:ami-b243eedd
    • Eu-west-1:ami-61de3718
    • Eu-west-2:ami-904e5ff4
    • Sa-east-1:ami-c22057ae
    • Us-east-1:ami-05b6e77e
    • Us-east-2:ami-9b4161fe
    • Us-west-1:ami-efc9e08f
    • Us-west-2:ami-ce8c94b7

Bug fixes

• Fixed some bugs related to GitHub API response handling and webhook handling.
• Fixed issue with Scheduled Workflows when the services machine is restarted.
• Fixed changing the RabbitMQ hostname for Scheduled Workflows when externalizing.
• You can no longer create contexts with no names. If you are using a context with no names, it will no longer be accessible from your execution environment.
• We have optimized our handling of large amounts of build output and test results XML, to avoid out-of-memory errors.
• The CIRCLE_PULL_REQUEST environment variable was not being populated correctly in all cases when building across forks. This has been fixed.
• Fixed a bug where some commits with [ci skip] in the message were still being built.
• Fixed a bug causing Workflows to get stuck when infrastructure_failure happens after a job fails.
• Fixed a bug causing duplicate docker networks on same nomad client (if running build using machine:true AND vm-provider=on_host).
• Improved performance when using local storage. Previously, caching issues had been experienced when local storage was used rather than the default option of using S3 (selecting None under Storage Driver options from the Management Console).
• We have added more error checking and validation around Github’s API so the existing list commit endpoint no longer causes issues.
• Datadog API token field was stored in plaintext, now set as a password field.
• Fixed issue where workflows were constrained from fanning out to large number of jobs.

Fill out this form to receive updates about CircleCI through email.

What’s Fixed in Release 2.16.1

• Bug Fix: This release includes a new flag to disable GitHub cache warming. Note: Only add this flag if a high thread count is impacting the API service. To disable cache warming after completing the upgrade, complete the following steps:

1. Add the export DOMAIN_SERVICE_REFRESH_USERS=false flag to the /etc/circleconfig/api-service/customizations file on the Services machine.

2. Restart CircleCI by going to the Administrative console and clicking Stop Now, waiting for it to stop, then clicking Start.

• Bug Fix: Fixed issue preventing metrics from domain service from being forwarded.

• Security Fix: The SMTP password field in the dashboard was set to plain text. The password field will now be hidden.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.16

The following updates and fixes are also of note:

IMPORTANT We have upgraded Replicated to version 2.29.0 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

New Feature We are excited to announce that you can now distribute your data and workload external to the Services Machine. The following services can be externalized; MongoDB, Redis, Nomad Server, RabbitMQ, Postgres and Vault. Please contact your CSM for the latest documentation.

New Feature Custom Metrics can now be accomplished via a Telegraf Output Configuration File. See the Custom Metrics section of the Monitoring section. Note: The Custom Metrics section in the admin UI been disabled in favor of a configuration file.

New Feature Users can now receive email notifications about Workflows

New Feature The PostgreSQL image has been updated to allow modifying the default configuration by creating the following file: /etc/circleconfig/postgres/extra.conf A list of configuration options can be found here.

Updated Removing EOL Banner on build emails

Updated VM Service stability improvements

Bug Fix Fix issue with contexts that would break after 32 days

Bug Fix Vault auth tokens will now renew periodically to prevent expiration

Bug Fix Fix for an issue where workflow listing pages would intermittently fail to render

Bug Fix We have fixed a small number of bugs affecting processing incoming web hooks from GitHub to CircleCI.

Bug Fix For installs that are using an HTTP or HTTPS proxy, the jobs will now ignore that proxy so that setup_remote_docker works.

Bug Fix Reduced the number of connections 1.0 builders make to the PostgreSQL database.

Security Fix Security fixes for potential cross-site scripting vulnerability and HTTP header injection vulnerability

Security Fix Forked PRs can no longer write the caches of parent projects by default for security reasons. It is possible to still write parent project caches from the fork if the “Pass secrets to builds from forked pull requests” (in Advanced settings) is enabled.

Update Improved metrics for VM provisioning for machine executor. This changed metric names, so if you are monitoring VM provisioning in your install already, you will need to reconfigure the monitoring dashboards for these new metrics.

• New metrics:
  • vm-service.gauges.available-vms and vm-service.gauges.running-vms replaced by vm-service.gauges.vms_by_status
  • vm-service.gauges.running-tasks replaced by vm-service.gauges.tasks_by_status
  • vm-service.gauges.oldest-unassigned-task replaced by vm-service.gauges.unassigned_tasks_age

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
In some instances the metrics section becomes hidden in the settings page. Please see this Knowledge article for instructions.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.15

This release contains an upgrade to MongoDB, service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

MongoDB Upgrade The MongoDB version running on Services VM will be automatically updated from 3.2 to 3.6 for you. If you have externalized MongoDB and wish to upgrade to 3.6, please contact your CSM

Bug Fix Restrict interaction of Workflows ‘Rerun’ dropdown to users who are part of GitHub organization for open-source projects.

Bug Fix Known issue for broken UI due to workflows with “.” in the name

Bug Fix Known performance issue with freezed browsers for organizations with many branches.

Bug Fix Known issue for Vault auth tokens not being automatically renewed. If you have externalized Vault, please contact your CSM before you upgrade to 2.15.0

New Feature: Splitting test files via Workflows. If your project has a large number of tests, you can use circleci tests split feature to reduce your job run time. For more information on how to setup tests split, please follow this document.

New Feature: Workflows UI will now show actors who have taken the following actions:

• Rerun workflows
• Approved jobs
• Canceled jobs

New Feature We are excited to announce more Metrics! Nomad job metrics can be enabled and will be emitted by the Nomad Server. Please see Nomad Metrics documentation for instructions on how to configure Nomad Clients. The following five types of metrics are reported:

• circle.nomad.server_agent.poll_failure: Returns 1 if the last poll of the Nomad agent failed, otherwise it returns 0.
• circle.nomad.server_agent.jobs.pending: Returns the total number of pending jobs across the cluster.
• circle.nomad.server_agent.jobs.running: Returns the total number of running jobs across the cluster.
• circle.nomad.server_agent.jobs.complete: Returns the total number of complete jobs across the cluster.
• circle.nomad.server_agent.jobs.dead: Returns the total number of dead jobs across the cluster.

New Feature Host (Services VM) and docker metrics can now be forwarded via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to DataDog or a custom location. To enable and/or configure please go tohttps://example.com/settings#metrics. For more information please see the Monitoring documentation.

Performance Improvement Reduced the load time for all workflows pages.

Performance Improvement CircleCI has upgraded the mechanism that returns gettimeofday calls. As a result, heavy users of syscall will see an aggregate improvement in performance. Most noticeably this will improve the run time of projects with heavy usage of debugging/profiling tools and Xdebug with PHP.

Updated JVM_HEAP_SIZE is configurable for frontend, output-processor, and test-result containers. Please see JVM Heap Size Configuration documentation.

Updated For customers running the Static installation, Ubuntu 16.04 (Xenial) has been added as a supported OS. For additional details please see the docs.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

What’s New in Release 2.14

This release contains service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

Fixed: Pre-allocated VMs not being used. Customers reported that jobs using machine executor and setup_remote_docker were continuing to spin up (and terminate) new VMs at runtime instead of the pre-allocated pool.

Fixed: Unable to build in AWS region eu-west-2.

Updated: The title of the Builds page in the CircleCI application was changed to Jobs in several places to better reflect the structure of builds.

Updated: Contexts are now checked for unique naming across your installation of CircleCI instead of just within an organization. You will no longer be able to create duplicate context names within an account. Existing duplicates will be honored by the system, but should be changed to avoid future collisions. Please see the Contexts documentation for further information.

Updated: The test-results services cleans up the /tmp directory on restart and the JVM heap size increased from 256m to 512m.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

Migration Tools: The 1.0 migration center was updated with additional content and helpful tools to migrate projects from CircleCI 1.0 to 2.0.

What’s New in Release 2.13*

• Fixed: Due to a certificate not being imported jobs were unable to submit after a container restart.
• Fixed: Stability issues within contexts service.
• Fixed: vm-service issue where AWS ECS and EBS volumes were left used. Added better idempotent client retries, API eventual consistency handling.
• Added: vm-service EBS Volumes cache evictor. Docker Layer Caching EBS volumes get deleted after 14 days of unused. You can control the inactivity period in Installation Admin Settings page.
• Added: Periodic checks to scale down any unused long running instances (e.g. due to reduction in prescaling configurations, or or due to a bug preventing termination of VMs in timely manner).
• Added: Optimizations to speed up provisioning. Resiliency to network/external changes.
• CircleCI is excited to announce that users on AWS can now forward host (Services VM) and docker metrics via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to AWS CloudWatch. This is is the first step into bringing you greater observability into your installation. To enable and configure please go tohttps://example.com/settings#metrics. In this release the following metrics may be enabled: Docker , Networking , Disk , Memory, and CPU. For more information please see the Monitoring documentation.
• CircleCI is enhancing visibility by capturing usage statistics of CircleCI installations. This aggregate data will be used to better support you and ensure a smooth transition off 1.0. This optional feature is automatically available when you upgrade to the latest version of your installation or once a support bundle is downloaded. Below are the metrics being captured. If you would like access to this data in order to better understand your users please contact a CircleCI account representative. Please reference exhibit C within your terms of contract and our standard license agreement for our complete security and privacy disclosures. For more information please see the Usage Statistics documentation.

Weekly Account Usage

Weekly Job Activity

* this release contains migrations and is a required upgrade

The installable version of CircleCI Server 2.9.0 includes the following improvements and bug fixes:

• Resolution for a potential LDAP injection vulnerability
• Addition of support for the following LDAP fields:
  • Group Membership
  • Group Object Classes

See the Authentication document for the steps.

• Addition of default values for OpenLDAP and Active Directory
• Stability and bug fixes for an issue with test results
• Improvements to ready agent logging for support bundles
• Stability fixes and speed improvements for provisioning remote Docker engine
• Reliability improvements for the permissions service client
• Differentiation between the LDAP login and email to remove email as the assumed login value
• Fix for a bug that deselected all projects when a new user logged in for the first time
• Fix for a bug that was preventing Workflows from working correctly when located behind a proxy
• Bug fixes and improvements for Contexts

The installable version of CircleCI 2.8.0 includes the following fixes and improvements:

• An issue involving pre-allocated remote docker, machine VM creation, and other VM usage on AWS when the on-host VM option is selected has been fixed.

• An issue with the VM provider timing out when waiting for SSH to connect has been fixed. This upgrade supersedes any custom values for the Number of SSH attempts for starting VM setting and adds an equivalent field named Timeout for waiting for SSH to connect to VM(s) that you must set.

The installable version of CircleCI 2.6.0 includes the following fixes and improvements:

• It is now possible to create multiple Contexts in the Organization Settings of the CircleCI app. See the Contexts documentation for details and naming conventions.

The installable version of CircleCI 2.5.0 includes the following fixes and improvements:

• This release fixes a problem with DNS resolution failure on alpine-based containers. This problem was observed in the 2.3 release and was fixed for some service containers in 2.4. The 2.5 update fixes the issue for all service containers.
• This release includes improvements to scaling and garbage collection in the VM Service that should improve performance for installations with heavy use of machine executors or remote Docker.