Before upgrading

See the CircleCI server 4.5 release notes and upgrade guide for this release.

Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s new in Release 4.5.0

The v4.5 release introduces security and performance improvements.

New features

• You can now restrict access to contexts using expression restrictions. Read more in the docs.
• You can now configure the Nomad Autoscaler evaluation_interval and cooldown.

Changes

• Very large and complex configuration files, notably those using a high number of pipeline parameters, will now have a shorter compile time.
• Optimizations have been made in orb-service for a reduction in both storage space used and timeouts for very large configuration compilation.
• The login page has been replaced with a new one. When logging in to server now you will be redirected to /server-login. The banner from the login page has been removed.
• Selecting items in the navigation bars of the settings pages no longer triggers a full page reload.
• Project and personal access tokens are now hashed rather than being stored as plaintext.
• Obsolete caches have been removed leading to improve Workflows Conductor performance and reduced Redis usage.
• Required domain-service caching variables are now defined by default.
• Various changes have been made in the UI to standardize style and icon usage.
• CircleCI 1.0 notify feature will no longer function, outbound webhooks should be used in the place of the notify feature
• We are introducing parameter limits to improve responsiveness of the trigger pipeline API and limit potential abuse. The new limits will be as follows: 100 max entries, 128 maximum key length, 512 maximum value length.

Bug fixes

• Fixed a bug that was causing duplicate workflows showing in the UI.
• Fixed a bug that was causing config compliation checks to attempt to access an invalid subdomain.
• The self-hosted runners Create Resource Class button in the web app has been fixed.

New services

New services introduced with this release:

• ciam-gateway
• ciam
• web-ui-authentication

Database migrations

The following databases will run migrations when upgrading to this version:

• builds_service
• permissions
• runneradmin
• conductor_production

Known issues

• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• Customers installing Server to GKE should wait to upgrade until further notice. There is an issue preventing a successful upgrade. New installations are not impacted.

To learn more about Server 4.5 installation, migration, or operations please review our documentation.

CHANGES:

• Improved the Pipelines UI performance by extending the duration of caching for commit status checks in domain-service.

FIXES:

• Fixed a bug that caused the Delete Cache Contents button on the Docker Layer Caching (DLC) Project Settings page to return an error without deleting the cache contents.
• Fixed a bug that resulted in the failure to download previous test results for a job when using a pre-signed URL.

CHANGES:

• Improved the Pipelines UI performance by extending the duration of caching for commit status checks in domain-service.

FIXES:

• Fixed a bug that caused the Delete Cache Contents button on the Docker Layer Caching (DLC) Project Settings page to return an error without deleting the cache contents.
• Fixed a bug that resulted in the failure to download previous test results for a job when using a pre-signed URL.

Before upgrading

See the CircleCI server 4.4 release notes and upgrade guide for this release.

Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s new in Release 4.4.0

The v4.4 release introduces the availability of OIDC tokens in jobs. Improvements to runner support including access to machine runner 3.0, rerun with SSH for container runner (open preview), and the introduction of the self-hosted runner UI in the web app. New configuration options available in values.yml are detailed below.

New features

• CircleCI provides OpenID Connect ID (OIDC) tokens in environment variables. A job can be configured to use these tokens to access compatible cloud services without long-lived credentials being stored in CircleCI.
• Machine runner 3.0 for improved setup, usability, and maintainability.
• The ability to rerun job with SSH option for container runners is in open preview.
• The Self-hosted runner UI is now available in the CircleCI web app. This includes a UI for creating and managing your runner resource classes.

Changes

• You can now create and manage your own secrets for the following values: Nomad mTLS (CACertificate, certificate, and privateKey), CircleCI server license, and Vault token.
• You can now configure CPU and memory limits for all services in your values.yaml.
• nodeSelector, nodeAffinity and toleration are now globally configurable in your values.yaml.
• The runner-admin and context-service user’s access scope has been reduced for improved security. User passwords can now be rotated as required.
• The PostgreSQL image is now configured without LLVM to avoid errors with use of JIT.

New services

New services introduced with this release:

• oidc-service
• oidc-tasks-service
• web-ui-runners

Database migrations

The following databases will run migrations when upgrading to this version:

• branch_service
• domain
• permissions

Known issues

• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.4 installation, migration, or operations please review our documentation.

Before Upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s New in Release 4.3.1

CHANGES

• Removes non-configurable settings from values.yaml for Runner admin to reduce complexity
• Adds logic to check for suspended or deleted GitHub users before fetching config with OAuth tokens

KNOWN ISSUES

• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.3 installation, migration, or operations please see our documentation.

Fixes

• Fixed a bug that caused the Runners page to appear in the side navigation on the Projects page, which is unsupported on server at this time.
• Fixed a bug that caused the Docker Layer Caching (DLC) settings to appear in Project Settings, which is only supported in server 4.3+.
• Bumped Nomad Autoscaler to version 0.3.7 to resolve an upstream regression that broke the default AWS credential chain for Nomad.

Before upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s new in Release 4.3.0

The v4.3 release introduces upgraded execution architecture. Previously, VM service was responsible for creating virtual machines for machine and remote Docker jobs. This has been replaced with machine provisioner. The new architecture has improved efficiency and stability. This upgrade also brings server inline with CircleCI cloud execution, meaning server will benefit from future updates faster. Some specific benefits include:

• A new UI for step output, which is better at handling larger output and colour.
• New systems for processing job output (step data, caches, artifacts etc.) is much more efficient.
• Machine and remote Docker jobs no longer run through Nomad so compute costs are cut down significantly.
• You do not need to run a Nomad cluster if you just want to run machine (and/or remote Docker jobs) on your server instance.
• DLC is now configured to run through S3, which is cheaper than running on SSD volumes, and there is no longer a limit on the amount of DLC jobs that can be run in parallel. DLC stays more consistent with better pruning and is shared between all jobs in a project instead of each cache being different.
• VM service could fall back to another region/zone, but machine provisioner can handle gaps in availability through:
  • Distributing load across availability zones through round-robining
  • Sending a percentage of traffic to a different zone
  • Instance type fallbacks based on deficits and success

New features

• Server execution architecture has been upgraded from VM service to machine provisioner. For further information, see the Installation guide.
• Server now ships with a custom postgreSQL image which now supports pg_crypto, pg_partman, and pg_cron support extensions.
• Added a REPL toggle to turn on a background which deletes workflow data older than 1 year. This is off by default.
• Config policies are now fully supported on server. _compiled_ is now a reserved word in config policies, so that parameterized configuration is fully available to the config policies engine. For further information see the Config policies overview docs.
• The circleci env subst command is now supported. See the Environment variables docs for more information.
• Added support overriding non-CircleCI-managed images.

Changes

• Nomad Autoscaler has been updated to 0.3.7.
• Server no longer deploys the Prometheus chart.
• The artifact paths have changed to improve download performance. The new path will now be <server-domain>/output/job/<job-id>/artifacts/0/home/circleci/artifacts/<artifact-file>

New services

New services introduced with this release

• feature-flags
• runner-admin
• docker-provisioner
• execution-gateway
• machine-provisioner
• output
• step

Services removed with this release

• output-processing
• picard-dispatcher
• test-results
• vm-service

Database migrations

The following databases will run migrations when upgrading to this version

• distributor
• machineprovisioner - Newly Created
• permissions
• runneradmin - Newly Created
• conductor_production

Fixes

• Fixed a bug that caused the Insights page to crash if a user tried to view job resource usage, which is unsupported on server at this time.
• If a job is retried due to an infrastructure failure, it should no longer cause a workflow to get stuck
• Fixed a bug that would try and pull a project’s config with a suspended user’s token.

Known issues

• All pipelines must re-run from start due to architectural changes.
• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.2 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.2.1 & 4.1.5

Changes

• Introduced the ability to disable the org-level concurrency limit in distributor. For more information, see the docs.

Fixes

• Fixed a bug that caused the /v2/tasks and /v2/tasks/running endpoints on the runner API to return errors.
• Fixed a bug that caused the self-hosted runners page to appear in the side navigation, but this page is unsupported on server at this time.
• Fixed a bug that caused the Insights page to crash if a user tried to view job resource usage, which is is unsupported on server at this time.
• Fixed a bug where the NGINX Deployment replicas were hard-coded to 1 and did not respect the nginx.replicas value in the Helm values.yaml file.

Before Upgrading

See the CircleCI server 4.2 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

From server v4.2, CircleCI server auto-generates some secrets if they are not present in the Kubernetes namespace. If you are upgrading from v4.1.x to 4.2, and you have managed Kubernetes secrets yourself, you will need to follow the steps outlined in the CircleCI server 4.2 upgrade guide to ensure the system knows some secrets already exist.

What’s New in Release 4.2.0

New Features

• Config policies now available as a preview on server. Use config policies to create organization-level policies to impose rules and scopes to govern which configuration elements are required, allowed, not allowed etc. See the Config policies overview for guidance on setting up config policies. To use config policies CLI commands with server v4.2, append all commands shown in the docs with --policy-base-url https://<CIRCLECI-SERVER-DOMAIN>.com
• You can now find your current server version by running: curl http(s)://<CIRCLECI-SERVER-DOMAIN>/version
• The 2xlarge+ resource class (32 vCPUs, 64 GB RAM) is now available for use with the Linux VM execution environment

Changes

• Introduced the ability to disable the org-level concurrency limit in distributor. For more information, see the docs
• By default, Server 4.2 auto-generates some secrets. If you wish to manage these secrets yourself, you should create these secrets (if not done already) and ensure Helm is aware so that auto-generation is skipped. See the CircleCI server 4.2 upgrade guide for steps.
• RabbitMQ version upgraded from 8.6.4 to 11.15.1.

New Services

The following new services are introduced in this release.

• policy-service

Database Migrations

The following databases will run migrations when upgrading to this version

• builds_service
• cron_service_production
• domain
• insights
• permissions
• conductor_production

Fixes

• Patched various security vulnerabilities.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.2 installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.4

Changes

• Introduced the ability to disable the org-level concurrency limit in distributor. For more information, see the docs

Fixes

• Fixed a bug that caused potential notification failures for pipelines and workflows. The bug affected communication between the legacy-notifier and builds-service services.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.1 installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.3

Fixes

• Query parameters are now supported on the task-agent download endpoint.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.1 installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.7

Fixes

• When using CircleCI runner, a repository can now be directly cloned with the checkout step in .circleci/config.yml.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised (for example, by using a VPN into your VPC).
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.2

Changes

• We have updated our PostgreSQL password requirements to allow special characters (except single or double quotes) in addition to alphanumeric characters.

Fixes

• For runner installations on server, you can now directly clone a repository with just the checkout step in .circleci/config.yml.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.1 installation, migration, or operations please see our documentation.

Before upgrading

See the CircleCI server 4.0 release notes for upgrade notes for this release.

What’s new in release 4.0.4

Changes

• We have updated our PostgreSQL password requirements to allow special characters (except single or double quotes) in addition to alphanumeric characters.

Fixes

• For runner installations on server, you can now directly clone a repository with just the checkout step in .circleci/config.yml.

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.0 installation, migration, or operations please see our documentation.

Before upgrading

See the CircleCI server 4.0 release notes for upgrade notes for this release.

What’s new in release 4.0.3

Changes

• In the checkout step in a CircleCI configuration, GitHub host keys are appended to ~/.ssh/known_hosts. Due to a change to GitHub’s RSA SSH key, CircleCI have updated the appended key to match. If you are using GitHub.com, and your installation of server has the old host key hardcoded into any projects, you will also need to update these to match.

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.0 installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.6

Changes

• In the checkout step in a CircleCI configuration, GitHub host keys are appended to ~/.ssh/known_hosts. Due to a change to GitHub’s RSA SSH key, CircleCI have updated the appended key to match. If you are using GitHub.com, and your installation of server has the old host key hardcoded into any projects, you will also need to update these to match.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised (for example, by using a VPN into your VPC).
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.1

Changes

• In the checkout step in a CircleCI configuration, GitHub host keys are appended to ~/.ssh/known_hosts. Due to a change to GitHub’s RSA SSH key, CircleCI have updated the appended key to match. If you are using GitHub.com, and your installation of server has the old host key hardcoded into any projects, you will also need to update these to match.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.1 installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.0

New features

• Air-gapped installation is now supported on server v4.1.
• Self-signed certificates or private certificate authorities are now supported when using GitHub Enterprise.
• For new installations, Tink can be used as the encryption backend instead of Vault.

Changes

• The minimum Telegraf version is now set to 1.8.14.
• Nomad and Vault images are now stored in templates, rather than hardcoded in charts.
• The Nomad Terraform configurations now have the Nomad port hardcoded. Previously this needed to be entered manually.

Database Migrations

The following databases will run migrations when upgrading this version.

• builds_service
• branch_service
• contexts_service_production
• orbs
• orb_analytics
• permissions
• webhook_service
• conductor_production

Fixes

• The Trigger Pipeline button in the web app now works for forked PRs, and an error message will be visible if the trigger fails.
• Fixed a bug that was preventing forked PRs from building if both the fork and original were building on CircleCI.
• Fixed a bug that was causing intermittent status updates for jobs in GitHub.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.5

Fixes

• Patched some CVEs.
• Updated runner’s launch-agent to use a query string instead of a GET body.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised (for example, by using a VPN into your VPC).
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before upgrading

See the CircleCI server 4.x release notes for upgrade notes for this release.

What’s new in release 4.0.2

Changes

• Pin Bitnami Postgres Helm chart url

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.4

Fixes

• Fixed a bug that was causing delays and failures to update git commit statuses.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.3

Changes

• Anonymous statistics reporting for Kong is now set to disabled by default.

Fixes

• Fixed a bug that was preventing email notifications from being sent with the following setting enabled: Notifications > Email Notifications> Build Notifications > My Branches.
• Fixed a bug that was causing a “Failed to load webhooks” error on project Webhooks pages in the CircleCI web app.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before upgrading

See the CircleCI server 4.x release notes for upgrade notes for this release.

What’s new in release 4.0.1

Changes

• When migrating to server v4.x, Telegraf configuration data is now decoded and available in your values.yaml file.
• Anonymous statistics reporting for Kong is now set to disabled by default.
• TLS settings are now available for Postgres.

Fixes

• Fixed a bug that was preventing email notifications from being sent with the following setting enabled: Notifications > Email Notifications> Build Notifications > My Branches.
• Fixed a bug that prevented webhook-service from working when Vault is externalized.

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

CircleCI server v4.0.0 is now generally available. The feature set for this release are equal to server v3.x. The installation and upgrade processes for v4.x have changed significantly.

Server v4.x offers greatly improved security handling, installation, and update processes. Server v4.x is installed using helm charts and images that can be pulled ahead of time to comply with your security processes. Installation processes can also use artifacts pulled from customer-managed registries. Server v4.x makes improved use of Kubernetes secrets and removes the requirement to grant permissions and network access to third-party tools.

Before upgrading

See the CircleCI server 4.x release notes for upgrade notes for this release.

What’s new in release 4.0.0

New features

• Two new Arm resource classes are now available xlarge and 2xlarge. For more information on using Arm, see the Arm docs.
• CircleCI server v4.x images and helm charts are available for licenced users.

Fixes

• Fixed a bug that was causing a huge quantity of error logs in the distributer-internal pod.
• Prometheus is no longer included by default in CircleCI server v4.x.
• The default concurrency limit has been increased to avoid it being reached.
• Docker layer caching volumes are now deleted after 3 days if unused. Previously they were deleted after 7 days.
• kong is now placed behind NGINX to simplify loadbalancer setup.
• The loadbalancer has been renamed from circleci-traefik to circleci-proxy.
• Workflow configurations that are stored in PostgreSQL will migrated to the Object Storage.

Known issues

• Telegraf needs to have its configuration overwritten and base64 encoded.
• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Webhook-service may not work with an externalized Vault.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.2

Fixes

• Links and redirects within the CircleCI server web app now correctly send users to their server login page. Previously, these links were directing people to the CircleCI cloud login page.
• Improvements to the user experience when upgrading and migrating server versions.
• Root volumes created by VM service are now tagged with the custom tag set for instances in the KOTS admin console. This covers machine executor and remote Docker VMs.
• A bug was fixed that was presenting users who need to re-autorize with a 404.
• CircleCI cloud marketing items have been removed.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.1

Fixes

• Kong resource limits have been increased. The previous limits were affecting the stability of some server installations.
• The KOTS config has been updated to allow private loadbalancers when ACM is in use.
• Fixed a bug that was preventing distributor-external from running in certain environments.

Known Issues

• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.0

New Features

• Android images for the machine executor are now available. For more information, see the Using Android Images with the Machine Executor guide.
• The CircleCI Build Agent can now be pulled from a custom Docker registry if required. For more information contact customer support.
• You can now use Amazon Certificate Manager (ACM) to manage your TLS certificates. For more information see the installation docs.
• Kubernetes v1.16 - v1.23 are now supported.
• Workload Identities for installations on GCP are now supported. Workload Identities can be used as an alternative to static credentials. For more informations see the installation docs

Fixes

• Fixed a bug that was causing the jobs page and step output access to run very slow in the CircleCI web app.
• Fixed a bug that was causing the workflows conductor pod to crash and issues with StatsD metrics.

Known Issues

• Before installing v3.4.0 delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Config files will now be stored in the chosen object store (e.g., AWS S3, Google Cloud Storage or Minio). Users can run into errors accessing certain pipelines, if these config files are deleted due to retention policies (e.g., AWS S3 Lifecycle).

To learn more about Server 3.x installation, migration, or operations please see our documentation.

What’s New in Release 2.19.15

Notes

In March 2022 updates to CircleCI server v2.x will stop.

Fixes

• Removed out-dated truststore for root CAs.
• Patched various security vulnerabilities.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.14

Notes

In March 2022 updates to CircleCI server v2.x will stop. Support for customers using CircleCI server v2.x will continue.

• There has been a change to how you will checkout using tags in jobs. Branch tags will now be referenced by either CIRCLE_TAG or +refs/heads/tag/path. For example, git checkout --force "$CIRCLE_TAG" or git fetch --force origin '+refs/heads/master:refs/remotes/origin/master'

Fixes

• Security patches for the exim image
• Replaced deprecated GitHub API endpoint for teams

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.1

Fixes

• CircleCI runner can now be used when server is installed behind a proxy.
• Fixed a bug that was causing some intermittent broken dependency caches.
• Removed outdated cert trust stores from some images. They were found to be causing access issues to some server installations.
• Fixed a bug that was affecting runner updates.
• SSH reruns now work when server is installed behind a proxy.
• Replaced deprecated GitHub API endoint for teams.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.0

Features

• Nomad Autoscaler can now be used to scale Nomad clients. For more information see the execution environments installation docs.
• Webhooks are now available.
• The Insights dashboard is now available.
• IRSA (AWS) can now be used as an alternative to keys for object storage authentication.
• The email address from which build notifications are sent is now configurable from the KOTS admin console.
• We have replaced Traefik with Kong as our reverse proxy. However, in order to minimize disruption when upgrading, we chose not to rename the service used by Kong. Therefore, you will see a service named circleci-server-traefik, however, this service is actually for Kong.

Fixes

• Upgraded Python to v3 in Vault container.
• Docs improvements on using the shared VPC architecture in GCP.
• The JVM heap size has been updated so that output-processor can use up to 80% of the pod memory limit.

Known Issues

• Updating GitHub Enterprise with a modified Let’s Encrypt certificate is unsupported.
• Let’s Encrypt’s new root certificate isrgrootx1 is not trusted.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.

To learn more about Server 3.x installation, migration, or operations please see our documentation.