Before upgrading

See the CircleCI server 4.4 release notes and upgrade guide for this release.

Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s new in Release 4.4.0

The v4.4 release introduces the availability of OIDC tokens in jobs. Improvements to runner support including access to machine runner 3.0, rerun with SSH for container runner (open preview), and the introduction of the self-hosted runner UI in the web app. New configuration options available in values.yml are detailed below.

New features

• CircleCI provides OpenID Connect ID (OIDC) tokens in environment variables. A job can be configured to use these tokens to access compatible cloud services without long-lived credentials being stored in CircleCI.
• Machine runner 3.0 for improved setup, usability, and maintainability.
• The ability to rerun job with SSH option for container runners is in open preview.
• The Self-hosted runner UI is now available in the CircleCI web app. This includes a UI for creating and managing your runner resource classes.

Changes

• You can now create and manage your own secrets for the following values: Nomad mTLS (CACertificate, certificate, and privateKey), CircleCI server license, and Vault token.
• You can now configure CPU and memory limits for all services in your values.yaml.
• nodeSelector, nodeAffinity and toleration are now globally configurable in your values.yaml.
• The runner-admin and context-service user’s access scope has been reduced for improved security. User passwords can now be rotated as required.
• The PostgreSQL image is now configured without LLVM to avoid errors with use of JIT.

New services

New services introduced with this release:

• oidc-service
• oidc-tasks-service
• web-ui-runners

Database migrations

The following databases will run migrations when upgrading to this version:

• branch_service
• domain
• permissions

Known issues

• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.4 installation, migration, or operations please review our documentation.

Our mission at CircleCI is to manage change so you can innovate faster. In 2023 we released 50 new features and feature enhancements to help you do just that. CircleCI shipped 3.8x more releases this year compared to the average of our competitors. By the numbers, that’s twice as many CI/CD enhancements as GitHub or GitLab and orders of magnitude more than AWS, Bitbucket, Travis, or Bitrise.

From security enhancements to productivity tools to guardrails to keep your business safe, no CI/CD company works harder to keep your team working.

Security releases in 2023

This year we invested heavily in releases that strengthen your ability to stay secure in an ever-changing threat climate. Highlights include:

• We onboarded additional resources to strengthen our 24/7 security monitoring, alerting and incident response capabilities.
• Our OIDC functionality expanded continually over the year, including:
  • An enhanced subject claim and a new SSH rerun claim allow customers to write granular access policies for tokens
  • A newly customizable audience claim opened up new integration opportunities
  • Tokens become available during pipeline spinup enabling the use of OIDC to pull from private image repos
  • A dedicated OIDC for ECR command enables ECR connections with only 2 lines of config.
• We released tightened scoping for secrets (project environment variables based on create and updated dates) as well as the ability to scope secrets down to individual projects.
• We made available additional APIs to support secret rotation.
• We released config policies:
  • CLI & API which allows org admins to manage org-wide controls for resource class usage, allowed orbs, and fine-tuned context access
  • Config policy CLI/UI allows developers to review company policies, test config changes for compliance, and resolve violations (GA in Cloud and in Server 4.3+)
• And we added additional functionality to our audit logs, including:
  • New audit log events to help customers identify and respond to incidents (project API token creation, project env var creation and deletion, project settings)
  • Updates on SSH key creation and deletion context accessed in pipeline
  • A new self-serve UI for downloading audit logs via the web app

Productivity releases in 2023

In addition to security enhancements, we also released dozens of new features to help you build better and faster.

• Release environments give you visibility into your Kubernetes deployments from within the CircleCI web app. This feature is currently in open preview, and more deploy and release functionality is coming in the new year.
• Major improvements to the CircleCI VS Code extension adapt it to track any type of CircleCI project, no matter the VCS, and to support multi-project VS Code workspaces. It now offers an improved pipeline debugging experience through in-IDE test output reporting, job logs visualization, and re-running of jobs with SSH in one click. Since December, it also lets users test out config changes easily by triggering pipelines directly from VS Code, based on their local config files.
• A new UI for steps in the CircleCI web app makes step URLs clickable with more legible data output. The new UI is more robust and removes previous display limits for step data.
• We pushed numerous enhancements to container runner, improving the functionality of self-hosted runners installed in your Kubernetes cluster.
• We released M1 for Mac, offering performant resources with GPU access and allowing for full end-to-end testing. Additionally, M1 provides a powerful platform for your AI/ML projects.
• We enabled Arm support for the Docker executor.
• New GPU resource classes were released for customers on Scale plans, with the option of an AI add-on plan for added performance on MLOps projects.
• We began offering support for GitLab self-managed instances.
• We released the ability to rerun failed tests only, saving time and credits by only re-running tests that have failed.
• You can now leverage an AI-powered smart assist in generating config files, which can save time and simplify complex configuration tasks.
• Inbound webhooks allow you to trigger pipeline runs from a change in any source, including non-VCS sources.
• Error summarization speeds up debugging by AI-contextualization of, and solutions to, common errors.
• Server 4.x added support for industry standard Helm install and upgrades. With Server 4.3, a major change in the execution architecture expands the reliability and robustness of the platform.

Support improvements in 2023

• Investments in support resources and documentation began allowing customers to self-serve for straightforward issues like billing and linking/unlinking VCS accounts.
• The release of AI-powered error summarization as well as documentation improvements also kept customers in flow and more productive this year.
• These releases enabled the support team to focus on resolving more complex issues. Median time to resolution actually increased by 28% year-over-year, and this is a good thing!

Platform reliability improvements in 2023

Our work on our platform’s reliability has also paid dividends this year. After investments in incident tooling and processes, and re-architecting key parts of our platform, we’ve been able to meet our goal of increasing our resilience to keep you building at all times (read more about our reliability work here). In short, over the course of 2023:

• Critical path uptime went from 99.3% –> 99.9+% uptime
• Average monthly downtime reduced from 5hrs –> 43 minutes
• Number of long (60m+) incidents down by 50%

We know how vital the velocity, efficiency, and continuity of the software development lifecycle is to any business, and we thank you for trusting us to be a part of yours. We’re looking forward to the new capabilities our platform will enable for your team in 2024!

You can now delete GitHub App & inbound webhook triggers from Project Settings –> Triggers.

This release provides critical security fixes for Go:

• Patched with Go v1.21.5 for critical security fixes.
• Go release announcement

For upgrade instructions, review the Release Agent documentation.

Critical security fix

• Patched with Go v1.21.5 for critical security fixes in Go. Go release announcement: https://groups.google.com/g/golang-announce/c/iLGK3x6yuNo

Enhancements to error logging and event handling:

• Now logging the outcome of the task Pod’s PreStop hook

• Clearer and actionable error messaging when the task Pod fails to startup or its liveness probe fails

General improvements

• The garbage collector period is now configurable for tuning how quickly failed Pods are removed

• Ability to customize individual values (e.g., the timeoutSeconds) on the task Pod’s primary container’s liveness probe. Any overrides are merged with the defaults, like a kubectl patch.

• See runner configuration documentation for all configuration options

Status badges display in a README a project’s build status.

This functionality now works as expected for projects that integrate with GitLab & GitHub App.

Fixes

• Fixed a bug that caused the Runners page to appear in the side navigation on the Projects page, which is unsupported on server at this time.
• Fixed a bug that caused the Docker Layer Caching (DLC) settings to appear in Project Settings, which is only supported in server 4.3+.
• Bumped Nomad Autoscaler to version 0.3.7 to resolve an upstream regression that broke the default AWS credential chain for Nomad.

CircleCI’s “Linux CUDA” image has been updated to include the Pytorch, Tensorflow, and HuggingFace transformers libraries out-of-the-box.

Users will no longer need to waste time pulling these large libraries into their execution environment when a job is running. This image is currently only available to jobs that use the gen2 GPU resource class.

To use this new image, specify:

linux-cuda-12:default or linux-cuda-12:edge in the image stanza.

Customers can now use a screen reader to read the output of individual steps of a job. The toggle is off by default.

Before upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s new in Release 4.3.0

The v4.3 release introduces upgraded execution architecture. Previously, VM service was responsible for creating virtual machines for machine and remote Docker jobs. This has been replaced with machine provisioner. The new architecture has improved efficiency and stability. This upgrade also brings server inline with CircleCI cloud execution, meaning server will benefit from future updates faster. Some specific benefits include:

• A new UI for step output, which is better at handling larger output and colour.
• New systems for processing job output (step data, caches, artifacts etc.) is much more efficient.
• Machine and remote Docker jobs no longer run through Nomad so compute costs are cut down significantly.
• You do not need to run a Nomad cluster if you just want to run machine (and/or remote Docker jobs) on your server instance.
• DLC is now configured to run through S3, which is cheaper than running on SSD volumes, and there is no longer a limit on the amount of DLC jobs that can be run in parallel. DLC stays more consistent with better pruning and is shared between all jobs in a project instead of each cache being different.
• VM service could fall back to another region/zone, but machine provisioner can handle gaps in availability through:
  • Distributing load across availability zones through round-robining
  • Sending a percentage of traffic to a different zone
  • Instance type fallbacks based on deficits and success

New features

• Server execution architecture has been upgraded from VM service to machine provisioner. For further information, see the Installation guide.
• Server now ships with a custom postgreSQL image which now supports pg_crypto, pg_partman, and pg_cron support extensions.
• Added a REPL toggle to turn on a background which deletes workflow data older than 1 year. This is off by default.
• Config policies are now fully supported on server. _compiled_ is now a reserved word in config policies, so that parameterized configuration is fully available to the config policies engine. For further information see the Config policies overview docs.
• The circleci env subst command is now supported. See the Environment variables docs for more information.
• Added support overriding non-CircleCI-managed images.

Changes

• Nomad Autoscaler has been updated to 0.3.7.
• Server no longer deploys the Prometheus chart.
• The artifact paths have changed to improve download performance. The new path will now be <server-domain>/output/job/<job-id>/artifacts/0/home/circleci/artifacts/<artifact-file>

New services

New services introduced with this release

• feature-flags
• runner-admin
• docker-provisioner
• execution-gateway
• machine-provisioner
• output
• step

Services removed with this release

• output-processing
• picard-dispatcher
• test-results
• vm-service

Database migrations

The following databases will run migrations when upgrading to this version

• distributor
• machineprovisioner - Newly Created
• permissions
• runneradmin - Newly Created
• conductor_production

Fixes

• Fixed a bug that caused the Insights page to crash if a user tried to view job resource usage, which is unsupported on server at this time.
• If a job is retried due to an infrastructure failure, it should no longer cause a workflow to get stuck
• Fixed a bug that would try and pull a project’s config with a suspended user’s token.

Known issues

• All pipelines must re-run from start due to architectural changes.
• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

The CircleCI web app now shows how much time is saved on each job that uses the “rerun failed tests” feature.

As of August 25, 2023 all projects now have an enhanced experience for viewing steps through the CircleCI UI. Benefits include:

• Vastly improved performance and rendering quality
• No cap on log output
• Terminal control codes are now processed in the UI
• URLs are clickable from the step output

More details are available in our Discuss post.

What’s New

CircleCI’s CLI now includes an env subst command, which serves as a wrapper around the popular envsubst CLI tool. The command is used to substitute environment variables in a text string or file.

circleci env subst is available in both the local CLI and inside all CircleCI jobs (coming to CircleCI server v4.2+). More information and usage examples can be found in the docs as well as in this Discuss post.

What’s New

We now offer a way for you to immediately delete a DLC cache’s contents. You can do so via the CircleCI web UI, or the CLI. Refer to this support article for details.

When using Docker layer caching (DLC), a project’s cache can occasionally become corrupted, leading to failed jobs. Previously, the only solution was to stop using DLC, wait 3 days for the cache to become invalid, then turn DLC back on.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.0

New features

• Air-gapped installation is now supported on server v4.1.
• Self-signed certificates or private certificate authorities are now supported when using GitHub Enterprise.
• For new installations, Tink can be used as the encryption backend instead of Vault.

Changes

• The minimum Telegraf version is now set to 1.8.14.
• Nomad and Vault images are now stored in templates, rather than hardcoded in charts.
• The Nomad Terraform configurations now have the Nomad port hardcoded. Previously this needed to be entered manually.

Database Migrations

The following databases will run migrations when upgrading this version.

• builds_service
• branch_service
• contexts_service_production
• orbs
• orb_analytics
• permissions
• webhook_service
• conductor_production

Fixes

• The Trigger Pipeline button in the web app now works for forked PRs, and an error message will be visible if the trigger fails.
• Fixed a bug that was preventing forked PRs from building if both the fork and original were building on CircleCI.
• Fixed a bug that was causing intermittent status updates for jobs in GitHub.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

What’s New

The job status badges in the CircleCI UI now include a badge for “Infrastructure Failure”. This badge indicates that CircleCI has run into issues with the underlying infrastructure on which the job runs.

Previously, it was difficult for users to discern whether a job failed due to an infrastructure failure, or another cause such as a test failure. This change allows for more transparency about what happens with the execution of jobs.

Refer to the Discuss post for an example.

What’s New

We have added out-of-the-box shell support for Node and Ruby. Invoking Node and Ruby commands within CircleCI jobs can now be as simple as the following:

steps:
  - run:
    name: Check Node shell
    shell: node
    command: console.log("node")

Usage examples in other languages can be found in our documentation.