Description:

For projects that are integrated with GitHub App, if the branch name contains the word “tags”, we now correctly identify the commits from the branch as “push” commits as opposed to “tag-triggered” builds.

Before: The getAllInsightsBranches API returned org-id and project-id as null.

Now: The getAllInsightsBranches API returns the actual org-id and project-id.

Before upgrading

See the CircleCI server 4.4 release notes and upgrade guide for this release.

Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s new in Release 4.4.0

The v4.4 release introduces the availability of OIDC tokens in jobs. Improvements to runner support including access to machine runner 3.0, rerun with SSH for container runner (open preview), and the introduction of the self-hosted runner UI in the web app. New configuration options available in values.yml are detailed below.

New features

• CircleCI provides OpenID Connect ID (OIDC) tokens in environment variables. A job can be configured to use these tokens to access compatible cloud services without long-lived credentials being stored in CircleCI.
• Machine runner 3.0 for improved setup, usability, and maintainability.
• The ability to rerun job with SSH option for container runners is in open preview.
• The Self-hosted runner UI is now available in the CircleCI web app. This includes a UI for creating and managing your runner resource classes.

Changes

• You can now create and manage your own secrets for the following values: Nomad mTLS (CACertificate, certificate, and privateKey), CircleCI server license, and Vault token.
• You can now configure CPU and memory limits for all services in your values.yaml.
• nodeSelector, nodeAffinity and toleration are now globally configurable in your values.yaml.
• The runner-admin and context-service user’s access scope has been reduced for improved security. User passwords can now be rotated as required.
• The PostgreSQL image is now configured without LLVM to avoid errors with use of JIT.

New services

New services introduced with this release:

• oidc-service
• oidc-tasks-service
• web-ui-runners

Database migrations

The following databases will run migrations when upgrading to this version:

• branch_service
• domain
• permissions

Known issues

• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.4 installation, migration, or operations please review our documentation.

CircleCI’s Visual Studio Code extension now lets customers trigger pipelines from VS Code based on local, unversioned configuration files. Users can choose to run a full pipeline, or select only the individual jobs and workflows they want to run by using checkboxes. This will shorten the feedback cycle and enable developers to iterate fast on their configuration, without having to commit their trial and error changes to the VCS.

This feature is controlled by an org-level opt-in setting, and an opt-out project level setting.

Check out the feature docs for more information.

The “Tests tab” has been updated so that the success/failure state is indicated by an icon in addition to the color red or green. Previously, the only way to tell if tests failed or succeeded on the “Tests tab” in CircleCI UI was the color.

Our mission at CircleCI is to manage change so you can innovate faster. In 2023 we released 50 new features and feature enhancements to help you do just that. CircleCI shipped 3.8x more releases this year compared to the average of our competitors. By the numbers, that’s twice as many CI/CD enhancements as GitHub or GitLab and orders of magnitude more than AWS, Bitbucket, Travis, or Bitrise.

From security enhancements to productivity tools to guardrails to keep your business safe, no CI/CD company works harder to keep your team working.

Security releases in 2023

This year we invested heavily in releases that strengthen your ability to stay secure in an ever-changing threat climate. Highlights include:

• We onboarded additional resources to strengthen our 24/7 security monitoring, alerting and incident response capabilities.
• Our OIDC functionality expanded continually over the year, including:
  • An enhanced subject claim and a new SSH rerun claim allow customers to write granular access policies for tokens
  • A newly customizable audience claim opened up new integration opportunities
  • Tokens become available during pipeline spinup enabling the use of OIDC to pull from private image repos
  • A dedicated OIDC for ECR command enables ECR connections with only 2 lines of config.
• We released tightened scoping for secrets (project environment variables based on create and updated dates) as well as the ability to scope secrets down to individual projects.
• We made available additional APIs to support secret rotation.
• We released config policies:
  • CLI & API which allows org admins to manage org-wide controls for resource class usage, allowed orbs, and fine-tuned context access
  • Config policy CLI/UI allows developers to review company policies, test config changes for compliance, and resolve violations (GA in Cloud and in Server 4.3+)
• And we added additional functionality to our audit logs, including:
  • New audit log events to help customers identify and respond to incidents (project API token creation, project env var creation and deletion, project settings)
  • Updates on SSH key creation and deletion context accessed in pipeline
  • A new self-serve UI for downloading audit logs via the web app

Productivity releases in 2023

In addition to security enhancements, we also released dozens of new features to help you build better and faster.

• Release environments give you visibility into your Kubernetes deployments from within the CircleCI web app. This feature is currently in open preview, and more deploy and release functionality is coming in the new year.
• Major improvements to the CircleCI VS Code extension adapt it to track any type of CircleCI project, no matter the VCS, and to support multi-project VS Code workspaces. It now offers an improved pipeline debugging experience through in-IDE test output reporting, job logs visualization, and re-running of jobs with SSH in one click. Since December, it also lets users test out config changes easily by triggering pipelines directly from VS Code, based on their local config files.
• A new UI for steps in the CircleCI web app makes step URLs clickable with more legible data output. The new UI is more robust and removes previous display limits for step data.
• We pushed numerous enhancements to container runner, improving the functionality of self-hosted runners installed in your Kubernetes cluster.
• We released M1 for Mac, offering performant resources with GPU access and allowing for full end-to-end testing. Additionally, M1 provides a powerful platform for your AI/ML projects.
• We enabled Arm support for the Docker executor.
• New GPU resource classes were released for customers on Scale plans, with the option of an AI add-on plan for added performance on MLOps projects.
• We began offering support for GitLab self-managed instances.
• We released the ability to rerun failed tests only, saving time and credits by only re-running tests that have failed.
• You can now leverage an AI-powered smart assist in generating config files, which can save time and simplify complex configuration tasks.
• Inbound webhooks allow you to trigger pipeline runs from a change in any source, including non-VCS sources.
• Error summarization speeds up debugging by AI-contextualization of, and solutions to, common errors.
• Server 4.x added support for industry standard Helm install and upgrades. With Server 4.3, a major change in the execution architecture expands the reliability and robustness of the platform.

Support improvements in 2023

• Investments in support resources and documentation began allowing customers to self-serve for straightforward issues like billing and linking/unlinking VCS accounts.
• The release of AI-powered error summarization as well as documentation improvements also kept customers in flow and more productive this year.
• These releases enabled the support team to focus on resolving more complex issues. Median time to resolution actually increased by 28% year-over-year, and this is a good thing!

Platform reliability improvements in 2023

Our work on our platform’s reliability has also paid dividends this year. After investments in incident tooling and processes, and re-architecting key parts of our platform, we’ve been able to meet our goal of increasing our resilience to keep you building at all times (read more about our reliability work here). In short, over the course of 2023:

• Critical path uptime went from 99.3% –> 99.9+% uptime
• Average monthly downtime reduced from 5hrs –> 43 minutes
• Number of long (60m+) incidents down by 50%

We know how vital the velocity, efficiency, and continuity of the software development lifecycle is to any business, and we thank you for trusting us to be a part of yours. We’re looking forward to the new capabilities our platform will enable for your team in 2024!

You can now delete GitHub App & inbound webhook triggers from Project Settings –> Triggers.

This release provides critical security fixes for Go:

• Patched with Go v1.21.5 for critical security fixes.
• Go release announcement

For upgrade instructions, review the Release Agent documentation.

Critical security fix

• Patched with Go v1.21.5 for critical security fixes in Go. Go release announcement: https://groups.google.com/g/golang-announce/c/iLGK3x6yuNo

Enhancements to error logging and event handling:

• Now logging the outcome of the task Pod’s PreStop hook

• Clearer and actionable error messaging when the task Pod fails to startup or its liveness probe fails

General improvements

• The garbage collector period is now configurable for tuning how quickly failed Pods are removed

• Ability to customize individual values (e.g., the timeoutSeconds) on the task Pod’s primary container’s liveness probe. Any overrides are merged with the defaults, like a kubectl patch.

• See runner configuration documentation for all configuration options

Before Upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s New in Release 4.3.1

CHANGES

• Removes non-configurable settings from values.yaml for Runner admin to reduce complexity
• Adds logic to check for suspended or deleted GitHub users before fetching config with OAuth tokens

KNOWN ISSUES

• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.3 installation, migration, or operations please see our documentation.

Container Runner SSH Rerun (Preview)

Container runner now supports SSH Rerun. Visit our documentation on setup and usage.

For more information on SSH rerun please see the following documentation.

Status badges display in a README a project’s build status.

This functionality now works as expected for projects that integrate with GitLab & GitHub App.

Fixes

• Fixed a bug that caused the Runners page to appear in the side navigation on the Projects page, which is unsupported on server at this time.
• Fixed a bug that caused the Docker Layer Caching (DLC) settings to appear in Project Settings, which is only supported in server 4.3+.
• Bumped Nomad Autoscaler to version 0.3.7 to resolve an upstream regression that broke the default AWS credential chain for Nomad.

Users can now orchestrate their model deployments across multiple environments through the Amazon SageMaker Orb. They can also track and manage their deployments from within the Releases UI.

See our documentation for more information.

Inbound Webhooks are now available in open preview.

With this new feature, you can trigger a pipeline from anywhere that emits a webhook or run a curl command. With this change, we also updated pages such as the Dashboard to accurately reflect that pipelines can now come from outside your VCS.

Only available to users authenticated through our GitHub App integration. See our community forum and an AI use case for more.

CircleCI’s “Linux CUDA” image has been updated to include the Pytorch, Tensorflow, and HuggingFace transformers libraries out-of-the-box.

Users will no longer need to waste time pulling these large libraries into their execution environment when a job is running. This image is currently only available to jobs that use the gen2 GPU resource class.

To use this new image, specify:

linux-cuda-12:default or linux-cuda-12:edge in the image stanza.

Now in preview, three new resource classes that use GPUs:

1. gpu.nvidia.small.gen2: uses 1 NVIDIA Tesla A10G GPU (4 CPU / 16 GB RAM 200 credits/min)
2. gpu.nvidia.small.multi: uses 2 NVIDIA Tesla T4 GPUs (4 CPU / 16 GB RAM 160 credits/min)
3. gpu.nvidia.medium.multi: uses 4 NVIDIA Tesla T4 GPUs (8 CPU / 30 GB RAM / 240 credits/min)

Only available to users on our Scale pricing plan. See our community forum for more.

Customers can now use a screen reader to read the output of individual steps of a job. The toggle is off by default.

Machine runner 3.0 is now Generally Available, providing improvements to usability, setup, and maintenance of machine runners.

Container runners and machine runners are now built using the same framework. Launch agent users will need to upgrade to the new application, but existing container runner installations will not require any additional action.

Faster feature enhancements: We’ll now bring many network resilience feature enhancements to both types of self-hosted runners (machine and container).

Improved usability: With more uniform logs and metrics, CircleCI’s machine runners ensure consistent and reliable performance and a seamless experience for users regardless of their runner setup.

Setup and maintenance: CircleCI’s machine runner 3.0 simplifies the setup and maintenance process by offering convenient packages that allow you to easily integrate with your preferred cloud templating tools.

Linux, macOS, and Windows support is available. See all the details on the runner documentation.

Machine Runner 1.1 variants will be supported through March of 2024, giving you time to plan a migration.

View our documentation to upgrade.

CircleCI is upgrading the Docker version that is used for jobs that use the Docker executor in November, 2023. This change is critical to ensuring the underlying infrastructure that runs your jobs continues to be secure, reliable, and performant.

This upgrade should be effectively invisible to customers, no action is needed. The upgrade is from Docker 20.10.23 to Docker 24.

See our community forum for more details.

Before upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s new in Release 4.3.0

The v4.3 release introduces upgraded execution architecture. Previously, VM service was responsible for creating virtual machines for machine and remote Docker jobs. This has been replaced with machine provisioner. The new architecture has improved efficiency and stability. This upgrade also brings server inline with CircleCI cloud execution, meaning server will benefit from future updates faster. Some specific benefits include:

• A new UI for step output, which is better at handling larger output and colour.
• New systems for processing job output (step data, caches, artifacts etc.) is much more efficient.
• Machine and remote Docker jobs no longer run through Nomad so compute costs are cut down significantly.
• You do not need to run a Nomad cluster if you just want to run machine (and/or remote Docker jobs) on your server instance.
• DLC is now configured to run through S3, which is cheaper than running on SSD volumes, and there is no longer a limit on the amount of DLC jobs that can be run in parallel. DLC stays more consistent with better pruning and is shared between all jobs in a project instead of each cache being different.
• VM service could fall back to another region/zone, but machine provisioner can handle gaps in availability through:
  • Distributing load across availability zones through round-robining
  • Sending a percentage of traffic to a different zone
  • Instance type fallbacks based on deficits and success

New features

• Server execution architecture has been upgraded from VM service to machine provisioner. For further information, see the Installation guide.
• Server now ships with a custom postgreSQL image which now supports pg_crypto, pg_partman, and pg_cron support extensions.
• Added a REPL toggle to turn on a background which deletes workflow data older than 1 year. This is off by default.
• Config policies are now fully supported on server. _compiled_ is now a reserved word in config policies, so that parameterized configuration is fully available to the config policies engine. For further information see the Config policies overview docs.
• The circleci env subst command is now supported. See the Environment variables docs for more information.
• Added support overriding non-CircleCI-managed images.

Changes

• Nomad Autoscaler has been updated to 0.3.7.
• Server no longer deploys the Prometheus chart.
• The artifact paths have changed to improve download performance. The new path will now be <server-domain>/output/job/<job-id>/artifacts/0/home/circleci/artifacts/<artifact-file>

New services

New services introduced with this release

• feature-flags
• runner-admin
• docker-provisioner
• execution-gateway
• machine-provisioner
• output
• step

Services removed with this release

• output-processing
• picard-dispatcher
• test-results
• vm-service

Database migrations

The following databases will run migrations when upgrading to this version

• distributor
• machineprovisioner - Newly Created
• permissions
• runneradmin - Newly Created
• conductor_production

Fixes

• Fixed a bug that caused the Insights page to crash if a user tried to view job resource usage, which is unsupported on server at this time.
• If a job is retried due to an infrastructure failure, it should no longer cause a workflow to get stuck
• Fixed a bug that would try and pull a project’s config with a suspended user’s token.

Known issues

• All pipelines must re-run from start due to architectural changes.
• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

The default Docker version for jobs that use the Remote Docker feature has been updated to Docker 24.0.6.

The default Docker version is used if a specific version is not specified in your .circleci/config.yml file or if you are using the current, edge, or previous tags in the image stanza in your .circleci/config.yml file.