Runner Release 3.1.0

Container Runner:

• Version 3.1.0 introduces substantial performance and reliability improvements to container runner. These updates are designed to enhance the overall efficiency and stability of operations at scale. For more background and technical details on these changes, you can visit our Runner Init repository and documentation.
• To upgrade, ensure that the Helm chart is at least version v101.1.2. The chart is pinned to the major version tag of the runner, enabling automatic upgrades on redeployment. If you wish to remain on the existing architecture (3.0.x), you can do so by setting the tag to kubernetes-3.0. This version will continue to receive only critical bug fixes and security patches. You can check the configuration details in the Helm chart repository.

NOTE: Release 3.0.23 has been removed from our package repository. It contained bug breaking OIDC token injection for Cloud Runner users. Please use this release 3.0.24 instead.

Container Runner

• Added support for customizable service containers, allowing users more control over a service’s resource usage on a per-image basis. For additional details, refer to the docs.
• Introduced configuration validation to avoid setting the garbage collection threshold below the max task runtime, preventing premature Pod removal and task interruptions.
• Modified the task agent directory to avoid conflicts with existing directories in the PATH variable.

Both Machine and Container Runner:

• Added injection of the CIRCLE_URL variable into task environments. This will resolve to https://runner.circleci.com on cloud, or to the host URL of your server installation on enterprise server.

For customers trying to find cost efficiencies in their runner environment we now support spot instances. The feature can also improve reliability when there are unexpected failures. It comes with some risks and we have documented risks and usage. This release also includes some enhancements and security fixes.

Container Runner:

• Support for unsafe retries, which enable container-agent to automatically rerun tasks that get interrupted during their execution. For example, unsafe retries can be used to run idempotent workloads on spot instances for cost-saving benefits without the worry of Pod preemption causing unexpected job failures. For important information and risk assessment on setting up and using unsafe retries, please read the docs: https://circleci.com/docs/container-runner/#unsafe-retries
• Changed the format of the arbitrary suffix appended to the names of Kubernetes objects that are managed by container-agent. This change has no operational impact on runner.

Both Machine and Container Runner:

• The CircleCI CLI is now prioritized over task agent on the $PATH. That is, invoking the circleci command within a task will point to the CLI binary, if present.
• Patched with Go v1.21.10 for security fixes in Go: https://groups.google.com/g/golang-announce/c/wkkO4P9stm0

Note: If you are running CircleCI Server, you will need version 4.4.3 of server to take advantage of these runner features.

Container Runner

• Reduce long term memory footprint of Container Runner
• Retry tasks where the pod has failed to start due to pod startup errors
• Improve Container Runner monitoring mechanisms for active task pods
• Use the home SSH directory for checkout keys instead of a temporary directory

Machine Runner

• Optionally use the user’s home SSH directory for checkout keys instead of a temporary directory

Critical security fix

• Patched with Go v1.21.5 for critical security fixes in Go. Go release announcement: https://groups.google.com/g/golang-announce/c/iLGK3x6yuNo

Enhancements to error logging and event handling:

• Now logging the outcome of the task Pod’s PreStop hook

• Clearer and actionable error messaging when the task Pod fails to startup or its liveness probe fails

General improvements

• The garbage collector period is now configurable for tuning how quickly failed Pods are removed

• Ability to customize individual values (e.g., the timeoutSeconds) on the task Pod’s primary container’s liveness probe. Any overrides are merged with the defaults, like a kubectl patch.

• See runner configuration documentation for all configuration options