Runner Release 3.0.25

Both Machine and Container Runner:

As previously reported https://circleci.com/changelog/launch-agent-1-x-variants-eol/.

Runner Release 3.0.24

Both Machine and Container Runner:

NOTE: Release 3.0.23 has been removed from our package repository. It contained bug breaking OIDC token injection for Cloud Runner users. Please use this release 3.0.24 instead.

For customers trying to find cost efficiencies in their runner environment we now support spot instances. The feature can also improve reliability when there are unexpected failures. It comes with some risks and we have documented risks and usage. This release also includes some enhancements and security fixes.

Container Runner:

• Support the user key in the docker executor configuration. This allows you to execute task-agent with a different user than the image’s default. Due to limitations of the Kubernetes API, this feature has some caveats:
  • This key is only supported on the primary container. Service containers will continue to run as the default user.
  • Some user environment variables, such as $PATH, may be reset.
• Further reduction in the memory footprint of Container Runner. This includes a fix for a memory leak that could cause the container-agent Pod to use excess memory and unexpectedly crash under certain workloads.
• Fix for a bug that caused tokens for private AWS ECR images to be cached beyond their expiry, potentially causing image pull errors.

Both Machine and Container Runner:

• Optimize the download and storage caching of new task agent versions.

We will end support for all 1.x variants of runner launch agent on July 31st 2024. It has been replaced with Machine Runner 3.0. You can learn more about machine runner and migration here.

Note: If you are running CircleCI Server, you will need version 4.4.3 of server to take advantage of these runner features.

Runner Release

• Fix bug causing the kube-api server to sometimes prematurely terminate task pods for containerized jobs.
• Improve Task Lifecycle step with dynamic Kubernetes pod lifecycle status for containerized jobs. Providing more detailed information on each step in the lifeycle, making it easier to debug issues if they arise.

Changes

• Fixed a bug where the Task lifecycle and service container steps appear to run indefinitely, even after the task has already completed.

• Log duration for downloads of a task-agent binary initiated by the runner agent are output to local downloads. This should help with troubleshooting issues when customers are experiencing problems downloading agents.
• Fix bug where failed task Pods weren’t removed on task cleanup and had to wait for garbage collection, wasting customers’ cluster resources.
• Additional security patches.

Critical security fix

Container Runner SSH Rerun (Preview)

• Improved retries of select errors on startup of a container task
• The task-agent binaries and pidfile are now installed at more conventional locations for both machine and container runner
• GA release of machine runner 3.0

Machine runner 3.0 is now Generally Available, providing improvements to usability, setup, and maintenance of machine runners.

• Improved the retry logic on task pod startup to prevent errors.
• Release of the Linux machine runner 3.0 preview. For more information see, Installing Machine Runner 3.0 on Linux.

• Reduce claiming interval from 10 seconds to 5 seconds
• Improve robustness when starting tasks on newly created nodes
• Default to a non-root user in the container runner Docker image

What’s New

• CircleCI built in environment variables set as labels on task pods
• Task pod garbage collection improvements
• Improve task-agent task pod injection reliability
• Improve task-agent download reliability

What’s New

• Add circleci symlink to the circleci-agent binary within tasks
• Retry loading resource class token secrets on retryable errors
• Update various dependencies

What’s New

• Upgrade to Go 1.20
• Migrate Helm chart to new image repository: https://hub.docker.com/r/circleci/runner-agent/
• Improve error handling and increase reliability in the task worker loop
• Sample low-value health check logs when using the informational logging level

What’s New

• Bugfixes and package updates

What’s New

• Add lifecycle pre-stop command to gracefully terminate task agent on shutdown
• Ensure all Task lifecycle step events can be sent, even on shutdown of the agent and the context is cancelled
• Don’t delete the task pod if task agent is still running
  • This is related to issue https://github.com/kubernetes/client-go/issues/724. The remotecommand package may not always handle network disconnects on the exec command, returning a nil error. Therefore, we don’t delete the task pod in this case, giving a chance for the task to complete in an “orphaned” state.
• Prevent Task lifecycle step output from being lost during long-running jobs
• Improve task cleanup liveness check robustness and add extra logging
• Fix draining and cleanup of a task on shutdown of the runner agent
• Use stricter task pod config parsing with more helpful error messaging

What’s New

• Various dependency updates
• Fix for task liveness probe retries not being respected
• Fix for task pods not being cleaned up when exiting with a run error

What’s New

• Fix Kubernetes cleanup panic when pod allocation fails
• Fix incorrect liveness probe failures if the task agent PID file is deleted
• Log platform info to the Task lifecycle step
• Make user agent header driver specific

What’s New

• Log errors from calls to Kubernetes API when loading token secrets on Kubernetes driver startup
• Set reasonable resource requests and limits for logging containers
• Add a liveness probe to check the health of the task-agent process
• Respect node RBAC permission setting in K8s GC
• Fix for custom Docker registry names
• Fix for broken pipes while copying task agent when using Alpine-based images on MicroK8s
• Fix for the task token from being able to be exposed in a job
• Support Bash parameter expansion syntax in the job config file
• Improve logging container configuration:
  • Only mount the service account token if there are service containers and therefore a logging container
  • Make the logging container respect the image pull policy of the task pod config, if set. This matches the behaviour of the primary and secondary containers in the task pod.
• Improve task pod event logging in the Task lifecycle step

What’s New

This release includes dependency updates and security patches.

What’s New

• First major release of container runner
• Going forward, this page will detail what has changed between versions which will be released at a regular cadence
• More details around the cadence will be shared shortly