Skip to content
Docs
Cancel

CircleCI Server v3.x What’s New

Last updated
Tags Server v3.x Server Admin

Server 3.x is now generally available. The newest version of server offers the ability to scale under heavy workloads, all within your own Kubernetes cluster and private network, while still enjoying the full CircleCI cloud experience.

Server 3.x includes the latest CircleCI features, such as orbs, scheduled workflows, matrix jobs, and more. For existing customers interested in migrating from 2.19 to 3.x, contact your customer success manager. Server 3.x will receive monthly patch releases and quarterly feature releases.

Release 3.4.2

Upgrade notes

Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics:

  1. Run kubectl delete deployment/circleci-server-kube-state-metrics --cascade=orphan

  2. Deploy the app from the KOTS Admin Console

Changelog

For full details of this release see the changelog.

Known issues

  • Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.

  • Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.

  • Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.

  • CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

  • The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

Release 3.4.1

Upgrade notes

Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics:

  1. Run kubectl delete deployment/circleci-server-kube-state-metrics --cascade=orphan

  2. Deploy the app from the KOTS Admin Console

Changelog

For full details of this release see the changelog.

Release 3.4.0

Upgrade notes

Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics:

  1. Run kubectl delete deployment/circleci-server-kube-state-metrics --cascade=orphan

  2. Deploy the app from the KOTS Admin Console

New Features

  • Android images for the machine executor are now available. For more information, see the Using Android Images with the Machine Executor guide.

  • The CircleCI Build Agent can now be pulled from a custom Docker registry if required. For more information contact customer support.

  • You can now use Amazon Certificate Manager (ACM) to manage your TLS certificates. For more information see the Server v3.x Installation Guide.

  • Kubernetes v1.16 - v1.23 are now supported.

  • Workload Identities for installations on GCP are now supported. Workload Identities can be used as an alternative to static credentials. For more information see the Server v3.x Installation Guide

Changelog

For full details of this release see the changelog.

Release 3.3.1

Changelog

For full details of this release see the changelog.

Release 3.3.0

New features

  • Nomad Autoscaler can now be used to scale Nomad clients. For more information see the execution environments installation docs.

  • Webhooks are now available.

  • The Insights dashboard is now available.

  • IRSA (AWS) can now be used as an alternative to keys for object storage authentication.

  • The email address from which build notifications are sent is now configurable from the KOTS Admin Console.

  • We have replaced Traefik with Kong as our reverse proxy. However, in order to minimize disruption when upgrading, we chose not to rename the service used by kong. Therefore, you will see a service named circleci-server-traefik, however, this service is actually for Kong.

Changelog

For full details of this release see the changelog.

Release 3.2.2

Upgrade notes

  • The rerun workflow endpoint now returns workflow ID rather than the message accepted.

Changelog

For full details of this release see the changelog.

Release 3.2.1

Upgrade notes

From the KOTS Admin Console, select Version History from the menu bar and click Deploy for server v3.2.0.

See Upgrade notes before upgrading from v3.1.x to v3.2.x.

Changelog

For full details of this release see the changelog.

Release 3.2.0

Upgrade notes

From the KOTS Admin Console, select Version History from the menu bar and click Deploy for server v3.2.0.

When upgrading from server 3.1.x to 3.2, there will be some downtime due to a change to the PostgreSQL pod. There are two issues you could run into with this update, which are covered in the following sections.

PostgreSQL pod stuck in pending

If you find that the PostgreSQL pod is stuck in a pending state after upgrading, scale down the pods to 0 and then scale up again by following the steps below.

To check if your PostgreSQL pod is stuck in pending, use the following command:

$ kubectl get pod -l app.kubernetes.io/name=postgresql
NAME           READY   STATUS    RESTARTS   AGE
postgresql-0   1/1     Pending   0          3m

The following command will scale down pods to 0 and terminate the application pods without any data loss:

kubectl scale deployment -l layer=application --replicas 0

Once all the application-layer pods have finished terminating, do one of the following:

  • either redeploy the update from the KOTS Admin Console

  • or run the following two commands to redeploy the pods and return server to a functional state:

    kubectl scale deployment -l layer=application --replicas 1

    Then scale output-processor up with the following command:

    kubectl scale deployment output-processor --replicas 2

Traefik pod fails to schedule

If you find that there are two Traefik pods after upgrading, you need to locate the older pod and remove it to allow the new pod to schedule correctly.

To see the status of your Traefik pod, use the following command:

$ kubectl get pod -l app=traefik
NAME                                      READY   STATUS    RESTARTS   AGE
circleci-server-traefik-9d6b86fd8-f7n2x   1/1     Running   0          24d
circleci-server-traefik-cf7d4d7f6-6mb5g   1/1     Error     0          3m

Remove the older Traefik pod with the following command:

kubectl delete pod circleci-server-traefik-<older pod hash>

The new Traefik pod will then start to schedule correctly.

New features

  • Customers who require a fully private installation can now access a setting in the KOTS Admin Console to ensure public IPs are not assigned to VMs. Note that with this non-public IP setting enabled, a workaround will be needed if SSH access to running jobs is required, for example, by using a VPN into your VPC.

  • Customers who manage outbound traffic through a proxy can now configure proxy settings through the KOTS Admin Console. Please see our documentation for specifics on proxy support for server.

  • We have expanded the machine execution environment options available to include additional resource classes, sizes, and executors. You now have access to Arm (medium, large), Linux (medium, large, X large, and XX large), and Windows (medium, large, XX large) resource classes.

  • The insights API is now available to all server customers. Leverage build and other data to better understand the performance of teams and the health of your build and testing efforts.

  • We have revamped the admin UI, and updated our installation instructions, making it easier to set up and manage server.

  • You can now supply a custom Linux AMI for VM service.

  • SSL termination can now be disabled. If you have put server login behind a firewall, this will enable SSL termination at the firewall.

  • You can now control the size of persistent volumes. For larger customers, the initial persistent volume size was too small, by default. You can now set this at install time, providing an easier migration for those customers that require it. For further information see the Internal Database Volume Expansion doc.

  • We have added an auto-scaling example to the nomad client terraform.

  • You can now choose to serve 'unsafe' build artifacts. Previously this option was hidden, meaning potentially unsafe artifacts were rendered as plain text. For more information see the Build Artifacts doc.

Changelog

For full details of this release see the changelog.

Release 3.1.0

Upgrade notes

With this release, the frontend-external load balancer has been removed. The traefik load balancer now handles all incoming traffic. When updating from a previous server 3.x version, you need to update the DNS record that was pointing to the frontend-external load balancer and point it to the circleci-server-traefik load balancer instead. Remember, you can retrieve the external IP address or DNS name of your Traefik load balancer by typing kubectl get svc/circleci-server-traefik in a terminal that has access to the cluster.

To update your DNS record and upgrade your server installation, follow these steps:

  1. Retrieve the external IP or DNS name for the Traefik load balancer as described, or by looking the DNS A record for app.<your domain name> - this should already point to your Traefik load balancer.

  2. Locate the DNS A record that points to the domain name of your server installation (not the one pointing to the app. subdomain).

  3. Edit the A record so that it points to the Traefik load balancer, the same as the app. subdomain record. Your changes might need a couple of minutes to take effect, depending on your DNS service.

Next, from the KOTS Admin Console, select Version History from the menu bar and click Deploy for server v3.1.0.

New features

  • Telegraf plugins can now be added to server and customized to use third-party monitoring solutions, for example, Datadog. For more information, see the Metrics and Monitoring doc.

  • The option to use only private load balancers has been introduced for customers who want a fully private installation. For more information, see the Load Balancers guide.

  • Server 3.x hosts build artifacts, test results, and other state in object storage. We support any S3-compatible storage and Google Cloud Storage. For more information, see the Installation guide.

  • Dynamic config via setup workflows is now available on server installations. For more information, see our blog post and the Dynamic Configuration docs page.

  • Runner is now available on server. For further information, including installation steps, see the Runner docs. Runner allows the use of the macOS executor in server installations and VM service functionality for customers with server installed in a private data centre.

  • The frontend load balancer from v3.0 has been removed and replaced with an Ingress resource and the Traefik Ingress controller. This is a breaking change requiring you to reconfigure your DNS. See the What’s New in server docs for further information and guidance.

  • The following services can now be externalized. For setup information, see the server v3.x installation guide:

    • Postgres

    • MongoDB

    • Vault

  • Backup and restore functionality is now available. For more information see the Backup and Restore guide.

  • Prometheus is now deployed by default with server to monitor your cluster health and usage. Prometheus can be managed and configured from the KOTS Admin Console. For further information, see the Metrics and Monitoring doc.

  • Server now supports the 2XL resource class. The Nomad cluster needs to be large enough to account for larger resource classes.

  • The lifecycle of build artifacts and test results can now be configured from the KOTS Admin Console under Storage Object Expiry, including the option to disable the expiration and retain artifacts and test results indefinitely.

Changelog

For full details of this release see the changelog.

Release 3.0.2

Changelog

For full details of this release see the changelog.

Release 3.0.1

Changelog

For full details of this release see the changelog.

To learn more about Server v3.x, see the following:


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

Creative Commons License
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

6,000 Minutes Free

CircleCI, now with more build minutes
than anyone else for FREE 🎉

Sign up for Free

These teams get it

CCI customer teams