Changelog

What’s New

Version 2 config with pipelines enabled will now automatically have the CIRCLE_COMPARE_URL injected into the job configuration. Previously, if you referenced CIRCLE_COMPARE_URL in a version 2 config, then turned on pipelines, it would disappear. This fixes that issue. For version 2.1 config we do not automatically inject CIRCLE_COMPARE_URL. Instead, we have exposed metadata about the pipeline for you to use more granularly. Mosts users end up parsing out CIRCLE_COMPARE_URL, so instead we did that work for you. In version 2.1 config you can now do the following:

version: 2.1
jobs:
  my-job:
    environment:
      CIRCLE_COMPARE_URL: << pipeline.project.git_url >>/compare/<< pipeline.git.base_revision >>..<<pipeline.git.revision>>

What’s New in Release 2.18

Upgrading your installation

Our upgrade guide takes you step by step through the process of upgrading to the latest version of CircleCI Server, including any checks and updates required before starting the upgrade process.

Note: If at any time your organization name has been changed, there is a script that must be run before starting the upgrade process.

Notes and Best Practices

• We now require a minimum 32GB of RAM for the Services Machine.

• We have made some changes to our Redis configuration. If you have externalized Redis then you’ll need to update your configuration. Please contact your Customer Success Manager.

• We have also made changes to our Postgres version and require at least postgreSQL v9.5.16. If you have externalized postgreSQL then please update to at least that version in 2.17.x before upgrading to 2.18.

Known Issues

Nomad garbage collection only runs every 4 hours. This could prevent nomad jobs from running in environments with heavy usage. If you hit this bug please follow this work around:

1. Create /etc/circleconfig/schedulerer/customizations on the services machine
2. Inside /etc/circleconfig/schedulerer/customizations enter the following:

   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_PENDING_JOBS=500 
   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_DEAD_JOBS=100000
   

3. Restart the picard-scheduler container:

   sudo docker restart picard-scheduler
   

Features/Improvements

• It is now possible to restrict environment variables at run time by adding security groups to contexts. Security groups are defined as GitHub teams or LDAP groups. After a security group is added to a context, only CircleCI users who are members of that security group may access or use the environment variables of the context. For more information see our guide to restricting a context.

• Customers running storage drivers external to AWS will see improved routing times when searching for build Artifacts.

• You can now customize the metrics that get output from CircleCI. For steps and options, see our Custom Metrics guide. Below is a short list of metrics that are included by default when enabling Custom Metrics:

• You can now provide individual AMIs for both Remote Docker and machine executor jobs. Previously we provided the option for a single custom AMI to be used across both, but with v2.18, this expanded customization gives you greater control over versioning and dependencies to meet your individual CICD needs. See our VM Service guide for more information.

Bug fixes

• Additional fixes around contexts and org renames.

• Fixed an issue where occasionally volumes would fail to attach to spun up Remote docker/machine instances.

• Fixed an issue where the CircleCI integration could not be installed on JIRA instances with the jira.com subdomain.

• Fixed an issue where the Workflows page would still point to an old repo after renaming an organization.

• Fixed issue where the Workflows UI would fail to refresh data automatically.

• Improved context loading times in cases when they could cause timeouts in the UI.

• Fixed an issue where contexts would cause builds to return CIRCLE_BUG.

What’s New

• Experimental UI now available to everyone

To try it out, Enable Pipelines in your Project Settings and then click the opt-in banner available on any Job Detail page. We are actively monitoring feedback and bug reports through a poll that appears in the app. The data and commentary we receive directly impacts what we build next, so opt-in to be a part of making CircleCI better!

• Rerun with SSH available on experimental new UI

Rerun with SSH, the top new requested feature from in-app feedback from new UI users, is now available. We are rolling out this feature slowly to monitor for bugs, but we are excited to honor the feedback.

What’s New

• Cancel Button in the New UI For users opted-in to the New UI experience, jobs can now be cancelled from the Job Detail page.

• New Navigation System in the New UI Previously, the Pipelines page was only available from the breadcrumb on the Jobs page, or through the URL. A new central navigation system now appears on the left-hand side of the screen with a Projects menu on the Pipelines page.

What’s New

Select users with pipelines enabled on their projects will see a banner on the Job Details page that enables opt-in to the experimental new UI as it is being built. This new UI includes the following:

• Improved organization of the Jobs page
• An option to open a specific step in a new tab
• Replacement for the current dashboard that lists recent Pipeline runs

Read more information in the Discuss post.

What’s New

When no additional user seats are left in your plan, new users will be blocked from initiating jobs.

The Plan Usage page has also been updated to include Unregistered user (previously Guest user) and Scheduled workflow in your user list if you are initiating jobs usging scheduled workflows.

What’s New

FREE minutes shift to 250/week

Customers on FREE plans will notice a change in how their monthly minutes are allocated. Customers will retain 1,000 FREE minutes per month and will receive them in increments of 250 minutes per week. This will allow customers to budget credits in a more manageable way while aligning with a Mon - Fri work week.

What’s New in Release 2.17

Known Issues

There have been reports of our backpressure settings being too aggressive due to lack of nomad garbage collecting, as well as issues with the Vm Service overwhelming AWS API Limits. If you are on 2.17, you should upgrade to 2.17.1.

Required Changes

IMPORTANT We have upgraded Replicated to version 2.34.1 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

• When using Github Enterprise, operators will now need to provide CircleCI with a Github User Token in the Management Console Settings. This is required due to a change in how the Github Enterprise API allows us to verify Organizations. It is recommended that you use a shared ops account to generate this token. After upgrading to CircleCI Server v2.17, please follow these steps to enact this change:
  • Sign into your GitHub Enterprise instance.
  • Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.
  • Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes.
  • Copy the new token and paste it into the Management Console settings. (Github Integration > GitHub Enterprise Default API Token)
  • Save the settings in Replicated.

Notes and Best Practices

• Fill out this form to receive updates about CircleCI through email.
• It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2.18 it will become required. See docs for our recommendation(s).
• We have updated our software packages to the following versions. This is not a required update for those with externalized environments at this time, but will be when v2.18 is released.
  • Vault
    • 1.1.2
  • Mongo
    • 3.6.12-xenial
  • Redis
    • 4.0.14
• We are removing the 1.0 Single-Box options from CircleCI 2.0. We found a few critical vulnerabilities in our 1.0 build image, and we have long stopped recommending it for trials. If this is absolutely critical to your workflow please reach out to us. This does not impact people who are running 1.0 in clustered mode.

Features/Improvements

• Workflows now has a Slack Integration! Users can choose to receive Slack notifications when their workflows complete.
• Administrators can now restrict what organizations are allowed into their CircleCI installation. For more details on how to enable this feature, please see the User Management Section of the 2.17 Ops Manual.
• We changed the renamed org flow so orgs that have been renamed will no longer result in errors going forward. Existing users who had applied workarounds for this use case will now no longer require said workarounds.
• Workflows now take up less DB space and will be easier to maintain going forward.
• Improved the cache in front of GraphQL API resulting in overall improved performance.
• Added backpressure to avoid overwhelming nomad with requests, this will result in increased performance from existing nomad clusters.

• New machine executor AMIs based on Ubuntu 16.04 for AWS.

• Ubuntu 16.04 with Docker 18.06.3 has apt-daily and apt-daily-upgrade services disabled.
  • It is highly recommended that customers try to experiment with the AMIs below before officially switching:
  • The new images are as follows
    • Ap-northeast-1:ami-0e49af0659db9fc5d
    • Ap-northeast-2:ami-03e485694bc2da249
    • Ap-south-1:ami-050370e57dfc6574a
    • Ap-southeast-1:ami-0a75ff7b28897268c
    • Ap-southeast-2:ami-072b1b45245549586
    • Ca-central-1:ami-0e44086f0f518ad2d
    • Eu-central-1:ami-09cbcfe446101b4ea
    • Eu-west-1:ami-0d1cbc2cc3075510a
    • Eu-west-2:ami-0bd22dcdc30fa260b
    • Sa-east-1:ami-038596d5a4fc9893b
    • Us-east-1:ami-0843ca047684abe87
    • Us-east-2:ami-03d60a35576647f63
    • Us-west-1:ami-06f6efb13d9ccf93d
    • us-west-2:ami-0b5b8ad02f405a909
  • They are replacing:
    • Ap-northeast-1:ami-cbe000ad
    • Ap-northeast-2:ami-629b420c
    • Ap-south-1:ami-97bac2f8
    • Ap-southeast-1:ami-63b22100
    • Ap-southeast-2:ami-dd6c73be
    • Ca-central-1:ami-d02c93b4
    • Eu-central-1:ami-b243eedd
    • Eu-west-1:ami-61de3718
    • Eu-west-2:ami-904e5ff4
    • Sa-east-1:ami-c22057ae
    • Us-east-1:ami-05b6e77e
    • Us-east-2:ami-9b4161fe
    • Us-west-1:ami-efc9e08f
    • Us-west-2:ami-ce8c94b7

Bug fixes

• Fixed some bugs related to GitHub API response handling and webhook handling.
• Fixed issue with Scheduled Workflows when the services machine is restarted.
• Fixed changing the RabbitMQ hostname for Scheduled Workflows when externalizing.
• You can no longer create contexts with no names. If you are using a context with no names, it will no longer be accessible from your execution environment.
• We have optimized our handling of large amounts of build output and test results XML, to avoid out-of-memory errors.
• The CIRCLE_PULL_REQUEST environment variable was not being populated correctly in all cases when building across forks. This has been fixed.
• Fixed a bug where some commits with [ci skip] in the message were still being built.
• Fixed a bug causing Workflows to get stuck when infrastructure_failure happens after a job fails.
• Fixed a bug causing duplicate docker networks on same nomad client (if running build using machine:true AND vm-provider=on_host).
• Improved performance when using local storage. Previously, caching issues had been experienced when local storage was used rather than the default option of using S3 (selecting None under Storage Driver options from the Management Console).
• We have added more error checking and validation around Github’s API so the existing list commit endpoint no longer causes issues.
• Datadog API token field was stored in plaintext, now set as a password field.
• Fixed issue where workflows were constrained from fanning out to large number of jobs.

Fill out this form to receive updates about CircleCI through email.

What’s Fixed in Release 2.16.1

• Bug Fix: This release includes a new flag to disable GitHub cache warming. Note: Only add this flag if a high thread count is impacting the API service. To disable cache warming after completing the upgrade, complete the following steps:

1. Add the export DOMAIN_SERVICE_REFRESH_USERS=false flag to the /etc/circleconfig/api-service/customizations file on the Services machine.

2. Restart CircleCI by going to the Administrative console and clicking Stop Now, waiting for it to stop, then clicking Start.

• Bug Fix: Fixed issue preventing metrics from domain service from being forwarded.

• Security Fix: The SMTP password field in the dashboard was set to plain text. The password field will now be hidden.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.16

The following updates and fixes are also of note:

IMPORTANT We have upgraded Replicated to version 2.29.0 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

New Feature We are excited to announce that you can now distribute your data and workload external to the Services Machine. The following services can be externalized; MongoDB, Redis, Nomad Server, RabbitMQ, Postgres and Vault. Please contact your CSM for the latest documentation.

New Feature Custom Metrics can now be accomplished via a Telegraf Output Configuration File. See the Custom Metrics section of the Monitoring section. Note: The Custom Metrics section in the admin UI been disabled in favor of a configuration file.

New Feature Users can now receive email notifications about Workflows

New Feature The PostgreSQL image has been updated to allow modifying the default configuration by creating the following file: /etc/circleconfig/postgres/extra.conf A list of configuration options can be found here.

Updated Removing EOL Banner on build emails

Updated VM Service stability improvements

Bug Fix Fix issue with contexts that would break after 32 days

Bug Fix Vault auth tokens will now renew periodically to prevent expiration

Bug Fix Fix for an issue where workflow listing pages would intermittently fail to render

Bug Fix We have fixed a small number of bugs affecting processing incoming web hooks from GitHub to CircleCI.

Bug Fix For installs that are using an HTTP or HTTPS proxy, the jobs will now ignore that proxy so that setup_remote_docker works.

Bug Fix Reduced the number of connections 1.0 builders make to the PostgreSQL database.

Security Fix Security fixes for potential cross-site scripting vulnerability and HTTP header injection vulnerability

Security Fix Forked PRs can no longer write the caches of parent projects by default for security reasons. It is possible to still write parent project caches from the fork if the “Pass secrets to builds from forked pull requests” (in Advanced settings) is enabled.

Update Improved metrics for VM provisioning for machine executor. This changed metric names, so if you are monitoring VM provisioning in your install already, you will need to reconfigure the monitoring dashboards for these new metrics.

• New metrics:
  • vm-service.gauges.available-vms and vm-service.gauges.running-vms replaced by vm-service.gauges.vms_by_status
  • vm-service.gauges.running-tasks replaced by vm-service.gauges.tasks_by_status
  • vm-service.gauges.oldest-unassigned-task replaced by vm-service.gauges.unassigned_tasks_age

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
In some instances the metrics section becomes hidden in the settings page. Please see this Knowledge article for instructions.