Changelog

With CircleCI’s Evals orb 1.x.x, users were able to configure CircleCI to orchestrate evaluations of their LLM-enabled applications within their CI pipeline.

Evals orb 2.0.0 builds upon version 1.x.x, introducing Evaluation Testing. With Evals orb 2.0.0, users are able to automate the entire process of running evaluations, reviewing evaluation results and determining whether results meet application performance criteria. This removes the need to manually trigger evaluations and manually review evaluation results. Automating evaluations with CircleCI enables developers to automatically determine whether or not to merge proposed changes, or deploy their AI application.

To use Evaluation Testing, users configure test cases to determine if the results of an evaluation meet specified conditions. Each test case represents a scenario with an assertion to check if a specified condition is true or false. If any test case fails, the evaluation test fails. You can configure test case assertions such as Thresholds (Check whether 1 or more results fall within a range), and Equality (Check whether results provide an expected answer)

Evaluation Testing results are presented in the CircleCI web UI in two locations:

1. In the step details
2. In the tests tab

The official CircleCI Evals Orb makes it easy to integrate LLM evaluations into a CI pipeline, and to review evaluation results without context switching. The output of evaluations run through the Evals Orb is stored in CircleCI, and is accessible as a job artifact and as a PR comment added automatically by CircleCI.

Currently, the Evals Orb exposes commands to run evaluations through two popular LLMOps tools: LangSmith and Braintrust. If your evals leverage a different tool, let us know at ai-feedback@circleci.com. You can also contribute directly to the official Orb, by opening a PR on the public repository.

More resources on evaluating LLM-enabled applications are available in our documentation.

Users can now orchestrate their model deployments across multiple environments through the Amazon SageMaker Orb. They can also track and manage their deployments from within the Releases UI.

See our documentation for more information.

What’s New

Dynamic config via Setup workflows is now generally available for customers to use CircleCI jobs to generate pipeline parameters and configurations. These generated parameters and configurations are appended to the existing pipeline and enable customers to dynamically generate configurations based on runtime inputs. Since this functionality is already used in existing workflows/jobs structure, orbs can be used as part of setup workflows, which will enable you, and the community at large, to cover different dynamic configuration use cases by building orbs for different use cases.

One specific use case that has been enabled by setup workflows is path filtering, where you can determine which workflows should be run based on which files are modified (determined via a set of Git operations in a script). This is a key feature if you are using monorepos and will help you optimize your build processes. An orb also been developed to assist you in enabling path filtering.

For more detailed information about Dynamic Configurations, please refer to our product announcement, which can be found here.

For information on getting started with this feature, go to the product documentation, which can be found here.

What’s New in Release 3.0.0

Server 3.0 is now generally available. The newest version of server offers the ability to scale under heavy workloads, all within your own Kubernetes cluster and private network, while still enjoying the full CircleCI cloud experience. Server 3.0 includes the latest CircleCI features, such as orbs, scheduled workflows, matrix jobs, and more. For existing customers interested in migrating from 2.19 to 3.x, contact your customer success manager. Server 3.0 will receive monthly patch releases and quarterly feature releases.

New Features

• New UI
• Orbs for Server
• Pipelines
• Config 2.1
• API 2.0
• Scheduled Workflows
• Matrix Jobs
• And much more…

Known Issues

• No support for external data stores (Postgres, Mongo, Vault). This feature will be implemented in a future release.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior. This will be fixed in a future patch release.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• CircleCI currently assigns a public load balancer for the frontend services. For some customers, their infrastructure or security groups won’t allow this. We will provide an optional internal local balancer for the frontend services in a future release.
• Telegraf metrics collection customization is not yet available.
• Support for GPU builders is not yet available.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

What’s New

Today we are enabling private orbs for our Scale customers.

Two years ago, open-source orbs were launched, helping users to simplify their entire development process. However, we also knew that many developers wanted to privately share configurations across multiple projects, exclusively within their own organization. Private orbs enable users to create orbs specific to their organization without exposing sensitive data or content to users outside of that organization.

You may now create, publish, and update private orbs via your organization’s admin. These private orbs will be restricted to, and accessible by, all users within the organization, and can be listed and imported into a config.yml file directly from the CircleCI CLI.

For more information about private orbs, refer to the Circle CI orbs documentation, which can be found here.

What’s New

Orbs are packages of CircleCI configuration shared across projects. Orbs help you simplify YAML configuration, enable you to build on top of CircleCI, and support sharing of standardized configurations across your projects.

This release provides you with access to certified orbs for AWS CodeDeploy, Rollbar, Artifactory, CodeCov, Heroku, Slack and more. You can also create and publish your own orbs using the CircleCI CLI. Refer to the CircleCI Orbs Registry for the complete list of certified orbs.

To learn about using and creating orbs, see the following documentation:

• Introduction to Orbs
• Using Orbs
• Creating Orbs
• Orbs FAQs