CircleCI Smarter Testing: Stop running tests that don't matter

Testing consumes up to 50% of CI/CD pipeline time. That’s the single most expensive place in your delivery pipeline, and a meaningful share of that time is often wasted running tests unaffected by the change.

Every time a developer pushes a commit, the full test suite runs. A two-line bug fix in auth.py triggers 2,000 tests, including 1,900 that don’t touch anything near auth.py. Those unrelated tests are overwhelmingly likely to pass, but the pipeline still spends 20 minutes running them. In the meantime, the developer context-switches to something else, making it harder to pick the original task back up when results arrive.

AI is making this worse. Developers can now generate code and write tests faster than ever. But faster code creation means more commits, more tests, and more pressure on a pipeline that was already running at capacity. When your test suite doubles in 12 months but your pipeline doesn’t get any smarter, you’ve just doubled the time developers spend waiting.

Smarter Testing is one of the ways CircleCI helps teams reclaim that time. It’s a suite of features that cut test execution time without sacrificing confidence in your results, and the same circleci testsuite run command can run your tests locally. That means you and/or your coding agent can efficiently test a change, no push required. It’s built natively into CircleCI, so there are no new vendors, no new contracts, and no new infrastructure. Users are seeing up to 4x faster test runs and a 2x-3x drop in containers running idle.

Watch the demo below to see Smarter Testing run end to end, then read on for how each feature works.

Why this problem is more expensive than it looks

Slow CI costs more than pipeline runtime. The bigger cost is the hit to developer productivity while tests run.

Research shows it takes an average of 23 minutes to fully regain focus after an interruption. A 30-minute test run doesn’t cost 30 minutes of productivity; it costs an entire working session. Multiply that by 4 commits per developer per day across a team of 50, and you’re looking at a significant fraction of your engineering capacity consumed by waiting.

Slow CI also makes it harder for teams to maintain good CI hygiene: smaller, more frequent commits, continuous validation, and fast integration back to main. These practices reduce risk, improve review quality, and accelerate delivery, but teams can’t build that habit if the feedback loop that would reinforce it is too slow to close.

The common response—buying more compute—addresses the symptom without fixing the cause. Upgrading to larger resource classes makes every pipeline run more expensive, forever. Smarter Testing makes the existing compute work smarter: fewer tests run, nodes are better balanced, and flaky tests don’t trigger full reruns.

At Clazar, a backend suite of roughly 4,000 tests was adding 12–15 minutes to every PR. After enabling Smarter Testing, feature branch runs dropped to 3–5 minutes. At Liti, PR jobs are now 4x faster.

How Smarter Testing works

Smarter Testing includes features designed to eliminate unnecessary test work. You can use them independently or combine them, depending on where your test pipeline is wasting time.

1. Intelligent test selection: run only what matters

The biggest time savings comes from not running tests that can’t possibly be affected by your changes. Intelligent test selection, powered by test impact analysis, does exactly that.

Our approach uses code coverage data to map how tests interact with your source code. To guarantee safety and prevent under-selection, the testsuite command starts with your full test suite selected, then conservatively deselects only the tests proven to be unaffected by modified files.

This isn’t sampling or heuristic guessing. If your change touches auth.py, only the tests that cover auth.py (and code it calls) are selected. Tests for unrelated modules are skipped. Your full suite still runs on your main branch to maintain the coverage baseline, but on feature branches—where fast feedback matters most—you can routinely skip 60–90%+ of your tests.

When intelligent test selection is the right fit: It has the highest potential impact for codebases where most changes are localized. If you have a large test suite and most PRs touch a narrow area of the codebase, intelligent test selection will skip the overwhelming majority of tests on every feature branch run. It’s particularly valuable in AI-assisted development workflows, where developers iterate rapidly and need tight feedback loops to stay in flow.

Intelligent test selection requires your test runner to generate code coverage data. Most modern frameworks support this (Jest, pytest, Go test, Vitest, RSpec, Mocha, Cypress, and others). Your tests don’t change, but analysis needs the coverage plugin for your runner (for example @circleci/jest-circleci-coverage), added as a dev dependency.

On test confidence: Because test impact analysis is based on actual coverage data rather than guesswork, it’s safe to rely on. And because Smarter Testing records which tests ran and why, you get more visibility into your test suite than you had before, not less. The test analytics dashboard shows skip rates, run time trends, and per-test history, so you can see exactly what ran on any given commit and what was skipped.

2. Dynamic test splitting: eliminate idle time

If you’re using parallelism to run tests across multiple nodes, you’ve likely noticed the nodes don’t always finish together. One node wraps up early while another is still grinding through a long test file. Your total job time is determined by the slowest node—so that idle time is pure waste.

Static test splitting assigns tests to nodes upfront based on historical JUnit timing data. The problem is that JUnit timings often don’t reflect real wall-clock runtime—they miss fixed costs like runner startup, per-batch setup overhead, and hook execution. The result is uneven splits even when reported durations look balanced.

Dynamic test splitting takes a different approach. Instead of fixing each node’s share upfront, it gives each node an initial batch sized so all nodes finish at about the same time, then shares out the remaining tests as nodes free up. It learns the gap between reported and actual runtime over time, accounting for fixed costs and per-test overhead. Nodes that start late or run slower receive smaller batches so they still finish at roughly the same time.

When dynamic splitting is the right fit: Dynamic splitting helps most when parallel nodes finish at noticeably different times because of uneven test durations, startup delays, or inaccurate timing data. If your nodes already finish within seconds of one another, the benefit will be limited.

3. Auto rerun failed tests: handle flakiness automatically

Flaky tests are nearly universal in active test suites—tests that fail intermittently due to timing issues, external service calls, or environment variance. The traditional response is to wait for a CI failure, manually identify which tests failed, navigate to the CircleCI UI, and trigger a rerun. This is tedious, slow, and entirely manual.

Auto rerun failed tests automates recovery. When a test fails, Smarter Testing immediately reruns the failed test atoms (usually the failing test files) on the same node, no human intervention required. The process repeats until all tests pass or the maximum number of retries is reached. If all tests eventually pass, the errors are suppressed and the job succeeds, keeping pipelines moving without noise or manual triage.

You control retry behavior via two settings in test-suites.yml:

• max-auto-rerun: the maximum number of retry attempts per failing test atom (0–10)
• auto-rerun-duration: a time limit or percentage of total runtime to spend on retries (e.g., 5m, 20%)

Both can be set simultaneously; retries stop when either condition is reached.

When auto rerun is the right fit: If your test suite has known flaky tests, this is an immediate improvement. It’s especially useful alongside intelligent test selection. When you’re running fewer tests overall, even occasional flakiness in a small subset can cause false failures. Auto rerun catches those without requiring manual intervention.

Note that auto rerun handles flakiness, not real bugs. If a test fails because of an actual code problem, it will keep failing on retry and the job will ultimately fail—which is correct behavior. The goal is to stop treating intermittent environmental failures as blocking issues.

4. Dynamic parallelism: only spin up the nodes you need

Parallelism is usually set in config and tuned to hit a feedback-time target on a full test run. That works fine if you run all tests every single time. With intelligent test selection, a feature branch might run only 10% of the suite, but a job set to 100 nodes still starts all 100. Most of them boot, find little or nothing to do, and shut down.

Dynamic parallelism solves that. An upstream job calculates how many tests will run and passes that number to the downstream test job, which starts only the nodes it needs. Unless you set a target duration, Smarter Testing picks the fewest nodes that will still finish in about the same time as running the full suite with maxed out parallelism. If you’d rather optimize for developer time, set a target duration and Smarter Testing dynamically sets the number of parallel nodes to hit it. You can size it two ways:

• Scale with the selected tests. A small selection uses fewer nodes and a full run on your default branch uses your maximum. Dynamic parallelism never uses more nodes than your max, or more nodes than there are test atoms to run. Only nodes with a workload are spun up.
• Target a duration. Set a target such as 2m and CircleCI picks enough nodes, within your min and max, for each to finish its tests in about that time.

Dynamic parallelism builds on test selection. Once test selection determines the tests a change needs, dynamic parallelism spins up the nodes to match. Setting a target duration also works without test selection if you use static or dynamic test splitting.

When dynamic parallelism is the right fit: It helps most when you run high parallelism and use intelligent test selection, so selection regularly cuts the work well below a full run. Savings come on feature branches and pull requests. Your default branch always runs every test, so it uses your maximum node count.

See Set parallelism dynamically for how to set it up.

What’s included: free vs. paid

When you adopt the circleci testsuite run command you get a free tier that includes everything the legacy circleci tests run command offered: test running, static test splitting across parallel nodes, and rerunning only failed tests. If you migrate to testsuite and use only these capabilities, there’s no additional cost.

On top of that foundation, Smarter Testing’s paid features go further: intelligent test selection (powered by test impact analysis), dynamic test splitting, dynamic parallelism, and auto rerun failed tests. These are billed by test atoms evaluated, but you can use as many or as few as you like without additional charges. A test atom is the smallest unit of work your test runner can execute independently, typically a test file such as tests/auth_test.py. Every pipeline run evaluates its test atoms, including reruns. Each plan includes a free monthly threshold: 1,000 test atoms on Free, 10,000 on Performance, and 100,000 on Scale. Beyond that, usage costs 3,000 credits per 100,000 test atoms. See the pricing page for plan details. A run’s test atoms evaluated are counted once, however many paid features are enabled. Running Smarter Testing locally isn’t charged.

Setting up Smarter Testing

All features build on the same foundation: a .circleci/test-suites.yml file that defines how your tests are discovered and run, and the circleci testsuite run command that replaces your existing test command in CI. You configure this once, then enable individual features by adding options to the YAML file.

Step 1: Run the doctor command

Ensure you have the latest CircleCI CLI installed. Then run the doctor command from the directory where your tests are located (typically your repository root):

$ circleci testsuite doctor "ci tests"

The CLI walks you through a series of checks, executes your tests, validates the output, and provides specific action items for anything that looks misconfigured. Follow the steps until all checks pass.

Monorepos: Run the CLI from each subpackage root, not the repository root. For example: cd service-1 && circleci testsuite doctor "ci tests". The CLI looks for .circleci/test-suites.yml relative to where it’s invoked.

You can also run the doctor command as a CI job step to validate in your actual CI environment:

version: 2.1
jobs:
  doctor:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run: circleci testsuite doctor "ci tests"

Step 2: Replace your test command

Once the doctor command passes, replace your existing test command in .circleci/config.yml with circleci testsuite run "ci tests".

Before:

version: 2.1
jobs:
  test:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run: npm test
      - store_test_results:
          path: test-reports

After:

version: 2.1
jobs:
  test:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run: circleci testsuite run "ci tests"

Step 3: Enable the features you want

Features are enabled via the options block in .circleci/test-suites.yml. A minimal configuration, storing tests, looks like this:

---
name: ci tests
discover: jest --listTests
run: JEST_JUNIT_OUTPUT_FILE="<< outputs.junit >>" jest --runInBand --reporters=jest-junit --bail << test.atoms >>
options:
  store-test-results: true

Then add the options for whichever features you want:

Intelligent test selection (paid): Add an analysis command that writes coverage to << outputs.circleci-coverage >>, using your runner’s coverage plugin (for Jest, @circleci/jest-circleci-coverage, with its test environment and reporter in your Jest config):

analysis: CIRCLECI_COVERAGE="<< outputs.circleci-coverage >>" jest --runInBand << test.atoms >>
options:
  test-impact-analysis: true

Dynamic test splitting (paid, requires parallelism > 1 in your job config):

options:
  dynamic-test-splitting: true

Auto rerun failed tests (paid):

options:
  max-auto-rerun: 3
  # Optional: also limit by time
  auto-rerun-duration: 5m

Dynamic parallelism (paid): This one is set up in .circleci/config.yml, not test-suites.yml. Add a job that lists the tests the test job will run, then point the test job at it (replace my-executor and setup with your own executor and your checkout and install steps):

jobs:
  list-tests:
    executor: my-executor
    steps:
      - setup  # checkout and install what your discover command needs
      - run: circleci testsuite list-tests "ci tests" --key=ci-tests-parallelism-key

  test:
    executor: my-executor
    parallelism: 20  # fallback, and the max for a full suite
    steps:
      - setup
      - run: circleci testsuite run "ci tests"

workflows:
  test-workflow:
    jobs:
      - list-tests
      - test:
          requires:
            - list-tests

experimental:
  dynamic-parallelism:
    test:
      key: ci-tests-parallelism-key
      min: 1
      max: 20
      target: 2m  # optional; remove to scale with the selected tests

Sizing uses test timings, which Smarter Testing records whenever the full suite runs. Merge to your default branch, or run once with --run-tests=all, before it has the data to adjust the number of nodes.

You can combine all of these features. The example below enables intelligent test selection, dynamic test splitting, and auto rerun failed tests. Because the charge is for test atoms evaluated, you can stack paid features without incurring additional cost as you add them:

---
name: ci tests
discover: jest --listTests
run: JEST_JUNIT_OUTPUT_FILE="<< outputs.junit >>" jest --runInBand --reporters=jest-junit --bail << test.atoms >>
analysis: CIRCLECI_COVERAGE="<< outputs.circleci-coverage >>" jest --runInBand << test.atoms >>
options:
  test-impact-analysis: true
  dynamic-test-splitting: true
  max-auto-rerun: 3
  store-test-results: true

Step 4: Verify locally and in CI

Locally

$ circleci testsuite run "ci tests"

In CI

Commit both .circleci/test-suites.yml and .circleci/config.yml to a feature branch and push. After the pipeline runs:

• Open the Tests tab and confirm the test count matches what you’d expect. A significant mismatch may indicate a misconfiguration in your discover command.
• Compare the job duration to a recent run without testsuite. It should be similar at baseline. Speedups from intelligent test selection appear on subsequent runs as the impact map is established.
• For dynamic splitting, check the Timings tab to confirm tests are distributed more evenly across nodes.
• For dynamic parallelism, open the test job and check Parallel runs on the Steps tab. The count should fall between your min and max.

The ROI case

For teams deciding whether the paid tier is worth it, the math is straightforward. Consider a 100-developer team running a 20-minute test suite, with each developer pushing 4 feature branch commits per day:

Without Smarter Testing, that’s 400 feature branch runs × 20 minutes = 8,000 minutes of pipeline time per day. With intelligent test selection reducing feature branch test volume by 60%, the average run drops to ~8 minutes—3,200 minutes per day. Over a month, that’s 105,600 minutes of compute savings, plus every developer recovering roughly 12 minutes per commit.

At a fully-loaded developer cost of $135/hour, recovering even 25% of that wait time as productive work translates to tens of thousands of dollars in recovered engineering capacity each month. The Smarter Testing fee, based on test atoms evaluated, is typically a rounding error compared to that figure.

The ROI is driven almost entirely by developer time. Compute savings matter, but the larger benefit is getting useful feedback to developers faster so they can keep changes moving through the pipeline.

Troubleshooting common issues

Test count in the Tests tab doesn't match expected results

This usually means the discover command in test-suites.yml isn’t finding all test files, or is picking up files it shouldn’t. Re-run circleci testsuite doctor "ci tests" and look for mismatches in the discovery output. Verify your glob patterns and that the CLI is being invoked from the correct directory.

First run after enabling intelligent test selection still runs all tests

Expected behavior: intelligent test selection needs a full coverage baseline before it can determine which tests are impacted by a change. The first run on your main branch establishes that baseline. After that, subsequent feature branch runs start selecting only impacted tests.

A change to a dependency or config file runs no tests

Some files, like dependency manifests or shared test setup (for example conftest.py), are often not covered by any test, so a change to them can result in no tests running. Add them to full-test-run-paths in test-suites.yml so changing them runs the full suite.

Dynamic test splitting doesn't improve node balance

Dynamic splitting improves as it accumulates timing data over multiple runs. Early runs may look similar to static splitting. If nodes are still unbalanced after many runs, check whether your test runner startup time is high relative to test runtime—if startup takes longer than the tests themselves, splitting across multiple batches can actually increase total runtime. Dynamic splitting works best when test runner startup is fast relative to test execution.

Dynamic parallelism always runs the maximum number of nodes

A few things cause this. There may be no timing data yet, so run the full suite once on your default branch. Intelligent test selection may not be on, and without it list-tests selects the whole suite. Or the flags may differ: if your test job passes --run-tests or --analyze-tests, pass the same flags to list-tests. If CircleCI can’t set parallelism, the job falls back to the parallelism value in your config.

Also check that key in your config matches the --key you pass to list-tests, and that target is a valid duration between 8s and 5h, such as 2m. An invalid target also falls back to the parallelism value in your config. If test selection hits an error, it selects every test, so the job uses your maximum.

Auto rerun keeps retrying the same test without passing

A test that fails consistently is not flaky—it’s broken. Auto rerun will exhaust the max-auto-rerun limit and the job will correctly fail. Increase your retry limit (up to 10) if the test is genuinely intermittent but needs more attempts, or add auto-rerun-duration to bound the time spent retrying.

circleci testsuite command not found

Ensure you have the latest CircleCI CLI installed. Run circleci update to upgrade, or reinstall from cli.circleci.com. The testsuite subcommand requires a recent CLI version.

Tests pass locally but fail in CI

Run circleci testsuite doctor "ci tests" as a CI job step to validate the configuration in your actual CI environment. Differences in working directory, available tools, or coverage configuration can cause the testsuite command to behave differently locally vs. in CI. The doctor command surfaces those differences.

Coverage data isn't being picked up for intelligent test selection

Test impact analysis needs an analysis command in test-suites.yml. For large JavaScript or Python suites, use the coverage plugin for your runner (for example @circleci/jest-circleci-coverage or pytest-circleci-coverage) and write its output to << outputs.circleci-coverage >>. Without it, analysis runs one test atom at a time, which is much slower.

Re-run the doctor command. It will flag missing or misconfigured coverage output specifically.

Monorepo: tests aren't being discovered correctly

Make sure you’re running the CLI from the subpackage root, not the repository root. The testsuite command executes relative to where it’s invoked. Avoid using cd commands or --directory flags within your discover or run commands. Set the working directory in the job step instead.

Getting started

Running fewer unnecessary tests means faster feedback without lowering confidence in the result. Smarter Testing gives teams a few different ways to get there, depending on where their test pipeline is losing time.

Start by adopting the circleci testsuite run command (free, full parity with the legacy command) and verifying your test setup. Then enable the paid features that address your biggest bottlenecks. Most teams see the largest immediate gains from intelligent test selection, and the benefits increase as dynamic test splitting and auto rerun failed tests eliminate additional sources of wasted test time. If you run high parallelism, dynamic parallelism then keeps the node count in step with how much of the suite actually runs and reduces compute cost.

Further reading:

• Set up Test Impact Analysis
• Use Dynamic Test Splitting
• Auto Rerun Failed Tests
• Test Suite Configuration Reference
• Set parallelism dynamically