How to improve agent experience (AX) with CI

Improving agent experience (AX) is one thing. Keeping it good as your product changes is harder. A renamed field, different error response, or overlapping tool can turn a workflow that worked yesterday into extra retries, wasted tokens, or human intervention.

CI gives teams a way to catch AX regressions as part of the development process. You can test the interfaces agents depend on, run representative agent workflows against product changes, and preserve fixed failures as regression cases.

A practical AX testing strategy has two layers: deterministic contract tests for agent-facing interfaces and agent evals for the workflows built on top of them. Here’s how to design both, decide what belongs in CI, and use the results to improve your product’s AX.

Start with tasks, not generic “agent quality”

Trying to measure agent experience with a single global score gives you very little to act on. Start with a small set of workflows where failure has a real cost, whether that’s a support ticket, a duplicated resource, or somebody’s afternoon.

Good candidates are workflows where an agent needs to:

• Create or update a resource through your API
• Diagnose a failed operation and recover from it
• Find the correct tool and construct a valid request on the first attempt
• Read current state and decide what to do next
• Complete a command line workflow with no interactive prompts

You only need a handful to start. For each workflow, define success before you build the eval. “The agent did well” is not something a pipeline can check.

Define	Example
Goal	Update an existing deployment
Starting state	The deployment exists but runs an old image
Available interfaces	MCP tools plus the REST API
Success	The correct deployment is updated
Failure conditions	Duplicate deployment, wrong environment, human intervention
Useful signals	Completion, tool calls, retries, latency

Clear success and failure conditions give the eval a concrete target. You can measure whether an agent completed the workflow and identify where it struggled along the way.

Test the contracts agents depend on

Start with the parts of the experience that do not require a model. Agent-facing contract tests are standard software testing applied to the interfaces agents consume.

Useful contract tests include:

• API response schemas and error codes
• CLI output in JSON mode, including on failure
• MCP tool names, descriptions, input schemas, and output shapes
• Exit codes
• Resource and operation state after an action completes
• Permission boundaries, including what an under-privileged caller is told
• Idempotent behavior when an action is retried
• Machine-readable error responses

WorkOS identifies much of the same surface area in its guidance on designing products that agents can actually use: publish machine-readable schemas, enumerate resource states explicitly, return errors with stable codes a caller can branch on, and accept idempotency keys so retries cannot duplicate work. Each behavior creates a contract you can test.

A model adds cost and variability where neither is necessary. If a JSON field disappears or an MCP tool changes names, a deterministic test should catch it. Model-based evaluation is better reserved for behavior that depends on how the model interprets and uses the interface.

In practice, these tests look like the tests you already run. A schema assertion against your MCP tool manifest can live in the same job as the rest of your contract checks:

version: 2.1

jobs:
  agent-contracts:
    docker:
      - image: cimg/node:22.17
    steps:
      - checkout
      - run: npm ci
      - run:
          name: Validate MCP tool manifest against schema
          command: npm run test:mcp-contract
      - run:
          name: Assert CLI JSON output and exit codes
          command: npm run test:cli-contract
      - store_test_results:
          path: ./test-results

workflows:
  validate:
    jobs:
      - agent-contracts

The job uses ordinary assertions and runs on the same commit as the rest of your tests. A missing tool, renamed field, or permission check returning the wrong status code can break the build while the change is still fresh. store_test_results expects JUnit XML, so the assertion library needs a reporter that writes it.

Run representative agent workflows as evals

Once the contracts are covered, test how an agent uses them. Each eval needs a starting state, instructions, available tools, observable behavior, and a defined outcome.

Run each scenario more than once, and capture more than pass or fail:

• Did the task complete?
• Did the agent select an appropriate tool?
• How many tool calls did it need?
• How often did it retry?
• Could it interpret an error and recover?
• How long did resolution take?
• How much inference budget did it consume?
• Did a person have to intervene?

Task completion alone can hide poor AX. An agent may eventually succeed while making excessive tool calls, retrying unnecessarily, or requiring a person to step in.

Netlify, for example, built its AXIS evaluation framework around scenarios that capture tool calls, responses, recovery attempts, time, and token use. Netlify uses those scenarios to detect regressions in agent-facing experiences as its product changes.

Contract tests and agent evals can feed the same CI workflow:

Each product change runs through CI, which exercises the deterministic contracts and the agent workflows built on top of them. Both produce signals you can track from one build to the next.

Running both kinds of checks in CI gives AX regressions a place to surface before users encounter them.

Turn agent experience failures into regression tests

A failed eval becomes useful when you can trace it back to a specific problem in the product. For example, the agent may:

• Choose the wrong tool because two descriptions overlap
• Fail to determine whether an operation has completed
• Burn four calls retrieving context that turns out to be irrelevant
• Retry an action that has already succeeded
• Give up on an error that never explains what to do next

Each failure points to something concrete in the interface. Once you understand the cause, fix it and keep the scenario in your test suite.

The process is similar to regression testing elsewhere in the product: identify the failure, diagnose the interface problem, make the change, verify the improvement, and add the scenario to the suite so CI continues checking it.

Every recurring agent failure you understand is a candidate regression test.

Over time, the suite becomes a record of what your team has learned about how agents use the product. Without a regression case, the same tool-selection or recovery problem can return in a later change.

Give non-deterministic signals a place in CI

A unit test can assert expected === actual. Agent evals are probabilistic, so they need different rules. Treating a noisy eval like a deterministic unit test quickly produces a pipeline people stop trusting.

Deterministic contracts can use hard gates. If the tool manifest loses a required tool or a permission boundary breaks, fail the build.

Agent evals need repeated runs, success-rate thresholds, acceptable ranges instead of exact values, task-specific scoring, and a baseline for comparison. Model and version changes also need to be controlled so you can distinguish a product regression from a change in model behavior.

Instead of requiring a task to succeed on every run, compare results with the established baseline. A useful check might ask whether task completion remains above its threshold without a material increase in retries or human intervention.

Avoid blocking every pull request on a single agent run. A noisy check that frequently blocks good changes will eventually be ignored or disabled.

Check	CI treatment
JSON response schema changed	Hard fail
Required MCP tool missing	Hard fail
Permission boundary violated	Hard fail
Agent task success	Repeated eval against a threshold
Tool-call count	Compare with baseline
Token use	Monitor for regression

Not every check needs to run on every commit. Different types of AX validation belong at different points in the development cycle:

• On every pull request: contract tests, fast deterministic evals, and two or three high-value smoke workflows
• On main, or on a schedule: multi-run agent evals, comparisons across models or providers, larger task suites, and cost and latency benchmarking
• Before a significant release: the full AX regression suite, including high-risk recovery and permission scenarios

Workflow config lets you control which checks run at each stage.

Pull request checks also need to stay fast enough for developers to keep them enabled. Running them in parallel can keep feedback times down as the suite grows.

Fix the interface, not the prompt

When an eval regresses, check what changed in the system around the model before changing the prompt:

• Did relevant context become harder to retrieve?
• Did a tool description become ambiguous against a neighboring tool?
• Did state become implicit where it used to be explicit?
• Did an error lose the detail that made it actionable?
• Did the workflow acquire another round trip?
• Did an action become unsafe to retry?

Prompt changes can help, and sometimes they are the right fix. But a prompt only affects the interactions that use it.

A prompt change may improve one interaction. Fixing the interface improves every agent that uses it.

Your own agent is only one consumer of the product. Other agents may run in tools you do not control with prompts you cannot edit. Improving the interface gives all of them a better chance of succeeding, which makes AX evals especially useful for finding problems in the product itself.

Building agent experience checks into your CircleCI pipelines

Contract tests and agent evals can run alongside the rest of your validation regardless of which eval framework you use. CircleCI gives AX checks the same triggers, compute, reporting, and workflow controls as the rest of your test suite, so agent-facing regressions stay in the existing development feedback loop.

Coding agents also interact directly with CI, making the delivery system part of their experience. We apply the same AX principles to CircleCI:

AX principle	In CircleCI
Programmatic action	CLI and MCP tools
Structured feedback	JSON output and machine-readable failure reports
Fast feedback	Targeted validation in the inner loop
Independent validation	Required CI checks in the outer loop

Smarter Testing helps keep pull request feedback fast by running only the tests a change affects, leaving more room for agent evals in pipelines developers are waiting on.

Make agent experience part of the quality bar

Agent experience changes as your product changes, so it needs the same regression discipline as other important product behavior. Test the contracts agents depend on, exercise representative workflows, track how efficiently agents complete them, and preserve diagnosed failures as regression cases.

CI gives agent-facing contract tests and workflow evals a permanent place in the development process, helping teams catch AX regressions before they reach users.

With CircleCI, you can run AX validation alongside the rest of your software testing and give coding agents structured, actionable feedback throughout the delivery loop.

Start building for free →