Troubleshooting test splitting

1 month ago • 2 min read

Cloud Server v3.x Server v2.x

Using test splitting with Python Django tests

To utilize test splitting with CircleCI, you must pass in a list of tests to run. However, with how you execute tests with Django, you are unable to simply glob the tests and pass them in.

Sometimes, users run into specific issues when performing test splitting for their own unique use cases. One example of a user resolving an issue when test splitting in Python Django did not perform correctly can be found in the following Discuss post, which can be found here.

Using this example, here is a quick example of how you can accomplish test splitting:

- run:
    command: |
      # get test files while ignoring __init__ files
      TESTFILES=$(circleci tests glob "catalog/tests/*.py" | sed 's/\S\+__init__.py//g')
      echo $TESTFILES | tr ' ' '\n' | sort | uniq > circleci_test_files.txt
      cat circleci_test_files.txt
      TESTFILES=$(circleci tests split --split-by=timings circleci_test_files.txt)
      # massage filepaths into format manage.py test accepts
      TESTFILES=$(echo $TESTFILES | tr "/" "." | sed 's/\.py$//g')
      echo $TESTFILES
      pipenv run python manage.py test --verbosity=2 $TESTFILES

Using test splitting with pytest

If you try to split your tests across containers with pytest, you may encounter any of the following errors:

No timing found for "tests/commands/__init__.py"
No timing found for "tests/commands/test_1.py"
No timing found for "tests/commands/test_2.py"

If any of these errors are returned, you may need to make a few adjustments, which are listed below.

Are you setting a custom working_directory?

If so, you may need to adjust the file paths that are saving to your test metadata XML file. Alternatively, if you are able to, try working out of the standard working directory we set for a container to see if that helps (you can do this by removing any instances of working_directory in your test run job).

Where does your `pytest.ini` live?

To ensure test splitting performs correctly, make sure you are running your tests in the root directory. If your tests are not being run in the root directory, you may need to run the following command before you test the run command:

cp -f .circleci/resources/pytest_build_config.ini pytest.ini

The .circleci/resources/pytest_build_config.ini path may need to be replaced to point to where it’s located in your project.

Are you setting the junit_family in your pytest.ini?

Check to see if you have something like junit_family=legacy set in your pytest.ini file. For more information on how to set junit_family, refer to the following page, which can be found here. Search for "families" to see the relevant information.

A breaking change was introduced in pytest 6.1 the junit_family xml format changed to xunit2, which does not include filenames. This means that --split-by=timings will not work unless you specify xunit1. For more information see the pytest changelog.

Example project that correctly splits by timings

The example below is from the sample-python-cfd project and shows an implementation of test splitting:

version: 2.1
orbs:
  python: circleci/python@1.2
jobs:
  build-and-test:
    parallelism: 2
    docker:
      - image: cimg/python:3.8
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - run:
          name: Run tests
          command: |
            set -e
            TEST_FILES=$(circleci tests glob "openapi_server/**/test_*.py" | circleci tests split --split-by=timings)
            mkdir -p test-results
            pytest --verbose --junitxml=test-results/junit.xml $TEST_FILES
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: test-results
workflows:
  sample:
    jobs:
      - build-and-test

Video: troubleshooting globbing

To follow along with the commands in the video below you will need to be SSH-ed into a job.

Other ways to split tests

Some third party applications and libraries might help you to split your test suite. These applications are not developed or supported by CircleCI. Please check with the owner if you have issues using it with CircleCI. If you’re unable to resolve the issue you can search and ask on our forum, Discuss.

Knapsack Pro - Enables allocating tests dynamically across parallel CI nodes, allowing your test suite exection to run faster. See CI build time graph examples.
phpunit-finder - This is a helper CLI tool that queries phpunit.xml files to get a list of test filenames and print them. This is useful if you want to split tests to run them in parallel based on timings on CI tools.
go list - Use the built-in Go command go list ./… to glob Golang packages. This allows splitting package tests across multiple containers.
```
go test -v $(go list ./... | circleci tests split)
```

Next steps

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Suggest an edit to this page (please read the contributing guide first).
To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.