Testing a Flask framework with Pytest

Writing tests in any programming language can be difficult, but it does not have to be. In this tutorial, I will show you how you can easily write and run tests using Flask and Pytest.

As a bonus, we will also integrate these Flask tests into a CI/CD pipeline using CircleCI.

Be sure to check out our other Flask tutorials to learn about application logging, authentication decorators, and automating Flask deployments. You can also learn more about Pytest in Pytest: Getting started with automated testing for Python.

Prerequisites

To get the most from this tutorial, you will need:

• Basic understanding of the Python programming language
• Understanding of Flask framework
• Basic understanding of software testing

You will also need to have these installations and setups:

1. Python - version >= 3.11 installed in your machine
2. A GitHub account
3. A CircleCI account
4. A clone of the sample project

Once the project is cloned, you also need to install the dependencies using the command pip install -r requirements.txt from the root folder of the project. That single command installs Flask, pytest, and the rest of the project’s pinned dependencies, so you won’t need to install pytest separately.

Before you get too far into the tutorial, it may be helpful to understand what Flask is and how it works.

Why use Flask?

Flask is a lightweight web framework written in Python. Flask comes preinstalled with only the core features so that you can pick customizations depending on the requirements of your project. Flask does not make decisions about which database to use or what the default parser is. All that can be changed with modifications, which makes it extensible and flexible for the emerging needs of your development team.

In this project tutorial, we will use a simple book-retrieval API built with Flask. We will use the API to show how we write tests in a Flask application. To keep this tutorial simple, we will focus on testing the API rather than building it. The API application can be found in the api.py file in the project root directory of the cloned repository.

Now that we know what Flask is and how to use it, we can build on that knowledge by learning about Pytest and how to set it up on a Flask app.

What is Pytest and how do I use it?

Pytest is a Python testing framework designed to assist developers with writing better systems and releasing them with confidence. Small and scalable tests are simple to write with Pytest. This code snippet shows the basic layout of a Pytest test:

from api import app # Flask instance of the API

def test_index_route():
    response = app.test_client().get('/')

    assert response.status_code == 200
    assert response.data.decode('utf-8') == 'Testing, Flask!'

As shown in this snippet, testing Flask requires that we first import a Flask instance app from our api (created in our application). The imported instance then exposes a test_client() method from Flask that contains the features needed to make HTTP requests to the application under test. In this case, it is to our default (/) API endpoint. We then assert that the responses received by test_client() are what we expected after decoding the byte object. In the next steps below, we’ll use this format to write our tests.

Note: Flask’s test_client() returns response bodies as bytes (the response.data attribute). We use the decode('utf-8') method to convert those byte objects to readable string responses before asserting on them.

Setting up Pytest for Flask

Installing Pytest is simple. If you cloned the repository and ran pip install -r requirements.txt, it’s already installed and you can skip this step. Otherwise, run this command in the terminal for your project:

pip install pytest

Note: If you are using a virtual environment, you will want to activate it before doing the installation. It is best practice to use a virtual environment for isolating different applications and their associated Python packages.

Importing the Pytest module

After the installation is complete, import the Pytest module. This snippet can be placed on any test file:

import pytest

Naming conventions for tests

For this tutorial, keep your tests in a tests directory in the root folder of the application. Pytest recommends that test file names begin with the format test_*.py or end with format **_test.py, Using these formats enables Pytest to auto-discover the test files easily and also minimizes confusion when running tests. We will also precede test methods with the word test when creating tests: test_index_route():.

Writing tests with Pytest

Now that we know how to set up Pytest on a Flask app, we can start writing tests for our book-retrieval API. Our first test will be for the /bookapi/books route. The definition of this route is below:

# get books data

books = [
    {
        "id": 1,
        "title": "CS50",
        "description": "Intro to CS and art of programming!",
        "author": "Harvard",
        "borrowed": False
    },
    {
        "id": 2,
        "title": "Python 101",
        "description": "little python code book.",
        "author": "Will",
        "borrowed": False
    }
]

@app.route("/bookapi/books")
def get_books():
    """ function to get all books """
    return jsonify({"Books": books})

For this route, all we are doing is returning a list of books that we have hardcoded in our books variable. We will test this endpoint in the next step. Using the same format, we will define our test function and assert that the response received by the test_client() is what we expect.

import json
from api import app

def test_get_all_books():
    response = app.test_client().get('/bookapi/books')
    res = json.loads(response.data.decode('utf-8')).get("Books")
    assert type(res[0]) is dict
    assert type(res[1]) is dict
    assert res[0]['author'] == 'Harvard'
    assert res[1]['author'] == 'Will'
    assert response.status_code == 200
    assert type(res) is list
    ....

This test snippet is used to make sure that we can receive a list of books. We are doing this by asserting that the response received by the test_client() is indeed our expected list of books. We are also asserting that the response contains a list of dictionaries, which are the individual books as in our defined book object. Finally, we are also asserting that the first book in the list has the author Harvard and the second book has the author Will. That is all we require to write a test to fetch all books from our /bookapi/books endpoint.

Next, we’ll go over how to run test in Pytest to make sure these tests are passing.

Running tests with Pytest

To run Pytest tests, you can either use py.test or pytest in the terminal. You can also run a single file by explicitly specifying the filename after the Pytest command: pytest test_api.py. When these commands are executed, Pytest automatically finds all the tests in either the root directory, or in the specified single file.

When no test files have been explicitly defined, Pytest executes any tests in the root directory that follow the standard naming pattern, without you needing to specify filenames or directories.

We have verification that our test was executed successfully! Pytest marks a passed test with a green dot .; it marks a failed test with a red F. Count the number of dots or Fs to figure out how many tests passed and failed, and in what execution order.

Note: If you are debugging the tests to the console and you need to print out a response, you can use $ pytest -s test_*.py to log to stdout. When the Pytest s option is defined, you can console messages inside the test and debug the output during the test execution.

Now that we have successfully executed our first test, we can integrate CircleCI to run our tests automatically when we push to our main branch.

Setting up CircleCI

Create a .circleci directory in your root directory and then add a config.yml file there. The config file contains the CircleCI configuration for every project. We will use the CircleCI Python orb for executing our tests as part of this configuration:

version: 2.1
orbs:
  python: circleci/python@4.0.0
jobs:
  build-and-test:
    docker:
      - image: cimg/python:3.13
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - run:
          name: Run tests
          command: pytest
workflows:
  sample:
    jobs:
      - build-and-test

CircleCI orbs are reusable packages of YAML config that wrap common build steps. The line python: circleci/python@4.0.0 pulls in the official Python orb and gives us the python/install-packages step we use below. You might need to enable organization settings to allow the use of third-party orbs in the CircleCI dashboard, or request permission from your organization’s CircleCI admin.

After setting up the configuration, commit your changes and then push your changes to GitHub. If you’re trying to push the sample project to GitHub, you’ll need to set your origin-url to your own repository in GitHub. To do that, create a new project on Github. Then run:

git remote set-url origin <your-repo-url>

Next, log into the CircleCI web app and pick your org. The exact onboarding flow depends on which GitHub integration your org uses; both paths are described in the CircleCI quickstart guide and the Create a project docs.

GitHub App orgs (the current default):

1. From the sidebar, select Home, then click Create Project.
2. Pick a project type that matches the work, give the project a name (3–40 characters, starting with a letter), and click Next.
3. Name the pipeline (build-and-test matches our config above) and select Next: Choose a repo.
4. Choose the repo from the list. If it isn’t there, select Add → GitHub → Authorize in GitHub.
5. CircleCI detects the existing .circleci/config.yml we just pushed and offers to use it. Confirm, then select Next: set up your triggers.
6. Leave the default trigger (build on every commit) and click Finish setup (or Commit config and run if no config existed yet).

GitHub OAuth orgs: From the sidebar, select Projects, find the repo, and click Set Up Project. In the modal, pick Fastest (use the config already in the repo), then Set Up Project.

After finishing setup, CircleCI lands on the project’s pipelines page and runs the first pipeline automatically — there’s no need to push a “meaningless” commit anymore.

From the pipelines page, click the running or completed pipeline to see the job and step output. A green check means our pytest tests ran inside the cimg/python:3.13 container and passed.

Now that you have successfully set up continuous integration, the next step is grouping tests and running batches of grouped tests.

Grouping and running batches of tests

As application features are added, you need to increase the number of tests to make sure that everything works. It is easy to get overwhelmed by the large number of test scripts available. You do not need to worry though, Pytest already has that figured out. Pytest allows you to run multiple tests from a single file by grouping tests.

Pytest provides markers that you use to set attributes and features for testing functions.

Using Pytest test markers

We can use markers to give tests different behaviors, like skipping tests, running a subset of tests, or even failing them. Some of the default markers bundled with Pytest include xfail, skip, and parameterize. For our project, we will create a custom marker that will group all the tests that perform GET requests to our /bookapi/books and /bookapi/book/:id endpoints.

Here is an example of the structure you will use to create a custom marker:

@pytest.mark.<markername>
def test_method():
  # test code

To use the custom marker in Pytest, define it as an argument in the pytest command:

$ pytest -m <markername>

-m <markername> is the custom marker name we will use for our tests.

It is important to note that you need to import pytest in your test file to use Pytest markers.

Markers also need to be registered so that Pytest can suppress warnings about them. Add this to your pytest.ini file:

[pytest]
markers =
    <markername>: Marker description

Using markers to group tests

For the tutorial, we want to group the tests that make the GET request to our /bookapi endpoints.

Create a marker called get_request and add it to the tests like this.

import pytest
...
# Other imports here

@pytest.mark.get_request
def test_get_book_by_id():
    response = app.test_client().get('/bookapi/books/1')
    res = json.loads(response.data.decode('utf-8')).get("Book")
    print(res)
    assert res['id'] == 1
    assert res['author'] == 'Harvard'
    assert res['title'] == 'CS50'
    assert response.status_code == 200

Running the test with the pytest -m get_request argument executes all the tests marked with the @pytest.mark.get_request decorator in the test file. You can also verify this by running it in the terminal.

Now, commit and push your changes to GitHub and verify that the pipeline executes successfully.

Excellent! All our tests were executed successfully.

Conclusion

In this tutorial, you have set up tests with Pytest, executed them, and learned to group tests using Pytest markers. You learned how to use Pytest command line arguments to execute tests and how to use the test_client() method to make HTTP requests, and you know how to use the received responses in your tests. Finally, you set up a continuous integration pipeline with CircleCI to automate your tests and and ensure consistent validation of your code changes.

I hope you enjoyed this tutorial. If you did, share what you learned with your team. Nothing solidifies learning like teaching someone else. Until the next one, enjoy your coding projects!