PyTest

PyTest is a testing framework based on Python.

Some features/advantages:

  • Can run multiple tests in parallel

  • Allows you to skip/run a subset of tests from the testsuite

  • Free and opensource!

Running PyTest without specifying filenames will run all files of format test_*.py or *_test.py in the current directory or subdirectories.

PyTest test function names must start with test or test_.

Basic Test Example

import math

def test_sqrt():
  num = 25
  assert math.sqrt(num) == 5

def testsquare():
  num = 7
  assert 7*7 == 40

# Note: this won't run since it doesn't start with "test"
def tesequality():
  assert 10 == 11

Run pytest in the same directory level.

Test Subsets

We don’t always want to specify all the tests to run/don’t want all tests to run automatically.

Can run a subset of tests in two ways:

  1. String matching of test name: pytest -k great -v will only run tests with ‘great’ in their name

  2. Test Markers: @pytest.mark.<markername> pytest -m <markername> -v

Test Classes

You can specify a class to group tests. This also allows you share parameters between tests, like value in the example.

# content of test_class_demo.py
class TestClassDemoInstance:
    value = 0

    def test_one(self):
        self.value = 1
        assert self.value == 1

    def test_two(self):
        assert self.value == 1

Grouping in classes also allows you to apply class level fixtures and marks and these will be implicitly added to all tests in that class.

Fixtures

Fixtures are functions which run before each test function to which it is applied. They can be used to feed data to a test function.

The scope is limited to the file you have created it in by default. However, if it is included in your conftest.py file, you can change the scope.

You can set the scope of a fixture to session, module, function, class, package by using @pytest.fixture(scope='session')

You can also set autouse=True if you want it enabled for all test functions in the scope.

import pytest

@pytest.fixture
def input_value():
  input = 39
  return input

def test_divisible_by_3(input_value):
  assert input_value % 3 == 0

def test_divisible_by_6(input_value):
  assert input_value % 6 == 0

Instead of passing the fixture as an argument, you can also use the @pytest.mark.usefixtures('<fixture_name>') decorator to use the fixture in a test function.

Note

You can view a list of builtin fixtures by using pytest --fixtures. These are ones you don’t have to specify but can just use the name of the fixture e.g. tmp_path

Using fixtures for setup and teardown

You can use fixtures for setup and teardown functions for tests. Using it as a setup is useful since you can pass in data to your test function. Using the teardown fixture is useful, since on an assert inside your test function, the remaining functionality won’t be run, meaning if you have teardown logic inside the test, it won’t be run on an assert.

Example using fixtures for setup and teardown
@pytest.fixture(scope="function")
def test_setup():
    print("Test Specific Setup")
    test_data = 5
    return test_data

@pytest.fixture(scope="function")
def test_teardown():
    yield
    print("Test Specific Teardown")

# test_teardown doesn't provide a value so can be declared like this
@pytest.usefixtures('test_teardown')
def test_example(test_setup):
  assert test_setup == 3

It is also possible to chain fixtures:

Example chaining fixtures
@pytest.fixture(scope="function")
def test_setup():
    print("Test Specific Setup")
    test_data = 5
    return test_data

@pytest.fixture(scope="function")
def test_teardown(test_setup):
    yield test_setup
    print("Test Specific Teardown")

def test_example(test_teardown):
  assert test_teardown == 3

In this example, the setup will run first, then the teardown yields to the test. Once the test is complete, due to the yield statement, the rest of the teardown is run.

Choosing fixtures at run-time

You can select which fixtures to run during runtime with pytest.

Example dynamically running fixture
return request.getfixturevalue(argname=fixture_name)

conftest.py

Can use the file conftest.py to share configurations across test files, e.g. fixtures.

import pytest

@pytest.fixture
def input_value():
  input = 39
  return input

Now this input_value can be used across all test files.

Running Tests in Parallel

To run tests in parallel, you need the pytest-xdist plugin.

pip install pytest-xdist

Now you can specify how many workers you want to use to run the tests in parallel:

pytest -n <number_of_workers>

Markers

  • Set a test group that can run as a subset: @pytest.mark.<markername>

    You can register your custom marks in the pytest.ini file. Unregistered marks will trigger a warning. This warning can be raised to an error by passing the strict-markers argument. This avoids accidental mark name errors. Or add something like this to your pytest.ini

    [pytest]
addopts = --strict-markers

Note

You can also use the other marker names specified here to run a subset too e.g. xfail

  • Set a set of inputs to test for a function:

import pytest

@pytest.mark.paramterize("num, ouput", [(1,11),(2,22),(3,35)])
def test_multiplication_11(num, output):
  assert 11*num == output

  • Execute a test, but don’t consider its result: @pytest.mark.xfail

  • Don’t execute a test: @pytest.mark.skip. Add (reason="<reson>") to give a skip reason.

Note

You can view a list of markers by using pytest --markers

Test Execution Results

The results from the test execution can be written to an XML file. This can be used in a dashboard that displays test results.

pytest test_multiplication.py -v --junitxml="results.xml"

PyTest Hooks

Hooks are part of PyTest’s plugin system which allows you to extend the functionality of PyTest Hooks allows you to run custom code at different stages of a pytest run.

The stage at which the code is run is defined by the name of the hook you use. These are predefined names which point to a different stage in the pytest process.

To define a hook, create a function and decorate it with @pytest.hookimpl.

Note

It doesn’t seem hooks can be declared in the test file, but work in the conftest.py file.

There are some main categories of hooks:

  • Bootstrapping hooks

    • Called at the very beginning and end of a test run.

    • Good for setup and teardown of resources used by all tests

  • Initialisation hooks

    • Called at beginning of a test run, after bootstrapping hooks.

    • Used to initialise resources before the test run

  • Collection hooks

    • Called during the process of collecting the tests that will be run in the test suite

    • Can sutomise the way that tests are collected and add additional tests to the collection

  • Test running (runtest) hooks

    • Customise the way tests are run and perform actions before and after a test is run

    Example in conftest.py
    import pytest

@pytest.hookimpl
def pytest_runtest_setup(item):
    print("Setting up test:", item.name)
    # Perform setup tasks here

  • Reporting hooks

    • Called throughout the test process

    • Customise the way results are reported

    • Allows you perform actions based on test results

  • Debugging/Interaction Hooks

    • Allows us to interact with the test session or debug issues that might arise

You can find a list of available hooks Here. These are the names you should use to target a specific part of the PyTest process.

Note

Hooks should start with pytest_* otherwise it won’t be recognised as a hook.

Example modifing list of tests to be run after they have been collected
@pytest.hookimpl
def pytest_collection_modifyitems(config, items):
  # modify the collected items after they have been collected...
  items.append(items[0])

You can also make a function a hookwrapper so that it will wrap another hook function. In the example, the hookwrapper function is called first. The yield statement yields to the wrapped hook function. When that finishes, execution returns to the hookwrapper to complete:

@pytest.hookimpl(hookwrapper=True)
def pytest_collection_modifyitems(config, items):
  print('Entering the collection_modifyitems hook')
  yield
  print('Finished the collection_modifyitems hook')

Options

  • -v: make the output more verbose

  • -k <substring>: run a subset of tests based on given substring

  • -m <markername>: only run tests with given marker

  • --maxfail <max_number_of_fails>: Number of fails after which to halt test execution

  • -n <num_of_workers>: How many parallel workers will run the tests

  • --junittxml=<path_to_file>: Outputs test results to XML

  • -s: This will show print() from test functions in the console

Running multiple instances of pytest on the same code

There might be a situation where you want to run multiple configs for pytest on the same repo.

To avoid using the same config, use a pytest.ini file to root the pytest instance you are calling.

This can cause an issue with your code trying to find some libraries in higher directory levels. One thing that might help, is to run python -m pytest ... instead of pytest directly. They are mostly the same except the first one adds more paths than just pytest by itself.

Output Parsing

Use the junitparser package to parse output results files produced by pytest with the --junit-xml=path option.

e.g. from junitparser import JUnitXml

Command line arguments

You can pass in arguments to pytest in the pytest call.

Example passing in command line args to pytest
# included in conftest.py

def pytest_addoption(parser):

    parser.addoption(
        "--access_token",
        action="store",
        default="",
        help="token for downloading docker image",
    )

    parser.addoption(
        "--use_local_image",
        action="store_true",
        default=False,
        help="Flag for using local docker image",
    )

The above example shows how you can make a simple flag or a argument with a value.

You can view the argument by using the request fixture: request.config.getoption("--access_token")

Note

It is also possible to add options from the pytest.ini file: addopts = --ignore=./dummy_tests in the [pytest] section.

Sources