Pytest parameterization

When writing automated tests, parameterization is frequently used, allowing the same test to be executed with different inputs. This approach addresses various challenges in test automation. This guide explores the problems parameterization can resolve, Pytest parameterization, and its integration with Allure Report.

1. Preparation

Prerequisites

Make sure the following prerequisites are met:

• Python installed
• Allure Report installed

Dependency List

This guide uses the following packages:

• Pytest 8.3.4
• allure-pytest 2.13.5

Code sample

The complete source code used in this guide is available at https://github.com/allure-examples/guide-pytest-parameterization.

Setup

To run the examples in this guide:

1. Install Python

2. Install Allure Report

3. Download a fresh project with Pytest from Allure Start

4. In the project directory, use the script that corresponds to your operating system:

   MacOS / Linux

   ./run.sh

   Windows

   .\run.bat

2. Why use parameterization

Let's start with a simple example: we want to test if a sum of two numbers is calculated correctly.

def add(a, b):
    return a + b

def test_sum():
    s = add(1, 2)
    assert s == 3

Now, how can we execute this test across various data sets?

One approach is implementing a loop through a data set within the test itself. For instance:

def test_sum_loop():
    data = [
        [1, 2, 3],
        [-1, 1, 0],
        [0, 0, 0]
    ]
    for x, y, ttl in data:
        s = add(x, y)
        assert s == ttl

This approach has many disadvantages:

1. If an assertion for one specific input fails, the whole test fails immediately without reporting the results of the remaining cases.
2. We lose visibility on which specific input set caused the test failure, making it harder to debug and maintain.
3. A test report only shows the test name, not the particular cycle of the loop. In this setup, there is no option to make it more readable.

Pytest framework provides a better alternative: parameterization. To parameterize a test with Pytest, you can use the @pytest.mark.parametrize decorator with parameter names and a list of test inputs. Pytest runs the test function with each test input creating separate test cases. With this feature, we can rewrite the previous test:

data = [
    [1, 2, 3],
    [-1, 1, 0],
    [0, 0, 0]
]

@pytest.mark.parametrize("x,y,ttl", data)
def test_sum_parameterized(x, y, ttl):
    s = add(x, y)
    assert s == ttl

With Pytest, you can also parameterize fixtures. Parameterized fixtures allow running the same test multiple times with different fixture values as test inputs. For example:

@pytest.fixture(params=[0, 1, 2])
def sum_arguments(request):
    yield prepare_test_data(request.param)

def test_sum_simple(sum_arguments):
    x, y, ttl = sum_arguments
    assert add(x, y) == ttl

def test_sum_keyword(sum_arguments):
    x, y, ttl = sum_arguments
    assert sum((x, y)) == ttl

In this example, we define a fixture sum_arguments and pass a list as the fixture params argument. The prepare_test_data function represents code that prepares the test data that goes to the test functions below. In the example repository, this function reads the data from the global variable. But it could have also retrieved it from another source: a file, a database, and so on.

This approach helps if you need to parameterize multiple tests using the same set of data.

Both parameterized tests and parameterized fixtures provide significant advantages:

1. Each test is executed multiple times with different inputs. If one run fails, you will still get feedback from all the others.
2. The test data is defined in one place and accessed through variables (x, y, and sum), which is much more convenient.
3. In a test report, multiple runs of a parameterized test with different inputs are shown separately.
4. The data preparation is separated from the test functions making the code more compact and easier to read.

3. Parameterization in Allure

Allure Report integrates with Pytest and supports parameterized automated tests. An example of parameterized tests in Allure Report:

Allure Report recognizes all Pytest parameterization mechanisms including fixtures, the @pytest.mark.parametrize annotation, and the pytest_generate_tests hook, and displays all test information correctly without adding any extra code.

For tests with parameterized fixtures, the report shows the original values passed to the fixtures.

Test name interpolation

You can pass Pytest parameters to the test title that appears in Allure Report with @allure.title:

import pytest
import allure

@pytest.mark.parametrize("username", ["j_doe", "admin"])
@allure.title("Check {username} can log in")
def test_login(username):
    # test logic

The formatting rules for @allure.title are the same as for str.format().

Adding dynamic parameters

You can manually add parameters and values for the test cases in Allure Report with the allure.dynamic.parameter() function:

def test_sum():
    x = 1
    y = 2
    ttl = 3
    allure.dynamic.parameter("x", x)
    allure.dynamic.parameter("y", y)
    allure.dynamic.parameter("sum", ttl)

    assert add(x, y) == ttl

The function adds a new parameter to the report, but it does not change or hide the existing parameters.

allure.dynamic.parameter() also affects the test history and retries. By default, Allure Report separates all retries and history of a parameterized test based on its parameter values.

Sometimes, you may want to avoid this behavior for a new parameter. To do so, set excluded to True. This is useful if you add a parameter, whose value changes on each run. The next example adds a timestamp to a test:

from datetime import datetime

def test_sum():
    x = 1
    y = 2
    ttl = 3
    allure.dynamic.parameter("Timestamp", str(datetime.now()), excluded=True)

    assert add(x, y) == ttl

Without excluded, each new run would create a new entry in the report. See more info here.

Another useful argument of allure.dynamic.parameter is mode. It accepts allure.parameter_mode.HIDDEN or allure.parameter_mode.MASKED values.

The allure.parameter_mode.HIDDEN mode hides the parameter from Allure Report. Note, that the parameter can still be extracted from the allure_results directory. A common use case is to hide a parameter whose only purpose is to separate test results obtained from different test environments. You can learn more here.

The allure.parameter_mode.MASKED mode obfuscates the parameter value in Allure Report. Use this mode for passwords, tokens, and other sensitive parameters.

Dealing with large values

When you parameterize tests, the values may become very large. For example, you may have:

@pytest.mark.parametrize(["user"], [
    # You may have a larger object here
    { "name": "John", "surname": "Doe", "login": "j_doe", "role": "admin" },
])
def test_user_login(user):
    login = user["login"]
    # test logic

Large parameterized values may cause problems: they can make the test report take much more disk space and are difficult to read.

To avoid these problems, we recommend parameterizing with the map keys and look up the values in the test functions. We also recommend parameterizing fixtures as they allow custom initialization. For example:

USERS = {
    "j_doe": { "name": "John", "surname": "Doe", "login": "j_doe", "role": "admin" },
}

@pytest.fixture(params=USERS.keys())
def user(request):
    # You can add a custom initialization here (for example, load data from DB ro something else)
    yield USERS[request.param]

def test_user_login(user):
    login = user["login"]

    # If we want to display more data about the user, we should add it as an `excluded` Allure parameter
    allure.dynamic.parameter("Full name", f"{user["name"]} {user["surname"]}", excluded=True)
    # test logic

4. Conclusion

Allure Report natively supports parameterization in Pytest, requiring no additional configuration or extra code. This testing technique enables the execution of the same test with multiple data sets, eliminating redundancy and avoiding the complexity of manually looping through test cases.

With parameterization, tests become more structured, readable, and easier to maintain. Additionally, Allure seamlessly captures and displays parameterized test runs in its reports, making it simple to analyze results and identify failures across different inputs. This improves both test coverage and debugging efficiency.