Allure Report Glossary

Test Case

A test case is a step-by-step instruction that defines how some functionality is tested.

• A test case is usually represented by a function.
• It can be parametrized.

The following example defines two test cases:

• test_authentication is parametrized
• test_signin_available is parameterless

@pytest.mark.parametrize(["login", "role], [
    ("John", "user"),
    ("Jane", "admin"),
])
def test_authentication(login, role):
    """Checks if a user with a specific role can sign in."""
    # ...

def test_signin_available():
    """Checks a sign page can be accessed."""
    # ...

Test

A test is a specific sequence of actions that can run independently and produces a single test result.

• A test case with no parameters produces exactly one test.
• A parametrized test case produces one test per parameter combination.

Our example defines three tests. Two are part of the test_authentication test case. The remaining one is the single test in the test_signin_available test case:

• test_authentication (login is "John", role is "user")
• test_authentication (login is "Jane", role is "admin)
• test_signin_available

@pytest.mark.parametrize(["login", "role], [
    ("John", "user"),
    ("Jane", "admin"),
])
def test_authentication(login, role):
    """Checks if a user with a specific role can sign in."""
    # ...

def test_signin_available():
    """Checks a sign page can be accessed."""
    # ...

Test Result

A test result is the object containing detailed information about a test's execution: the outcome, duration, steps, errors, attachments, etc.

• Allure Report converts any test results generated by test frameworks into its own format.
• An Allure test result maps to a single ...-result.json file, with optional attachment files.

Consider the the following example:

@pytest.mark.parametrize(["login", "role], [
    ("John", "user"),
    ("Jane", "admin"),
])
def test_authentication(login, role):
    """Checks if a user with a specific role can sign in."""
    # ...

def test_signin_available():
    """Checks a sign page can be accessed."""
    # ...

If we run the tests defined here, and require any failed tests to be retried several times until they pass, we would end up with a collection of test results that looks like this:

test case test_authentication (parameters: login, role)
├── test (login = "John", role = "user")
│   ├── test result (attempt 1, failed)
│   └── test result (attempt 2, passed) ← retry after failure
└── test (login = "Jane", role = "admin")
    └── test result (attempt 1, passed)

test case test_signin_available (no parameters)
└── test
    ├── test result (attempt 1, failed)
    ├── test result (attempt 2, failed) ← retry after failure
    └── test result (attempt 3, passed) ← retry after failure