Improving readability of your test reports

The goal of Allure is to help you make test reports as easy to understand as possible. Ideally, when your colleague sees a test failure in the report, we want them to quickly understand the context: how serious the problem can be, what features are affected by it, and who can help them with investigating it further and eventually fixing it.

While sometimes the existence of the failure itself is enough to make some conclusions, it is not always the case. And even if the context seems obvious at the time of writing the test, the details may become forgotten when it fails after a few years of stable work.

For that reason, it is a good practice to fill tests with additional information, describing everything that a future reader may need.

With the Allure adapter for your test framework, you can:

• provide description, links and other metadata;
• describe parameters used when running parametrized tests;
• divide the test into smaller, easier-to-read test steps;
• provide arbitrary environment information for the whole test report.

Some Allure adapters may support additional features, see their respective documentation.

Description, links and other metadata

You can assign some fields that will help reader understand what a test does.

Title

A human-readable title of the test.

If not provided, the function name is used instead.

Description

An arbitrary text describing the test in more details than the title could fit.

If not provided, some Allure adapters may automatically extract description from the programming language's native documentation comments. See the specific Allure adapter's documentation for more details.

The description will be treated as a Markdown text, so you can you some basic formatting in it. HTML tags are not allowed in such a text and will be removed when building the report.

Allure Report has the internal support for HTML-formatted descriptions, and some adapters may provide functions to use it. But please remember that doing so is unsafe and can potentially break the test report's layout or even create an XSS attack vector for your organization.

Tags

Any number of short terms the test is related to. Usually it's a good idea to list relevant features that are being tested.

Severity

A value indicating how important the test is. This may give the future reader an idea of how to prioritize the investigations of different test failures.

Allowed values are: “trivial”, “minor”, “normal”, “critical”, and “blocker”.

Owner

The team member who is responsible for the test's stability. For example, this can be the test's author, the leading developer of the feature being tested, etc.

Links

List of links to webpages that may be useful for a reader investigating a test failure. You can provide as many links as needed.

There are three types of links:

• a standard web link, e.g., a link to the description of the feature being tested;
• a link to an issue in the product's issue tracker;
• a link to the test description in a test management system (TMS).

It is recommended to configure the Allure adapter in such a way that it accepts short identifiers of issues and TMS links and uses a URL templates to generate full URLs. For example, an identifier BUG-123 can automatically turn into https://bugs.example.com/BUG-123. You can define your own types of links with their own URL templates. See the specific Allure adapter's documentation for more details.

ID

Unique identifier of the test.

The ID can be used by additional tools such as Allure TestOps.

Other labels

Other key-value pairs.

The labels can be used by additional tools such as Allure TestOps.

Parametrized tests

Sometimes you want to run the same test multiple times, each with different set of values. For instance, you may want to make sure that the system behavior does not change whether some string is empty or not. The pattern for writing tests with common logic but different values is called parametrized tests.

The recommended way to implement this pattern depends on the test framework.

Allure collects and displays data about the parameters where possible, while also providing functions for you to manually override parameter names and values for the report. This has multiple benefits for the readability of the report:

• You may customize how Allure displays the parameters in the report — for example, make one of the parameters hidden.

• If a test parameter is a complex object that is not displayed correctly, you can customize the value for the report specifically.

• If two tests are very similar to each other, even though their implementations are independent, you can assign them identical titles and different parameters to help reader understand both the similarities and the differences of the two cases.

  Note, however, that when test implementations differ significantly, it is usually better to organize them into suites, behaviors, or packages instead.

Test steps

When a test is long and complex enough, it may become harder to investigate what caused a failure. In such a case, it is recommended to divide the test into smaller steps. In the report, Allure visualizes the steps as an easy-to-read list and highlights the failed steps. It is also possible to create sub-steps within steps, thus forming an expandable tree.

Allure does not require a test to fail after one of its steps failed. For example, you may implement a test which makes multiple attempts to perform an action and does not fail as long as one of them eventually succeeds. In the code, this is usually implemented by catching an exception thrown by the failed step.

Depending on the programming language and the test framework, Allure can provide one or more ways to define test steps and sub-steps. See the specific Allure adapter's documentation for more details.

In general, there are two different approaches:

• Test steps as separate functions.

  This approach helps to avoid name conflicts between variables used in different steps. Also, this allows different tests to call the same methods as steps, e.g., two tests can share a preparation routine.

• Test steps as parts of one function.

  This approach involves using lambdas or similar language features. If the individual steps are small, this approach often leads to a short and self-explanatory code of the test.

Both approaches are interchangeable and can be used simultaneously in the same test. For example, you may split your test into large functions, each one further split into sub-steps.

Environment information

For the main page of the report, you can collect various information about the environment in which the tests were executed.

For example, it is a good idea to use this to remember the OS version, programming language version, etc. This may help the future reader investigate bugs that are reproducible only in some environments.

To provide such information, put a special environment file into the test results directory after running the tests. The file will be used when building the report by the allure command, see Allure command-line options. Some Allure adapters can generate this file automatically based on their configuration, see the specific adapter's documentation for more details.

Note that this feature should be used for properties that do not change for all tests in the report. If you have properties that can be different for different tests, consider using Parametrized tests.