Improving readability of your test reports

The goal of Allure is to help you make test reports as easy to understand as possible. Ideally, when your colleague sees a test failure in the report, we want them to quickly understand the context: how serious the problem can be, what features are affected by it, and who can help them with investigating it further and eventually fixing it.

While sometimes the existence of the failure itself is enough to make some conclusions, it is not always the case. And even if the context seems obvious at the time of writing the test, the details may become forgotten when it fails after a few years of stable work.

For that reason, it is a good practice to fill tests with additional information, describing everything that a future reader may need.

With the Allure adapter for your test framework, you can:

Some Allure adapters may support additional features, see their respective documentation.

You can assign some fields that will help reader understand what a test does.

Title

A human-readable title of the test.

If not provided, the function name is used instead.

Allure Report title

Description

An arbitrary text describing the test in more details than the title could fit.

If not provided, some Allure adapters may automatically extract description from the programming language's native documentation comments. See the specific Allure adapter's documentation for more details.

The description will be treated as a Markdown text, so you can you some basic formatting in it. HTML tags are not allowed in such a text and will be removed when building the report.

Allure Report has the internal support for HTML-formatted descriptions, and some adapters may provide functions to use it. But please remember that doing so is unsafe and can potentially break the test report's layout or even create an XSS attack vector for your organization.

Allure Report description

Tags

Any number of short terms the test is related to. Usually it's a good idea to list relevant features that are being tested.

Allure Report tags

Severity

A value indicating how important the test is. This may give the future reader an idea of how to prioritize the investigations of different test failures.

Allowed values are: “trivial”, “minor”, “normal”, “critical”, and “blocker”.

Allure Report severity

Owner

The team member who is responsible for the test's stability. For example, this can be the test's author, the leading developer of the feature being tested, etc.

Allure Report owner

List of links to webpages that may be useful for a reader investigating a test failure. You can provide as many links as needed.

There are three types of links:

  • a standard web link, e.g., a link to the description of the feature being tested;
  • a link to an issue in the product's issue tracker;
  • a link to the test description in a test management system (TMS).

It is recommended to configure the Allure adapter in such a way that it accepts short identifiers of issues and TMS links and uses a URL templates to generate full URLs. For example, an identifier BUG-123 can automatically turn into https://bugs.example.com/BUG-123. You can define your own types of links with their own URL templates. See the specific Allure adapter's documentation for more details.

Allure Report links

ID

Unique identifier of the test.

The ID can be used by additional tools such as Allure TestOps.

Other labels

Other key-value pairs.

The labels can be used by additional tools such as Allure TestOps.

Parametrized tests

Sometimes you want to run the same test multiple times, each with different set of values. For instance, you may want to make sure that the system behavior does not change whether some string is empty or not. The pattern for writing tests with common logic but different values is called parametrized tests.

The recommended way to implement this pattern depends on the test framework.

Allure Report tests with parameters

Allure collects and displays data about the parameters where possible, while also providing functions for you to manually override parameter names and values for the report. This has multiple benefits for the readability of the report:

  • You may customize how Allure displays the parameters in the report — for example, make one of the parameters hidden.

  • If a test parameter is a complex object that is not displayed correctly, you can customize the value for the report specifically.

  • If two tests are very similar to each other, even though their implementations are independent, you can assign them identical titles and different parameters to help reader understand both the similarities and the differences of the two cases.

    Note, however, that when test implementations differ significantly, it is usually better to organize them into suites, behaviors, or packages instead.

Test steps

When a test is long and complex enough, it may become harder to investigate what caused a failure. In such a case, it is recommended to divide the test into smaller steps. In the report, Allure visualizes the steps as an easy-to-read list and highlights the failed steps. It is also possible to create sub-steps within steps, thus forming an expandable tree.

Allure Report test steps

Allure does not require a test to fail after one of its steps failed. For example, you may implement a test which makes multiple attempts to perform an action and does not fail as long as one of them eventually succeeds. In the code, this is usually implemented by catching an exception thrown by the failed step.

Allure Report failed test steps

Depending on the programming language and the test framework, Allure can provide one or more ways to define test steps and sub-steps. See the specific Allure adapter's documentation for more details.

In general, there are two different approaches:

  • Test steps as separate functions.

    This approach helps to avoid name conflicts between variables used in different steps. Also, this allows different tests to call the same methods as steps, e.g., two tests can share a preparation routine.

  • Test steps as parts of one function.

    This approach involves using lambdas or similar language features. If the individual steps are small, this approach often leads to a short and self-explanatory code of the test.

Both approaches are interchangeable and can be used simultaneously in the same test. For example, you may split your test into large functions, each one further split into sub-steps.

Screenshots and other files

When you are investigating an unexpected test failure, sometimes it is convenient to not just read the list of completed steps, but also see the same data the test saw. For example, if you could see a screenshot of the web page just before the failure, you often could quickly understand whether it looks the way it was expected by the test author.

Some test frameworks allow you to attach any kind of files to a test result. The Allure adapter for each framework either makes use of these attachments or, if such functionality is not available, implements it from scratch.

You can attach as many screenshots and files as you need. In the test report, you can see all attached files next to the list of test steps, see How to read a test report.

Allure Report display attachments

Environment information

For the main page of the report, you can collect various information about the environment in which the tests were executed.

For example, it is a good idea to use this to remember the OS version, programming language version, etc. This may help the future reader investigate bugs that are reproducible only in some environments.

Allure Report test run environment

To provide such information, put a special environment file into the test results directory after running the tests. The file will be used when building the report by the allure command, see Allure command-line options. Some Allure adapters can generate this file automatically based on their configuration, see the specific adapter's documentation for more details.

Note that this feature should be used for properties that do not change for all tests in the report. If you have properties that can be different for different tests, consider using Parametrized tests.

Powered by
logo

Join our newsletter

Join our community

We aim to make Allure Report as reliable and user-friendly as possible, and together with the community, we're here to help when problems arise.

© 2024 Qameta Software Inc. All rights reserved.