History and retries

We know that one of the reasons regular testing is important is because it helps understand how the quality of your product is changing over time. If a certain test fails now, it is useful to know whether it also failed yesterday or even last week.

In Allure Report, you can navigate between multiple runs of the same test — both across different reports via Tests history and within a single one via Retries.

Tests history

Each report includes test results collected during a new test launch. However, the tests themselves are usually not new at all: you most likely use the same test code each time, slightly updating it only when necessary. Using each test's unique identifier, Allure Report is able to link its latest result to the results of the same test from previous launches. This collection of links is known as the tests history.

In a report with the tests history included, you can:

• see what statuses a test had previously (see History tab),
• find tests that changed their status since last report (see Filter tests by marks),
• see how various metrics change over time (see Trend graphs).

For most CI servers that we support, a corresponding Allure Report plugin will automatically keep the history and let you seamlessly navigate to previous test report. The same can be done when building reports locally, too, see How to enable history below.

How to enable history

Each time Allure generates a test report (see How it works), it also generates data into the history subdirectory. The JSON files in this subdirectory store all the information that Allure needs to include this test report's data in the next test report, see History files.

Enabling the history-related features is a matter of copying the history subdirectory into the test results directory before generating the next test report.

Here is an example of how to do it, assuming that your project is configured to use allure-results and allure-report directories:

1. Make sure you have the previous report generated in the allure-report directory.

2. Remove the allure-results directory.

3. Run the tests.

4. Copy the allure-report/history subdirectory to allure-results/history.

5. Generate the new report.

If you follow such a routine regularly, Allure will each time keep data from up to 20 latest reports in allure-results/history. This creates up to 20 history entries for each test and up to 20 columns in each graph.

Retries

When you run the same test multiple times, Allure produces a separate test result file for each run. The test report then includes the status and other information from the latest file, with the other runs shown as retries.

A common reason to retry a test is if it failed for some environment-specific reasons (poor internet connection, no disk space available, etc.) and you need to run it again for more informative results. Some test frameworks may even retry failed tests automatically. In these cases, the total number of retries can become a useful metric in itself — you can see if it grows or declines on one of the Trend graphs.

Another case is when one or more tests are known to be unstable, but it is still important to differentiate between the “fails always” and “fails sometimes” states. In Allure Report, you can find all tests that have changed their status after a retry using a special filter, see Filter tests by marks.

Note that unlike the latest attempt at each test, the retries do not get included in the Tests history.

How to keep retries

To make sure the information about retries gets added to the test report, just avoid removing your test results directory (which is usually called allure-results). If the Allure adapter for the framework that you use has an option to clean the directory before each run, disable that option. Then, when building a report, Allure Report will include all the files present in the directory, i.e., all the tests runs since you last cleaned the directory.

Note, however, that keeping all the retries can skew the data for the Retries trend graph. The Allure Report plugins for CI servers always run tests with an empty allure-results directory, so that the report only includes automatic retries done by the test framework.

Making sure tests are identified correctly

For a test result to be linked to its history or to be placed into a collection of retries, it must be correctly assigned an internal identifier called “history ID”. Allure Report relies on the Allure adapter for the specific framework to assign such an identifier to each test result. With some limitations and exceptions, each adapter assigns the history ID based on two factors:

• the test's fully-qualified name,
• the test's non-excluded parameters (see below).

To ensure that the test report will properly deal with the history and retries, make sure to pass to Allure all parameters that distinguish different versions of a test. Most Allure adapters take care of it automatically, but some may need additional code, such as Allure::parameter() for Allure PHPUnit or @TestInstanceParameter for some configurations in Allure TestNG.

Common pitfall: tests from different environments are missing in the report

Let's assume that you run a test on multiple platforms before collecting all the results into a single allure-results directory and building the report. Unless you make sure that different runs get different sets of parameters, Allure Report will show only one test result by default, moving other ones to its list retries.

We recommend adding one or more special parameters that will make up a different combination for each environment. For example, you may want to add such parameters as “os_version”, “browser_version”, “java_version”, etc., depending on the technologies used by your test.

Be sure to check the “Parametrized tests” section of your Allure adapter's documentation to see if it has any limitations. When an adapter has multiple ways of declaring test parameters, it may or may not support all of them for the purposes of identifying a test. For example, Allure JUnit 5 and some other adapters can not distinguish tests that only differ in “dynamic” parameters, i.e., the parameters that were set after the test execution started.

Common pitfall: a test's retries are displayed as separate tests

Normally, when you run a test with a set of parameters different from the previous times, Allure Report treats it as a new test. This means the runs are displayed in the test report separately instead of being grouped into a list of a single test's retries. Additionally, Allure may fail to find links to the previous results in the test's history.

For example, if a parameter derives from current date and time or from a randomized value, you will get seemingly duplicated test results when you run the test more than once.

If this is not the desired behavior for your test, consider removing such a parameter or marking it as excluded. See your adapter's reference to find out if it allows marking parameters excluded and how to do it.