Visual analytics
A test report generated by Allure displays some data in the form of graphs. When someone is familiar with your project and looks at its test reports regularly, they can get a rough idea of the project's current status by just glancing at the latest test report.
Here are some of the tricks a reader can adopt in a well-organized project:
- The more green color there is on the Overview tab, the better: see the Test statuses, the Suites and Features by stories, and the Categories graphs.
- The less critical are the severity values for the failed tests, the better: see the Severity graph.
- The faster most tests run, the better: see the Duration and the Duration trend graphs.
- The fewer retries the runner has to make to complete all the tests, the better: see the Retries trend graph.
- Unless the tests have been rewritten, there should be no significant changes on the Categories trend graph.
Test statuses graph
Both the Overview tab and the Graphs tab present a pie diagram showing how many tests got each status.
The number in the center of the pie chart indicates the percentage of test cases that passed successfully.
Hover the mouse pointer over a section of the pie chart to see both the percentage and the absolute number of tests with the corresponding status.
Suites and Features by stories
The Overview tab contains the Suites and Features by stories blocks that present groups of tests from the Suite-based hierarchy and Behavior-based-hierarchy, respectively.
Each group (a suite or a feature) is shown as a horizontal line, split into sections of different colors. Sections represent numbers of test with a certain status, with the colors having the same meanings as on the test statuses graph. The number on a section indicates how many tests in the group ended with the corresponding status.
In each block, only up to ten most attention-demanding groups (suites or features) are displayed. To choose which groups to show, Allure sorts them by the number of failed tests, then by the number of broken tests.
Click on a suite or a story to open it in the Suites or Behaviors tab. Click Show all to open the Suites or Behaviors tab and explore the entire hierarchy.
Trend graphs
The trend graphs on the Overview and Graphs tabs show how a certain value changed over time. Each vertical line corresponds to a certain version of the test report, with the last line on the right corresponding to the current version.
There are different trend graphs on the Graphs tab of an Allure report.
- The Trend graph (also shown on the Overview tab) shows how many tests had certain statuses. The colors that indicate tests with each status are the same as in the test statuses graph.
- The Duration trend graph shows how long it took to run all the tests. See also: Duration graph.
- The Retries trend graph shows the total number of retries the test runner did for all the tests.
- The Categories trend graph shows the number of test defects and product defects. See also: Categories graph.
Click on a vertical line to open a previous version of the test report.
See Tests history to learn how to build test reports with the trend graphs enabled.
Severity graph
The Graphs tab contains the severity graph which shows number of tests, grouped by both their statuses and severity. If the Severity metadata field is not explicitly set for a test, the test is treated as having the “normal” severity level and is displayed in the middle section, while the tests with other severity levels are placed to the left or right of it.
The graph uses a logarithmic scale to make it easier to distinguish columns that represent similar values.
In some projects, it may be considered acceptable to have some number of “normal”, “minor” or “trivial” test failures, but not acceptable to have any “critical” or “blocker” ones. To roughly understand the current stability in such a project, a reader can just check how the different test statuses are distributed across the graph. As a general rule, the farther a red bar is placed to the left, the more attention it needs.
The test statuses colors have the same meanings as on the test statuses graph.
Hover the mouse pointer over a column to see both the names of the tests with the corresponding status and severity.
Duration graph
The Graphs tab contains the duration graph, where all the tests are divided into groups based on how long it took to complete them.
The graph uses a logarithmic scale to make it easier to distinguish columns that represent similar values.
If your project does not have any tests that take long time by design, this graph often displays the largest groups of tests to the left. If a test takes longer than others and thus is placed into the groups to the right, this may be an indicator of a problem within the test or even the product feature it works with: for example, an attempt to load a resource from a slow server. Also, a test's long duration is caused by a large number of retries.
Note, however, that sometimes a test's duration can be affected by various external factors, such as the amount of free operating memory on the machine. It is usually a good idea to run the tests again, if possible, to confirm that a test's long duration is a persistent problem.
Hover the mouse pointer over a column to see both the names of the tests with durations from the corresponding interval.
Categories graph
The Graphs tab contains the Categories block that present categories of tests. This includes the custom categories, as well as the predefined categories “Product defects” and “Test defects”, assigned based on test statuses.
Each category is shown as a horizontal line, split into sections of different colors. Sections represent numbers of test with a certain status, with the colors having the same meanings as on the test statuses graph. The number on a section indicates how many tests in the category ended with the corresponding status.
The line that correspond to the predefined categories are not split into sections, since each test in such a category has the same status.
Click on a category to view it in the Categories tab. Click Show all to open the Categories tab and explore the entire hierarchy.
