Graphs

A test report generated by Allure displays some data in the form of graphs. When someone is familiar with your project and looks at its test reports regularly, they can get a rough idea of the project's current status by just glancing at the latest test report.

Here are some of the tricks a reader can adopt in a well-organized project:

Test statuses graph

Both the Overview tab and the Graphs tab present a pie diagram showing how many tests got each status.

Allure Report test statuses graph

The number in the center of the pie chart indicates the percentage of test cases that passed successfully.

Hover the mouse pointer over a section of the pie chart to see both the percentage and the absolute number of tests with the corresponding status.

Suites and Features by stories

The Overview tab contains the Suites and Features by stories blocks that present groups of tests from the Suite-based hierarchy and Behavior-based-hierarchy, respectively.

Allure Report suites graph

Allure Report stories graph

Each group (a suite or a feature) is shown as a horizontal line, split into sections of different colors. Sections represent numbers of test with a certain status, with the colors having the same meanings as on the test statuses graph. The number on a section indicates how many tests in the group ended with the corresponding status.

In each block, only up to ten most attention-demanding groups (suites or features) are displayed. To choose which groups to show, Allure sorts them by the number of failed tests, then by the number of broken tests.

Click on a suite or a story to open it in the Suites or Behaviors tab. Click Show all to open the Suites or Behaviors tab and explore the entire hierarchy.

Trend graphs

The trend graphs on the Overview and Graphs tabs show how a certain value changed over time. Each vertical line corresponds to a certain version of the test report, with the last line on the right corresponding to the current version.

Allure Report trend graph

There are different trend graphs on the Graphs tab of an Allure report.

  • The Trend graph (also shown on the Overview tab) shows how many tests had certain statuses. The colors that indicate tests with each status are the same as in the test statuses graph.
  • The Duration trend graph shows how long it took to run all the tests. See also: Duration graph.
  • The Retries trend graph shows the total number of retries the test runner did for all the tests.
  • The Categories trend graph shows the number of test defects and product defects. See also: Categories graph.

Click on a vertical line to open a previous version of the test report.

See Tests history to learn how to build test reports with the trend graphs enabled.

Severity graph

The Graphs tab contains the severity graph which shows number of tests, grouped by both their statuses and severity. If the Severity metadata field is not explicitly set for a test, the test is treated as having the “normal” severity level and is displayed in the middle section, while the tests with other severity levels are placed to the left or right of it.

The graph uses a logarithmic scale to make it easier to distinguish columns that represent similar values.

Allure Report severity graph

In some projects, it may be considered acceptable to have some number of “normal”, “minor” or “trivial” test failures, but not acceptable to have any “critical” or “blocker” ones. To roughly understand the current stability in such a project, a reader can just check how the different test statuses are distributed across the graph. As a general rule, the farther a red bar is placed to the left, the more attention it needs.

The test statuses colors have the same meanings as on the test statuses graph.

Hover the mouse pointer over a column to see both the names of the tests with the corresponding status and severity.

Duration graph

The Graphs tab contains the duration graph, where all the tests are divided into groups based on how long it took to complete them.

The graph uses a logarithmic scale to make it easier to distinguish columns that represent similar values.

Allure Report duration graph

If your project does not have any tests that take long time by design, this graph often displays the largest groups of tests to the left. If a test takes longer than others and thus is placed into the groups to the right, this may be an indicator of a problem within the test or even the product feature it works with: for example, an attempt to load a resource from a slow server. Also, a test's long duration is caused by a large number of retries.

Note, however, that sometimes a test's duration can be affected by various external factors, such as the amount of free operating memory on the machine. It is usually a good idea to run the tests again, if possible, to confirm that a test's long duration is a persistent problem.

Hover the mouse pointer over a column to see both the names of the tests with durations from the corresponding interval.

Categories graph

The Graphs tab contains the Categories block that present categories of tests. This includes the custom categories, as well as the predefined categories “Product defects” and “Test defects”, assigned based on test statuses.

Allure Report categories graph

Each category is shown as a horizontal line, split into sections of different colors. Sections represent numbers of test with a certain status, with the colors having the same meanings as on the test statuses graph. The number on a section indicates how many tests in the category ended with the corresponding status.

The line that correspond to the predefined categories are not split into sections, since each test in such a category has the same status.

Click on a category to view it in the Categories tab. Click Show all to open the Categories tab and explore the entire hierarchy.

Timeline

The graph on the Timeline tab shows the order and the durations of all tests' executions.

Allure Report Timeline graph

When a test framework runs tests in parallel, it does so using a set of worker processes. Each worker can only run one test at a time, but then the worker can be re-used to run another test. In the graph, each row represents a single worker process, and each rectangle on it represents a single test's execution. The color of each rectangle represents the test status, with the colors having the same meanings as on the test statuses graph.

Move the thumb of the slider above the graph to show only those tests that took longer than a certain threshold.

Move the thumbs of the range slider below the graph to zoom in to a time interval. The labels near the range slider indicate time passed since the beginning of the first test's execution.

Hover the mouse pointer over a rectangle to see the test's title and the time its execution started and finished (expressed in seconds and milliseconds since the beginning of the first test's execution). Click on a rectangle to open the test details.

You can find the causes of various test failures just by looking at the timeling. For example:

  • When some tests are significantly slower than the others, the corresponding rectangles on the graph will be the largest.

    Allure Report Timeline durations

  • If your tests work on a web resource, and the resource goes offline for a while, a lot of seemingly unrelated tests may fail simultaneously. Such a situation can manifest itself as a set of failed (red) tests in one area.

    Allure Report Timeline period

  • Some test frameworks wait for all running tests to finish before attempting to run the failed ones again. This usually leads to noticeable areas of almost blank space.

    Allure Report Timeline retries

  • A test framework may spend too much time loading the environment before each test. This leads to wide horizontal gaps between tests.

    Allure Report Timeline gaps

Powered by
logo

Join our newsletter

Join our community

We aim to make Allure Report as reliable and user-friendly as possible, and together with the community, we're here to help when problems arise.

© 2024 Qameta Software Inc. All rights reserved.