Attachments

When you are investigating an unexpected test failure, sometimes it is convenient to not just read the list of completed steps, but also see the same data the test saw. For example, if you could see a screenshot of the web page just before the failure, you often could quickly understand whether it looks the way it was expected by the test author.

With Allure, you can attach any kinds of files to the whole test result or to the current test step or fixture (depending on the Allure adapter implementation).

In the report, Allure will provide both a link for downloading the attachment and a preview for supported media types.

Screenshots

Media types: image/bmp, image/gif, image/jpeg, image/png, image/svg+xml, image/tiff, image/*.

Whenever a test interacts with a graphical interface (whether it is a desktop app, a mobile app, or a website), attaching screenshots to the test report increases its informativeness significantly. Just glancing at a screenshot answers a lot of questions, like “Is the website loaded properly?”, “Is a form filled correctly?”, “Aren't there any unexpected popups in the way?”, etc.

• A common approach is to take a screenshot as soon as you detect a failure — this is usually when the reader has the most questions. In a lot of programming languages, you can perform post-failure operations using a catch or except statement — these are great places to attach screenshots for the report.

• If you don't mind using some more disk space, you can help the reader even more by taking screenshots on each step of your test, even before a failure. For complex test scenarios, this will help understand where exactly something went wrong, without resorting to manual testing.

  Note that in some situation, it may be even more informative to attach a video.

The functions for taking screenshots are different in each test framework, and some Allure adapters may attach each taken screenshot automatically. Generally, though, you need to get either a filename of the screenshot or its content as raw bytes and then use an Allure adapter's function to attach it to the report.

Visual comparisons

Show diff

Show overlay

---

Media type: application/vnd.allure.image.diff.

A visual comparison, also known as a “visual diff”, is a way to check if one state of a user interface is identical to another state or to an expected state. The comparison is done by analyzing two screenshots on a pixel-by-pixel basis. If the number of pixels that do not match is higher than a certain threshold, it is considered a failure. Visual comparisons can be performed by various test frameworks or additional utilities, such as pixelmatch.

It is a good idea to attach the result of a visual comparison to the test report for a human inspection. With Allure Report, the reader will be able to switch between two display modes for each visual comparison:

• Show diff — view the visual comparison, as produced by the test framework or utility,
• Show overlay — view the two original images on both sides of the mouse pointer.

Some Allure adapters provide functions for attaching visual comparisons.

INFO

If the adapter for your framework does not provide such a function yet, follow the steps below to construct the necessary data. You will need to use your programming language's JSON- and Base64-related functions.

1. Prepare all three images: the expected screenshot, the actual screenshot, and a visual comparison image.
2. Convert each image into a Base64 data URL, beginning with data:image/png;base64,.
3. Put the three strings into a JSON object under the keys expected, actual, and diff.
4. Encode the JSON object into a string or a byte array, depending on the Allure adapter.
5. Attach the data and specify the application/vnd.allure.image.diff media type.

JavaScript (Mocha)

const { allure } = require("allure-mocha/runtime");
const fs = require("fs");

it("attachment", () => {
  // ...

  // Read the three files and encode to base64
  const expected = fs.readFileSync("expected.png", "base64");
  const actual = fs.readFileSync("actual.png", "base64");
  const diff = fs.readFileSync("diff.png", "base64");

  // Wrap in a JSON, encode as string
  const content = JSON.stringify({
    expected: `data:image/png;base64,${expected}`,
    actual: `data:image/png;base64,${actual}`,
    diff: `data:image/png;base64,${diff}`,
  });

  // Attach to the test report
  allure.attachment("Screenshot diff", content, "application/vnd.allure.image.diff");
});

Java (JUnit 5)

import io.qameta.allure.Allure;
import org.json.JSONObject;
import org.junit.jupiter.api.Test;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Base64;

class TestImageDiff {

    @Test
    void testImageDiff() throws IOException {
        // ...

        // Read the three files as bytes
        byte[] expected = Files.readAllBytes(Paths.get("expected.png"));
        byte[] actual = Files.readAllBytes(Paths.get("actual.png"));
        byte[] diff = Files.readAllBytes(Paths.get("diff.png"));

        // Encode the data, wrap in a JSON, encode as bytes
        String content = new JSONObject()
                .put("expected", "data:image/png;base64,"
                        + Base64.getEncoder().encodeToString(expected))
                .put("actual", "data:image/png;base64,"
                        + Base64.getEncoder().encodeToString(actual))
                .put("diff", "data:image/png;base64,"
                        + Base64.getEncoder().encodeToString(diff))
                .toString();

        // Attach to the test report
        Allure.addAttachment("Screenshot diff",
                "application/vnd.allure.image.diff",
                content);
    }
}

Python (pytest)

import json
import base64
from pathlib import Path

import allure


def test_attach_image_diff():
    ...

    # Read the three files and encode to base64
    expected = base64.b64encode(Path('expected.png').read_bytes()).decode()
    actual = base64.b64encode(Path('actual.png').read_bytes()).decode()
    diff = base64.b64encode(Path('diff.png').read_bytes()).decode()

    # Wrap in a JSON, encode as bytes
    content = json.dumps({
        'expected': f'data:image/png;base64,{expected}',
        'actual': f'data:image/png;base64,{actual}',
        'diff': f'data:image/png;base64,{diff}',
    }).encode()

    # Attach to the test report
    allure.attach(content,
                  name='Screenshot diff',
                  attachment_type='application/vnd.allure.image.diff')

C# (NUnit)

using Allure.Net.Commons;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestImageDiff
{
    [Test]
    public void Test_ImageDiff()
    {
        // ...
        AllureApi.AddScreenDiff("expected.png", "actual.png", "diff.png");
    }
}

Videos

Media types: video/mp4, video/ogg, video/webm.

Some test frameworks and libraries provide means to record videos of what happens in a window or on a webpage. Attaching such a video to the test report serves a purpose similar to attaching a screenshot. The reader will be able to better understand how the system reacted to the user input or other events and what caused an unexpected behavior.

In the test report, click on an attached video to play it in the browser.

Text

Plain text

Formatted text

---

Media types: text/html, text/plain, text/*.

Attaching texts to the test report can serve various purposes — from checking texts fround in a user interface under test to storing full logs of some complex operations.

Allure supports two types of text attachments: plain text and formatted text. Formatted text allows for a limited subset of HTML tags.

Some Allure adapters may automatically attach the content from stdout and stderr as plain text attachments. Check your adapter's documentation for more details.

Tables

Media types: text/csv, text/tab-separated-values.

You can present arbitrary data as tables in Allure Report. To do so, format the data as comma-separated values (CSV) or tab-separated values (TSV) and attach it to the test result. The table will be displayed on the test result page without need to download and open the file.

URI lists

Media types: text/uri-list.

Sometimes, when testing an application, especially a web application, you may want to collect and analyze a list of links. For example, you may want to check if all links on a website are valid. By attaching the list of links to the test report, you will help the future reader inspect them again manually when investigating a failure.

The URI list format is just that — a list of URIs, one URI per line. Allure Report will make each link clickable in the test report.

Documents

Media types: text/xml, application/xml, application/json, application/yaml.

If you attach an XML, JSON, or YAML document to your test report and specify the corresponding media type, Allure Report will display the document with syntax highlighting enabled.

Other files

Allure Report does not enforce any limitations on what files can be attached to a test report, even though it cannot provide previews for all of them. For any unsupported type, a Click to download attachment link will be displayed instead of a preview.

If you believe Allure Report lacks support of an important media type, feel free to start a discussion!