Allure TestNG reference

These are the functions that you can use to integrate your TestNG tests with Allure.

Allure TestNG provides more than one way to use some features. In most cases, one way involves using an annotation on the test method, while the other involves a method call inside the test method body. The latter style is called “dynamic” and allows to construct values dynamically.

Metadata

Assign a test's description, links and other metadata.

Title

• Allure.getLifecycle().updateTestCase(Consumer<TestResult> update)

Set the test's title.

Allure recognizes the TestNG's standard testName argument as the title for the test.

If you need to construct a test's title dynamically, write a lambda function that does that and pass it to the updateTestCase() method of the AllureLifecycle object, see the example below.

Java
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test(testName = "Test Authentication")
    public void testAuthentication() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.getLifecycle().updateTestCase(result -> {
            result.setName("Test Authentication");
        });
        // ...
    }
}

Description

• @Description(String value="", boolean useJavaDoc=false)
• Allure.description(String description)

Set the test's description.

Use the @Description() annotation to set a description statically or use the description() method to set it dynamically in runtime.

Alternatively, you can let Allure JUnit 5 parse the description from the test method's JavaDoc comment. To do so, first add a special annotation processor to your project's Maven or Gradle configuration. Then, use the Annotations API with the useJavaDoc=true argument on the test methods for which you want to parse the description from JavaDoc. Note that on a large project, processing the annotations may significantly increase both the compilation time and the file size of the test executable files.

Markdown formatting is allowed. Any HTML formatting, if present, will be stripped for security purposes.

Java
import io.qameta.allure.Description;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    @Description("This test attempts to log into the website using a login and a password. Fails if any error happens.\n\nNote that this test does not test 2-Factor Authentication.")
    public void testAuthentication() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.description("This test attempts to log into the website using a login and a password. Fails if any error happens.\n\nNote that this test does not test 2-Factor Authentication.");
        // ...
    }
}

Kotlin
val allureVersion = "2.24.0"

dependencies {
    // ...
    testAnnotationProcessor("io.qameta.allure:allure-descriptions-javadoc:$allureVersion")
}

Java
import io.qameta.allure.Description;
import org.testng.annotations.Test;

public class TestMyWebsite {

    /**
     * This test attempts to log into the website using a login and a password. Fails if any error happens.
     * <p>
     * Note that this test does not test 2-Factor Authentication.
     */
    @Test
    @Description(useJavaDoc = true)
    public void testAuthentication() {
        // ...
    }
}

Owner (Allure TestOps)

• @Owner(String value="")

Set the test's owner.

Java
import io.qameta.allure.Owner;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    @Owner("John Doe")
    public void testAuthentication() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.label("owner", "John Doe");
        // ...
    }
}

Tag

Set the test's tags.

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.label("tag", "NewUI");
        Allure.label("tag", "Essentials");
        Allure.label("tag", "Authentication");
        // ...
    }
}

Severity

• @Severity(SeverityLevel value)

Set the test's severity.

Allowed values are: “trivial”, “minor”, “normal”, “critical”, and “blocker”.

Java
import io.qameta.allure.Severity;
import org.testng.annotations.Test;

import static io.qameta.allure.SeverityLevel.*;

public class TestMyWebsite {

    @Test
    @Severity(CRITICAL)
    public void testAuthentication() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.label("severity", "critical");
        // ...
    }
}

Label

• @LabelAnnotation(String name, String value=DEFAULT_VALUE)
• Allure.label(String name, String value)

Set an arbitrary label for the test. This is the underlying implementation for a lot of Allure's other functions.

To set a label dynamically, just call the label() function with a name and a value for a label. You can do it multiple times to create an array of values under that name.

Another way of setting a label is to create a custom class using @LabelAnnotation and annotate the tests with the new annotation, see the example below. Please be careful to copy @Retention, @Target and the other necessary annotations as this is shown in the example, otherwise you custom annotation may not work properly.

Java
import io.qameta.allure.LabelAnnotation;
import org.testng.annotations.Test;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Documented
@Inherited
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
@LabelAnnotation(name = "CustomLabelName")
@interface MyLabel {
    String value();
}

public class TestMyWebsite {

    @Test
    @MyLabel("custom label value")
    public void testAuthentication() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.label("language", "java");
        Allure.label("framework", "testng");
        Allure.label("CustomLabelName", "custom label value");
        // ...
    }
}

Allure ID (Allure TestOps)

• @AllureId(String value)

Set the test's ID.

Java
import io.qameta.allure.AllureId;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    @AllureId("123")
    public void testAuthentication() {
        // ...
    }
}

Link

• @Link(String value="", name="", String url="", String type=CUSTOM_LINK_TYPE)
• @Links(Link[] value)
• @Issue(String value="")
• @Issues(Issue[] value)
• @TmsLink(String value="")
• @TmsLinks(TmsLink[] value)
• Allure.link(String url)
• Allure.link(String name, String url)
• Allure.link(String name, String type, String url)
• Allure.issue(String name, String url)
• Allure.tms(String name, String url)

Add a link related to the test.

Based on the type (which can be any string), Allure will try to load a corresponding link pattern to process the URL, as defined by the allure.link.*.pattern configuration option. If no pattern found for the given type, the URL is left unmodified.

The name will be used as the link's text. If it is omitted, the unprocessed URL will be used instead.

For convenience, Allure provides two shorthand functions with pre-selected link types: issue() and tms().

Java
import io.qameta.allure.Issue;
import io.qameta.allure.Link;
import io.qameta.allure.TmsLink;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    @Link(name = "Website", url = "https://dev.example.com/")
    @Issue("AUTH-123")
    @TmsLink("TMS-456")
    public void testAuthentication() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.link("Website", "https://dev.example.com/");
        Allure.issue("AUTH-123", "https://jira.example.org/browse/AUTH-123");
        Allure.tms("TMS-456", "https://tms.example.org/TMS-456");
        // ...
    }
}

Behavior-based hierarchy

• @Epic(String value="")
• @Epics(Epic[] value)
• @Feature(String value="")
• @Features(Feature[] value)
• @Story(String value="")
• @Stories(Story[] value)
• Allure.epic(String value)
• Allure.feature(String value)
• Allure.story(String value)

Assign names of epics, features or user stories for a test, as part of Allure's behavior-based hierarchy.

Java
import io.qameta.allure.Epic;
import io.qameta.allure.Feature;
import io.qameta.allure.Story;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    @Epic("Web interface")
    @Feature("Essential features")
    @Story("Authentication")
    public void testAuthentication() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.epic("Web interface");
        Allure.feature("Essential features");
        Allure.story("Authentication");
        // ...
    }
}

Suite-based hierarchy

• Allure.suite(String value)

Assign the name of suite, as part of Allure's suite-based hierarchy.

To assign names of parent suite or sub-suite, use label() with the corresponding label names.

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        Allure.label("parentSuite" "Tests for web interface");
        Allure.suite("Tests for essential features");
        Allure.label("subSuite", "Tests for authentication");
        // ...
    }
}

Test steps

• @Step(String value="")
• Allure.step(String name)
• Allure.step(String name, Status status)
• Allure.step(String name, ThrowableRunnableVoid runnable)
• Allure.step(String name, ThrowableRunnable<T> runnable)
• Allure.step(ThrowableContextRunnableVoid<StepContext> runnable)
• Allure.step(String name, ThrowableContextRunnableVoid<StepContext> runnable)
• Allure.step(String name, ThrowableContextRunnable<T, StepContext> runnable)
• Allure.step(ThrowableContextRunnable<T, StepContext> runnable)

Define a test step with the given name.

There are three ways of defining a step.

• “An annotated step”: define a method containing a test step and decorate it with @Step().
• “A lambda step”: write a test step in a lambda function and pass it to Allure.step().
• “A no-op step”: just pass a name and an optional status to Allure.step() to immediately add a corresponding entry to the results as a sub-step of the current step.

When you are implementing a function for a lambda step, it can accept either no arguments or a single argument of class StepContext. This object provides the following methods:

• name() — override the step name during its execution.
• parameter() — indicate arbitrary parameters used for the step. The available signatures of this method are the same as for the test-wide implementation, see Parametrized tests.

Java
import io.qameta.allure.Step;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {
        step1();
        step2();
    }

    @Step("Step 1")
    public void step1() {
        subStep1();
        subStep2();
    }

    @Step("Sub-step 1")
    public void subStep1() {
        // ...
    }

    @Step("Sub-step 2")
    public void subStep2() {
        // ...
    }

    @Step("Step 2")
    public void step2() {
        // ...
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthentication() {

        Allure.step("Step 1", step -> {

            // ...
            Allure.step("Sub-step 1");

            // ...
            Allure.step("Sub-step 2");
        });

        Allure.step("Step 2", step -> {
            // ...
        });
    }
}

Parametrized tests

• @TestInstanceParameter(String value="")
• Allure.parameter(String name, T value)
• Allure.parameter(String name, T value, Boolean excluded)
• Allure.parameter(String name, T value, Parameter.Mode mode)
• Allure.parameter(String name, T value, Boolean excluded, Parameter.Mode mode)

TestNG provides multiple approaches for implementing the parametrized tests pattern. Depending on the approach you use, you may or may not need to modify your code to add the list of parameters to the test report.

Test method parameters via XML

With this approach, TestNG uses the same test method for multiple tests, each one defined by a <test> tag in the XML file. The names and values of parameters are specified in <parameter> tags and mapped to the method argument using the @Parameters annotation.

1. In a test class, define a test method with one or more arguments. Above the method signature, add the @Parameters annotation and pass to it an array of names.

   The names passed to @Parameters will be used both for mapping the values from the XML file and for displaying the parameters in the test report. These names can differ from the actual names of the method arguments.

2. In the XML file that you use for running TestNG tests, add one or more <test> tags with the same test implementation in <class>, but with different sets of values in <parameter> tags.

Java
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    @Parameters({"Login", "Password"})
    public void testAuthentication(String login, String password) {
        // ...
    }
}

XML
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Test my website">

    <test name="Test authentication with login">
        <parameter name="Login" value="johndoe"/>
        <parameter name="Password" value="qwerty"/>
        <classes>
            <class name="com.example.TestMyWebsite"/>
        </classes>
    </test>

    <test name="Test authentication with email">
        <parameter name="Login" value="[email protected]"/>
        <parameter name="Password" value="qwerty"/>
        <classes>
            <class name="com.example.TestMyWebsite"/>
        </classes>
    </test>

</suite>

Test instance parameters via XML

With this approach, TestNG instantiates a test class multiple times, each one defined by a <test> tag in the XML file. The names and values of parameters are specified in <parameter> tags and mapped to the constructor using the @Parameters annotation.

1. In a test class, define a constructor with one or more arguments. Above the constructor signature, add the @Parameters annotation and pass to it an array of names.

   The names passed to @Parameters will be used both for mapping the values from the XML file and for displaying the parameters in the test report. These names can differ from the actual names of the constructor arguments.

2. Define one or more test methods in the class.

   In the test report, the parameters defined for the constructor will be displayed for each of the test methods. If a test method has its own @Parameters annotation (as with the “Test method parameters via XML” approach), its parameters will be appended to the list for this particular method.

3. In the XML file that you use for running TestNG tests, add one or more <test> tags with the same test implementation in <class>, but with different sets of values in <parameter> tags.

Java
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class TestMyWebsite {

    private String login;
    private String password;

    @Parameters({"Login", "Password"})
    public TestMyWebsite(String login, String password) {
        this.login = login;
        this.password = password;
    }

    @Test
    public void testAuthentication() {
        // ...
    }
}

XML
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="parametrized">

    <test name="Test authentication with login">
        <parameter name="Login" value="johndoe"/>
        <parameter name="Password" value="qwerty"/>
        <classes>
            <class name="org.example.TestMyWebsite"/>
        </classes>
    </test>

    <test name="Test authentication with email">
        <parameter name="Login" value="[email protected]"/>
        <parameter name="Password" value="qwerty"/>
        <classes>
            <class name="org.example.TestMyWebsite"/>
        </classes>
    </test>

</suite>

Test method parameters via DataProvider

This approach does not involve any modifications in the XML files and instead gets both names and values of parameters from a special “data provider” method.

1. In the test class, define a data provider — a static method that returns an array or arrays of values, i.e., an Object[][]. Annotate it with @DataProvider.

2. Define a test method that accept arguments. Specify the name of the data provider in the dataProvider argument for the @Test annotation. TestNG will run this test multiple times, each time with a new set of values from the data provider.

   Allure TestNG will recognize the parameters and display them to the test report for each test method of the class.

3. Optionally, customize the parameter names via a @Parameter annotation.

Java
import org.testng.annotations.DataProvider;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @DataProvider
    public static Object[][] authenticationData() {
        return new Object[][]{
                {"johndoe", "qwerty"},
                {"[email protected]", "qwerty"},
        };
    }

    @Test(dataProvider = "authenticationData")
    @Parameters({"Login", "Password"})  // optional
    public void testAuthentication(String login, String password) {
        // ...
    }
}

Test instance parameters via DataProvider

With this approach, TestNG instantiates a test class multiple times, each one with a new set of values provided by a special “data provider” method.

1. In the test class, define a data provider — a static method that returns an array or arrays of values, i.e., an Object[][]. Annotate it with @DataProvider.

2. Define the fields for storing the values for the parameters. Add the @TestInstanceParameter annotation to each of them, with an optional argument specifying the display name of the parameter for the test report.

3. Define a constructor that accepts arguments. Above the constructor signature, add the @Factory annotation and specify the name of the data provider in its dataProvider argument.

4. Define one or more test methods in the class.

   In the test report, the parameters defined via @TestInstanceParameter will be displayed for each of the test methods. If a test method has its own @Parameters annotation (as with the “Test method parameters via XML” approach), its parameters will be appended to the list for this particular method.

Java
import io.qameta.allure.testng.TestInstanceParameter;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @TestInstanceParameter("Login")
    private String login;

    @TestInstanceParameter("Password")
    private String passwordForTests;

    @DataProvider
    public static Object[][] authenticationData() {
        return new Object[][]{
                {"johndoe", "qwerty"},
                {"[email protected]", "qwerty"},
        };
    }

    @Factory(dataProvider = "authenticationData")
    public TestMyWebsite(String login, String password) {
        this.login = login;
        this.passwordForTests = password;
    }

    @Test
    public void testAuthentication() {
        // ...
    }
}

Runtime API

Additionally, the Allure.parameter() method can be used at any point for adding the parameters even to those tests that do not use the TestNG's parametrized testing approaches.

If you have multiple tests that only differ in dynamic parameters (i.e., the Allure.parameter() calls), they will be treated as one test for the purposes of History and retries.

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

public class TestMyWebsite {

    @Test
    public void testAuthenticationWithUsername() {
        Allure.parameter("login", "johndoe");
        Allure.parameter("password", "qwerty");
        // ...
    }

    @Test
    public void testAuthenticationWithEmail() {
        Allure.parameter("login", "[email protected]");
        Allure.parameter("password", "qwerty");
        // ...
    }
}

Attachments

• @Attachment(String value="", String type="", String fileExtension="")
• Allure.addAttachment(String name, String content)
• Allure.addAttachment(String name, String type, String content)
• Allure.addAttachment(String name, String type, String content, String fileExtension)
• Allure.addAttachment(String name, InputStream content)
• Allure.addAttachment(String name, String type, InputStream content, String fileExtension)
• Allure.attachment(String name, String content)
• Allure.attachment(String name, InputStream content)

Add content as an attachment to the test result under the given name (defaults to a unique pseudo-random string).

You can use data produced by any function, not necessarily read from an actual file.

To create an attachment using the Annotations API, define a method that returns some data and annotate it with @Attachment. Call the method at any point during your test.

To create an attachment using the Runtime API, just call addAttachment() or attachment() at any point during your test. Pass the data as the content argument.

The data will be processed as following:

• If the data type is byte[], Allure will use it without modification.
• If the data type is String, Allure will convert it to byte[].
• If the data is of any other type, Allure will call the toString() method and then convert the string to byte[].

To ensure that the reader's web browser will display attachments correctly, it is recommended to specify each attachment's type. To do so, pass the MIME type of the content as type and, optionally, a filename extension as fileExtension. The MIME type affects how the data will be displayed in the test report, while the filename extension is appended to the filename when user wants to save the file.

Java
import io.qameta.allure.Attachment;
import org.testng.annotations.Test;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class TestMyWebsite {

    @Test
    public void testAuthentication() throws IOException {
        // ...
        attachDataTXT();
        attachScreenshotPNG();
    }

    @Attachment(value = "data", type = "text/plain", fileExtension = ".txt")
    public String attachDataTXT() {
        return "This is the file content.";
    }

    @Attachment(value = "screenshot", type = "image/png", fileExtension = ".png")
    public byte[] attachScreenshotPNG() throws IOException {
        return Files.readAllBytes(Paths.get("/path/to/image.png"));
    }
}

Java
import io.qameta.allure.Allure;
import org.testng.annotations.Test;

import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Paths;

public class TestMyWebsite {

    @Test
    public void testAuthentication() throws IOException {
        // ...
        Allure.attachment("data.txt", "This is the file content.");
        try (InputStream is = Files.newInputStream(Paths.get("/path/img.png"))) {
            Allure.attachment("image.png", is);
        }
    }
}

Additionally, Allure provides a way to spawn threads that will add attachments to the test results asynchronously without blocking the test runner. While not recommended for the majority of cases, this approach can improve tests that otherwise have to wait for processing very large files, such as videos larger than 1 GB. Please refer to the source code for Allure.addByteAttachmentAsync() and Allure.addStreamAttachmentAsync() to learn more.

Results history

Flaky

• @Flaky

Indicate that the test is known to be unstable and can may not succeed every time.