Getting started with Allure Cucumber-JVM

Generate beautiful HTML reports using Allure Report and your Cucumber-JVM tests.

INFO

Check out the example projects at github.com/allure-examples to see Allure Cucumber-JVM in action.

Setting up

INFO

This guide is tailored exclusively for Cucumber-JVM 7. While Allure maintains compatibility with older versions of Cucumber-JVM (version 4 onwards), configurations for these versions are not included to maintain simplicity in this documentation. The general configuration steps remain similar for older versions, but you will need to adjust the dependency and plugin names accordingly. For example, when working with Cucumber-JVM 6, the dependency would be allure-cucumber6-jvm and the plugin to use would be io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm.

To integrate Allure into an existing Cucumber-JVM project, you need to:

1. Add Allure dependencies to your project.
2. Activate the Allure Cucumber-JVM plugin.
3. Set up AspectJ for @Step and @Attachment annotations support.
4. Designate a location for Allure results storage.

The specific implementation of these steps varies based on how your Cucumber-JVM is set up. Cucumber-JVM can be used with three different types of runners:

• JUnit Platform (recommended)
• TestNG
• JUnit 4 (deprecated)

Using JUnit Platform

JUnit Platform is the recommended runner for Cucumber-JVM tests. Full configuration demonstrated in this example.

Add Allure dependencies

Maven

<!-- Define the version of Allure you want to use via the allure.version property -->
<properties>
    <allure.version>2.24.0</allure.version>
</properties>

<!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-bom</artifactId>
            <version>${allure.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<!-- Add necessary Allure dependencies to dependencies section -->
<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-cucumber7-jvm</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-junit-platform</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Gradle (Kotlin)

// Define the version of Allure you want to use via the allureVersion property
val allureVersion = "2.24.0"
// ...
dependencies {
    // Import allure-bom to ensure correct versions of all the dependencies are used
    testImplementation(platform("io.qameta.allure:allure-bom:$allureVersion"))
    // Add necessary Allure dependencies to dependencies section
    testImplementation("io.qameta.allure:allure-cucumber7-jvm")
    testImplementation("io.qameta.allure:allure-junit-platform")
}

Gradle (Groovy)

// Define the version of Allure you want to use via the allureVersion property
def allureVersion = "2.24.0"

dependencies {
    // Import allure-bom to ensure correct versions of all the dependencies are used
    testImplementation platform("io.qameta.allure:allure-bom:$allureVersion")
    // Add necessary Allure dependencies to dependencies section
    testImplementation "io.qameta.allure:allure-cucumber7-jvm"
    testImplementation "io.qameta.allure:allure-junit-platform"
}

Enable Allure Cucumber-JVM plugin

Using junit-platform.properties file

# Create a `src/test/resources/junit-platform.properties` file and add property `cucumber.plugin` with value `io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm`:

#junit-platform.properties
cucumber.plugin=io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm

Using @ConfigurationParameter

// Add `org.junit.platform.suite.api.ConfigurationParameter` annotation and specify property `cucumber.plugin` with value `io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm`:

import org.junit.platform.suite.api.ConfigurationParameter;

import static io.cucumber.core.options.Constants.PLUGIN_PROPERTY_NAME;

@Suite
@IncludeEngines("cucumber")
@ConfigurationParameter(
        key = PLUGIN_PROPERTY_NAME,
        value = "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"
)
class CucumberTest {
}

Using TestNG

INFO

To ensure compatibility with Cucumber-JVM 7, you must use TestNG 7.8 or higher, which in turn requires Java 11 or newer. However, if you are also using Allure TestNG, which is compatible with TestNG 6, you need to be careful with the order of dependencies in your project. Declare the io.qameta.allure:allure-testng dependency after the io.cucumber:cucumber-testng dependency, or any explicitly stated TestNG dependency, to avoid conflicts. This sequence is important to ensure that the correct version of TestNG is used.

Add Allure dependencies

Maven

<!-- Define the version of Allure you want to use via the allure.version property -->
<properties>
    <allure.version>2.24.0</allure.version>
</properties>

<!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-bom</artifactId>
            <version>${allure.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<!-- Add necessary Allure dependencies to dependencies section -->
<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-cucumber7-jvm</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-testng</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Gradle (Kotlin)

// Define the version of Allure you want to use via the allureVersion property
val allureVersion = "2.24.0"
// ...
dependencies {
    // Import allure-bom to ensure correct versions of all the dependencies are used
    testImplementation(platform("io.qameta.allure:allure-bom:$allureVersion"))
    // Add necessary Allure dependencies to dependencies section
    testImplementation("io.qameta.allure:allure-cucumber7-jvm")
    testImplementation("io.qameta.allure:allure-testng")
}

Gradle (Groovy)

// Define the version of Allure you want to use via the allureVersion property
def allureVersion = "2.24.0"

dependencies {
    // Import allure-bom to ensure correct versions of all the dependencies are used
    testImplementation platform("io.qameta.allure:allure-bom:$allureVersion")
    // Add necessary Allure dependencies to dependencies section
    testImplementation "io.qameta.allure:allure-cucumber7-jvm"
    testImplementation "io.qameta.allure:allure-testng"
}

Enable Allure Cucumber-JVM plugin

Add io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm plugin to your Cucumber Options:

@Test
@CucumberOptions(
        plugin = {
                "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"
        }
)
public class CucumberTest extends AbstractTestNGCucumberTests {
}

Using JUnit 4

Add Allure dependencies

Maven

<!-- Define the version of Allure you want to use via the allure.version property -->
<properties>
    <allure.version>2.24.0</allure.version>
</properties>

<!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-bom</artifactId>
            <version>${allure.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<!-- Add necessary Allure dependencies to dependencies section -->
<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-cucumber7-jvm</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-junit4</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Gradle (Kotlin)

// Define the version of Allure you want to use via the allureVersion property
val allureVersion = "2.24.0"
// ...
dependencies {
    // Import allure-bom to ensure correct versions of all the dependencies are used
    testImplementation(platform("io.qameta.allure:allure-bom:$allureVersion"))
    // Add necessary Allure dependencies to dependencies section
    testImplementation("io.qameta.allure:allure-cucumber7-jvm")
    testImplementation("io.qameta.allure:allure-junit4")
    testImplementation("io.qameta.allure:allure-junit4-aspect")
}

Gradle (Groovy)

// Define the version of Allure you want to use via the allureVersion property
def allureVersion = "2.24.0"

dependencies {
    // Import allure-bom to ensure correct versions of all the dependencies are used
    testImplementation platform("io.qameta.allure:allure-bom:$allureVersion")
    // Add necessary Allure dependencies to dependencies section
    testImplementation "io.qameta.allure:allure-cucumber7-jvm"
    testImplementation "io.qameta.allure:allure-junit4"
    testImplementation "io.qameta.allure:allure-junit4-aspect"
}

Enable Allure Cucumber-JVM plugin

Add io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm plugin to your Cucumber Options:

@RunWith(Cucumber.class)
@CucumberOptions(
        plugin = {
                "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"
        }
)
public class CucumberTest {
}

Configure AspectJ

Allure leverages AspectJ for the functionality of @Step and @Attachment annotations. Additionally, certain frameworks, such as JUnit 4 or AssertJ, rely on AspectJ integration to function correctly. These frameworks will require appropriate configuration to ensure they work as intended.

Maven

<!-- Define the version of AspectJ -->
<properties>
    <aspectj.version>1.9.20.1</aspectj.version>
</properties>

<!-- Add the following options to your maven-surefire-plugin -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.1.2</version>
    <configuration>
        <argLine>
            -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
        </argLine>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>${aspectj.version}</version>
        </dependency>
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-junit4-aspect</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
</plugin>

Gradle (Kotlin)

// Define the version of AspectJ
val aspectJVersion = "1.9.20.1"

// Define configuration for AspectJ agent
val agent: Configuration by configurations.creating {
    isCanBeConsumed = true
    isCanBeResolved = true
}

dependencies {
    // Add aspectjweaver dependency
    agent("org.aspectj:aspectjweaver:${aspectJVersion}")

    // Add AspectJ integration for Allure
    testImplementation("io.qameta.allure:allure-junit4-aspect")
}

// Configure javaagent for test execution
tasks.test {
    jvmArgs = listOf(
       "-javaagent:${agent.singleFile}"
    )
}

Gradle (Groovy)

// Define the version of AspectJ
def aspectJVersion = "1.9.20.1"

// Define configuration for AspectJ agent
configurations {
    agent {
        canBeResolved = true
        canBeConsumed = true
    }
}

dependencies {
    // Add aspectjweaver dependency
    agent "org.aspectj:aspectjweaver:$aspectJVersion"

    // Add AspectJ integration for Allure
    testImplementation "io.qameta.allure:allure-junit4-aspect"
}

// Configure javaagent for test execution
test {
    jvmArgs = [ "-javaagent:${configurations.agent.singleFile}" ]
}

Specifying Allure Results location

Allure, by default, saves test results in the project's root directory. However, it is recommended to store your test results in the build output directory.

To configure this, create an allure.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:

Maven

#allure.properties
allure.results.directory=target/allure-results

Gradle (Kotlin)

#allure.properties
allure.results.directory=build/allure-results

Gradle (Groovy)

#allure.properties
allure.results.directory=build/allure-results

Running tests

Execute your Cucumber-JVM tests as you normally would. Below are the commands for Gradle and Maven users:

For Gradle:

MacOS/Linux

./gradlew test

Windows

gradlew test

For Maven:

MacOS/Linux

./mvnw verify

Windows

mvnw verify

After running the tests, Allure will gather the test execution data and store it in the allure-results directory. You can then generate an HTML report from these results using Allure's reporting tools.

Writing tests

The Allure Cucumber-JVM adapter extends the standard reporting features of Cucumber-JVM by providing additional capabilities for crafting more informative and structured tests. This section highlights key enhancements that can be utilized:

• Metadata Annotation: Enhance test reports with descriptions, links, and other metadata.
• Test Organization: Structure your tests into clear hierarchies for better readability and organization.
• Step Division: Break down tests into smaller test steps for easier understanding and maintenance.
• Parametrized Tests: Clearly describe the parameters for parametrized tests to specify different scenarios.
• Attachments: Automatically capture screenshots and other files during test execution.
• Test Selection: Use a test plan file to select which tests to run, allowing for flexible test execution.
• Environment Details: Include comprehensive environment information to accompany the test report.

Adding Metadata

Allure allows you to enrich your reports with a variety of metadata. This additional information provides context and details for each test, enhancing the report's usefulness. Refer to the metadata reference section for an exhaustive list of what can be added.

You can assign metadata to a Cucumber-JVM features in two ways:

1. Using Gherkin tags: Use a naming convention to add metadata to test scenarios.
2. Runtime API: Use a method call within the body of a test step method to dynamically assign metadata during test runtime. This approach allows for the construction of strings and values on-the-fly.

When assigning metadata to your tests using Allure's Runtime API, it's crucial to consider the balance between control and potential risks. While the Runtime API offers more flexibility in handling metadata dynamically, it also comes with certain drawbacks. For instance, if a test fails before all the dynamic metadata is applied, the resulting report may lack important information, leading to reduced clarity and completeness. Therefore, it's essential to ensure that metadata is set early in the test to mitigate this risk.

Gherkin Tags

@allure.label.layer:web
@allure.label.owner:eroshenkoam
@allure.label.page:/{org}/{repo}/labels
Feature: Labels

  @critical
  @allure.label.jira:AE-2
  Scenario: Create new label for authorized user
    When I open labels page
    And I create label with title "hello"
    Then I should see label with title "hello"

Runtime API

import io.cucumber.java.en.Given;
import io.qameta.allure.Allure;

public class CucumberSteps {

    @When("^I open labels page$")
    public void openLabelsPage() {
        Allure.label("layer", "web");
        Allure.label("owner", "eroshenkoam");
        Allure.label("page", "/{org}/{repo}/labels");
        Allure.label("severity", "critical");
        Allure.label("jira", "AE-2");
        // ...
    }
}

WARNING

To use Gherkin tags for assigning data, be sure to upgrade to Allure Report 2.27 or higher or consider uploading your test results to Allure TestOps.

Organize tests

Allure facilitates enhanced navigation in test reports by allowing tests to be organized into hierarchical structures. As explained in the section on Improving navigation in your test report, Allure Cucumber-JVM supports this feature.

To define a test's position within the behavior-based hierarchy:

Gherkin Tags

@allure.label.epic:Web
Feature: Labels

    Scenario: Create new label for authorized user
        When I open labels page
        And I create label with title "hello"
        Then I should see label with title "hello"

Runtime API

import io.cucumber.java.en.Given;
import io.qameta.allure.Allure;

public class CucumberSteps {

    @When("^I open labels page$")
    public void openLabelsPage() {
        Allure.epic("Web");
        // ...
    }
}

To specify a test's location in the suite-based hierarchy:

Gherkin Tags

@allure.label.parentSuite:Cucumber
@allure.label.suite:Labels
@allure.label.subSuite:Authorized
Feature: Labels

  Scenario: Create new label for authorized user
    When I open labels page
    And I create label with title "hello"
    Then I should see label with title "hello"

Runtime API

import io.cucumber.java.en.Given;
import io.qameta.allure.Allure;

public class CucumberSteps {

    @When("^I open labels page$")
    public void openLabelsPage() {
        Allure.label("parentSuite", "Cucumber");
        Allure.label("suite", "Labels");
        Allure.label("subSuite", "Authorized");
        // ...
    }
}

A test's placement within the package-based hierarchy is determined by the location of the feature files, relative to test resources directory.

Creating Sub-Steps

Allure Cucumber-JVM enhances test reports by allowing the creation of sub-steps within a single Gherkin test step, which can be particularly useful in cases where you have a set of actions that are:

• Sufficiently cohesive to be described as one step in a Gherkin scenario.
• Intricate enough that they involve several potential failure points.

Allure provides three methods for defining sub-steps:

1. Annotated Sub-Steps: Defined using @Step annotation.
2. Lambda Sub-Steps: Utilizing lambda expressions for more concise and inline step definitions.
3. No-Op Sub-Steps: Employed as placeholders or for structuring steps without additional functionality.

For detailed guidance on implementing each type of sub-step, consult the test steps reference.

Annotated sub-steps

import io.cucumber.java.en.Then;
import io.qameta.allure.Step;

public class CucumberSteps {

    @When("^I open labels page$")
    public void openLabelsPage() {
        subStep1();
        subStep2();
    }

    @Step("Sub-step 1")
    public void subStep1() {
        subSubStep1();
        subSubStep2();
    }

    @Step("Sub-sub-step 1")
    public void subSubStep1() {
        // ...
    }

    @Step("Sub-sub-step 2")
    public void subSubStep2() {
        // ...
    }

    @Step("Sub-step 2")
    public void subStep2() {
        // ...
    }
}

Lambda steps and no-op steps

import io.cucumber.java.en.Then;
import io.qameta.allure.Allure;

public class CucumberSteps {

    @When("^I open labels page$")
    public void openLabelsPage() {

        Allure.step("Sub-step 1", (step) -> {
            // ...
            Allure.step("Sub-sub-step 1");
            // ...
            Allure.step("Sub-sub-step 2");
        });

        Allure.step("Sub-step 2", (step) -> {
            // ...
        });
    }
}

Support for Scenario Outlines

Allure Cucumber-JVM provides complete support for Scenario Outlines, a feature of Cucumber-JVM that allows for parametrized tests. No special setup is needed to take advantage of this capability within Allure.

Feature: Parameters

  Scenario Outline: test authorization as "<login>"
    Then authorize as "<login>"

    Examples:
      | login               |
      | johndoe             |
      | [email protected] |

import io.cucumber.java.en.Then;
import io.qameta.allure.Allure;

public class CucumberSteps {

    @Then("authorize as {string}")
    public void testAuthentication(String login) {
        // ...
    }
}

Furthermore, Allure's Runtime API can be utilized to include extra parameters in the test report, offering enhanced documentation capabilities and greater detail for each test scenario:

import io.cucumber.java.en.Then;
import io.qameta.allure.Allure;

public class CucumberSteps {

    @Then("authorize as {string}")
    public void testAuthentication(String login) {
        Allure.parameter("Login", login);
        // ...
    }
}

Attaching screenshots and other files

In Allure reports, you have the ability to attach various types of files, which can greatly enhance the comprehensibility of the report. A common practice is to attach screenshots that capture the state of the user interface at specific moments during test execution.

Allure Cucumber-JVM offers multiple methods for creating attachments, whether from pre-existing files or from content generated on-the-fly. For detailed instructions on how to implement attachments, refer to the attachments section in the Allure Cucumber-JVM reference.

import io.cucumber.java.en.Then;
import io.qameta.allure.Allure;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class CucumberSteps {

    @When("^I open labels page$")
    public void openLabelsPage() throws IOException {
        Allure.attachment("data.txt", "This is the file content");
        try (InputStream is = Files.newInputStream(Paths.get("/path/img.png"))) {
            Allure.attachment("image.png", is);
        }
        // ...
    }
}

Selective test run (Allure TestOps)

WARNING

For this feature to work, the AspectJ integration for Allure Report must be added to the project.

In Allure TestOps, you have the option to rerun only a subset of tests by utilizing the same Continuous Integration (CI) job that was used for the full test run. The capability to perform selective test runs varies depending on the test runner you are using:

• JUnit Platform Runner: This runner supports selective test runs natively, allowing you to easily specify which tests to include in the rerun without any additional setup.
• JUnit 4 Runner: For Gradle users, JUnit 4 also supports selective test runs right out of the box. However, Maven users might encounter some issues with this functionality due to limitations within the Maven Surefire plugin.
• TestNG Runner: To use selective test runs with TestNG, you must be using TestNG version 7.8 or later. Additionally, some extra configurations might be needed to enable this feature.

TestNG

For integrating selective test runs in a TestNG project with Allure TestOps, follow these steps:

1. Create CucumberTestNgFilter: Implement a filter class named CucumberTestNgFilter. This class will determine which tests to run based on your criteria. Place this class within your project's source directory, ensuring it's correctly packaged.
2. Register CucumberTestNgFilter: In your project, create a file named META-INF/services/org.testng.ITestNGListener in your test resources. Inside this file, write the fully qualified name of your CucumberTestNgFilter class. This step is crucial as it registers your filter as a TestNG listener, which TestNG will invoke during the test run.

// Step 1: Create CucumberTestNgFilter.java
package com.yourproject.filters;

import io.cucumber.testng.FeatureWrapper;
import io.cucumber.testng.Pickle;
import io.cucumber.testng.PickleWrapper;
import org.testng.IDataProviderInterceptor;
import org.testng.IDataProviderMethod;
import org.testng.ITestContext;
import org.testng.ITestNGMethod;

import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;

public class CucumberTestNgFilter implements IDataProviderInterceptor {

    private static final String CUCUMBER_WORKING_DIR
            = Paths.get("").toUri().toString();
    private static final String CLASSPATH_PREFIX = "classpath:";

    private static final String CUCUMBER_RUN_METHOD
            = "io.cucumber.testng.AbstractTestNGCucumberTests.runScenario";

    @Override
    public Iterator<Object[]> intercept(
      final Iterator<Object[]> original,
      final IDataProviderMethod dataProviderMethod,
      final ITestNGMethod method,
      final ITestContext iTestContext
    ) {
        if (!CUCUMBER_RUN_METHOD.equals(method.getQualifiedName())) {
            return original;
        }

        final List<Object[]> filtered = new ArrayList<>();
        original.forEachRemaining(objects -> {
            if (objects.length != 2) {
                filtered.add(objects);
            }
            final PickleWrapper first = (PickleWrapper) objects[0];
            final FeatureWrapper second = (FeatureWrapper) objects[1];

            final Pickle pickle = first.getPickle();
            final String fullName = String.format("%s:%d",
                    getUri(pickle),
                    pickle.getLine()
            );

            System.out.println(fullName);

            filtered.add(new Object[]{first, second});
        });

        return filtered.iterator();
    }

    private String getUri(final Pickle pickle) {
        final String testCaseUri = pickle.getUri().toString();
        if (testCaseUri.startsWith(CUCUMBER_WORKING_DIR)) {
            return testCaseUri.substring(CUCUMBER_WORKING_DIR.length());
        }
        if (testCaseUri.startsWith(CLASSPATH_PREFIX)) {
            return testCaseUri.substring(CLASSPATH_PREFIX.length());
        }
        return testCaseUri;
    }
}

For Step 2, the structure of the project should include:

src/
└── test/
    └── resources/
        └── META-INF/
            └── services/
                └── org.testng.ITestNGListener

Environment information

For the main page of the report, you can collect various information about the environment in which the tests were executed.

For example, it is a good idea to use this to remember the OS and Java versions. This may help the future reader investigate bugs that are reproducible only in some environments.

To provide environment information, put a file named environment.properties into the allure-results directory after running the tests. See the example in Environment file.

Note that this feature should be used for properties that do not change for all tests in the report. If you have properties that can be different for different tests, consider using Parametrized tests.