Allure Cucumber-JVM

Allure Cucumber-JVM latest version

Generate beautiful HTML reports using Allure Report and your Cucumber-JVM tests.

Allure Report Cucumber-JVM Example

How to start

This guide is tailored exclusively for Cucumber-JVM 7. While Allure maintains compatibility with older versions of Cucumber-JVM (version 4 onwards), configurations for these versions are not included to maintain simplicity in this documentation. The general configuration steps remain similar for older versions, but you will need to adjust the dependency and plugin names accordingly. For example, when working with Cucumber-JVM 6, the dependency would be allure-cucumber6-jvm and the plugin to use would be io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm.

To integrate Allure into an existing Cucumber-JVM project, you need to:

  1. Add Allure dependencies to your project.
  2. Activate the Allure Cucumber-JVM plugin.
  3. Set up AspectJ for @Step and @Attachment annotations support.
  4. Designate a location for Allure results storage.

The specific implementation of these steps varies based on how your Cucumber-JVM is set up. Cucumber-JVM can be used with three different types of runners:

  • JUnit Platform (recommended)
  • TestNG
  • JUnit 4 (deprecated)

Using JUnit Platform

JUnit Platform is the recommended runner for Cucumber-JVM tests. Full configuration demonstrated in this example.

Add Allure dependencies

XML
<!-- Define the version of Allure you want to use via the allure.version property --> <properties> <allure.version>2.24.0</allure.version> </properties> <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used --> <dependencyManagement> <dependencies> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-bom</artifactId> <version>${allure.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <!-- Add necessary Allure dependencies to dependencies section --> <dependencies> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-cucumber7-jvm</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-junit-platform</artifactId> <scope>test</scope> </dependency> </dependencies>
kts
// Define the version of Allure you want to use via the allureVersion property val allureVersion = "2.24.0" // ... dependencies { // Import allure-bom to ensure correct versions of all the dependencies are used testImplementation(platform("io.qameta.allure:allure-bom:$allureVersion")) // Add necessary Allure dependencies to dependencies section testImplementation("io.qameta.allure:allure-cucumber7-jvm") testImplementation("io.qameta.allure:allure-junit-platform") }
Groovy
// Define the version of Allure you want to use via the allureVersion property def allureVersion = "2.24.0" dependencies { // Import allure-bom to ensure correct versions of all the dependencies are used testImplementation platform("io.qameta.allure:allure-bom:$allureVersion") // Add necessary Allure dependencies to dependencies section testImplementation "io.qameta.allure:allure-cucumber7-jvm" testImplementation "io.qameta.allure:allure-junit-platform" }

Enable Allure Cucumber-JVM plugin

Create a src/test/resources/junit-platform.properties file and add property cucumber.plugin with value io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm:

INI
#junit-platform.properties cucumber.plugin=io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm

Add org.junit.platform.suite.api.ConfigurationParameter annotation and specify property cucumber.plugin with value io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm:

Java
import org.junit.platform.suite.api.ConfigurationParameter; import static io.cucumber.core.options.Constants.PLUGIN_PROPERTY_NAME; @Suite @IncludeEngines("cucumber") @ConfigurationParameter( key = PLUGIN_PROPERTY_NAME, value = "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm" ) class CucumberTest { }

Using TestNG

To ensure compatibility with Cucumber-JVM 7, you must use TestNG 7.8 or higher, which in turn requires Java 11 or newer. However, if you are also using Allure TestNG, which is compatible with TestNG 6, you need to be careful with the order of dependencies in your project. Declare the io.qameta.allure:allure-testng dependency after the io.cucumber:cucumber-testng dependency, or any explicitly stated TestNG dependency, to avoid conflicts. This sequence is important to ensure that the correct version of TestNG is used.

Add Allure dependencies

XML
<!-- Define the version of Allure you want to use via the allure.version property --> <properties> <allure.version>2.24.0</allure.version> </properties> <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used --> <dependencyManagement> <dependencies> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-bom</artifactId> <version>${allure.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <!-- Add necessary Allure dependencies to dependencies section --> <dependencies> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-cucumber7-jvm</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-testng</artifactId> <scope>test</scope> </dependency> </dependencies>
kts
// Define the version of Allure you want to use via the allureVersion property val allureVersion = "2.24.0" // ... dependencies { // Import allure-bom to ensure correct versions of all the dependencies are used testImplementation(platform("io.qameta.allure:allure-bom:$allureVersion")) // Add necessary Allure dependencies to dependencies section testImplementation("io.qameta.allure:allure-cucumber7-jvm") testImplementation("io.qameta.allure:allure-testng") }
Groovy
// Define the version of Allure you want to use via the allureVersion property def allureVersion = "2.24.0" dependencies { // Import allure-bom to ensure correct versions of all the dependencies are used testImplementation platform("io.qameta.allure:allure-bom:$allureVersion") // Add necessary Allure dependencies to dependencies section testImplementation "io.qameta.allure:allure-cucumber7-jvm" testImplementation "io.qameta.allure:allure-testng" }

Enable Allure Cucumber-JVM plugin

Add io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm plugin to your Cucumber Options:

Java
@Test @CucumberOptions( plugin = { "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm" } ) public class CucumberTest extends AbstractTestNGCucumberTests { }

Using JUnit 4

Add Allure dependencies

XML
<!-- Define the version of Allure you want to use via the allure.version property --> <properties> <allure.version>2.24.0</allure.version> </properties> <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used --> <dependencyManagement> <dependencies> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-bom</artifactId> <version>${allure.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <!-- Add necessary Allure dependencies to dependencies section --> <dependencies> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-cucumber7-jvm</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-junit4</artifactId> <scope>test</scope> </dependency> </dependencies>
kts
// Define the version of Allure you want to use via the allureVersion property val allureVersion = "2.24.0" // ... dependencies { // Import allure-bom to ensure correct versions of all the dependencies are used testImplementation(platform("io.qameta.allure:allure-bom:$allureVersion")) // Add necessary Allure dependencies to dependencies section testImplementation("io.qameta.allure:allure-cucumber7-jvm") testImplementation("io.qameta.allure:allure-junit4") testImplementation("io.qameta.allure:allure-junit4-aspect") }
Groovy
// Define the version of Allure you want to use via the allureVersion property def allureVersion = "2.24.0" dependencies { // Import allure-bom to ensure correct versions of all the dependencies are used testImplementation platform("io.qameta.allure:allure-bom:$allureVersion") // Add necessary Allure dependencies to dependencies section testImplementation "io.qameta.allure:allure-cucumber7-jvm" testImplementation "io.qameta.allure:allure-junit4" testImplementation "io.qameta.allure:allure-junit4-aspect" }

Enable Allure Cucumber-JVM plugin

Add io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm plugin to your Cucumber Options:

Java
@RunWith(Cucumber.class) @CucumberOptions( plugin = { "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm" } ) public class CucumberTest { }

Configure AspectJ

Allure leverages AspectJ for the functionality of @Step and @Attachment annotations. Additionally, certain frameworks, such as JUnit 4 or AssertJ, rely on AspectJ integration to function correctly. These frameworks will require appropriate configuration to ensure they work as intended.

XML
<!-- Define the version of AspectJ --> <properties> <aspectj.version>1.9.20.1</aspectj.version> </properties> <!-- Add the following options to your maven-surefire-plugin --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.1.2</version> <configuration> <argLine> -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" </argLine> </configuration> <dependencies> <dependency> <groupId>org.aspectj</groupId> <artifactId>aspectjweaver</artifactId> <version>${aspectj.version}</version> </dependency> </dependencies> </plugin>
kts
// Define the version of AspectJ val aspectJVersion = "1.9.20.1" // Define configuration for AspectJ agent val agent: Configuration by configurations.creating { isCanBeConsumed = true isCanBeResolved = true } dependencies { // Add aspectjweaver dependency agent("org.aspectj:aspectjweaver:${aspectJVersion}") } // Configure javaagent for test execution tasks.test { jvmArgs = listOf( "-javaagent:${agent.singleFile}" ) }
Groovy
// Define the version of AspectJ def aspectJVersion = "1.9.20.1" // Define configuration for AspectJ agent configurations { agent { canBeResolved = true canBeConsumed = true } } dependencies { // Add aspectjweaver dependency agent "org.aspectj:aspectjweaver:$aspectJVersion" } // Configure javaagent for test execution test { jvmArgs = [ "-javaagent:${configurations.agent.singleFile}" ] }

Specifying Allure Results location

Allure, by default, saves test results in the project's root directory. However, it is recommended to store your test results in the build output directory.

To configure this, create an allure.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:

INI
#allure.properties allure.results.directory=target/allure-results
INI
#allure.properties allure.results.directory=build/allure-results
INI
#allure.properties allure.results.directory=build/allure-results

Running tests

Execute your Cucumber-JVM tests as you normally would. Below are the commands for Gradle and Maven users:

For Gradle:

Bash
./gradlew test
Bash
gradlew test

For Maven:

Bash
./mvnw verify
Bash
mvnw verify

After running the tests, Allure will gather the test execution data and store it in the allure-results directory. You can then generate an HTML report from these results using Allure's reporting tools.

Writing tests

The Allure Cucumber-JVM adapter extends the standard reporting features of Cucumber-JVM by providing additional capabilities for crafting more informative and structured tests. This section highlights key enhancements that can be utilized:

Adding Metadata

Allure allows you to enrich your reports with a variety of metadata. This additional information provides context and details for each test, enhancing the report's usefulness. Refer to the metadata reference section for an exhaustive list of what can be added.

You can assign metadata to a Cucumber-JVM features in two ways:

  1. Using Gherkin tags (Allure TestOps): Use a naming convention to add metadata to test scenarios.
  2. Runtime API: Use a method call within the body of a test step method to dynamically assign metadata during test runtime. This approach allows for the construction of strings and values on-the-fly.

When assigning metadata to your tests using Allure's Runtime API, it's crucial to consider the balance between control and potential risks. While the Runtime API offers more flexibility in handling metadata dynamically, it also comes with certain drawbacks. For instance, if a test fails before all the dynamic metadata is applied, the resulting report may lack important information, leading to reduced clarity and completeness. Therefore, it's essential to ensure that metadata is set early in the test to mitigate this risk.

Gherkin
@allure.label.layer:web @allure.label.owner:eroshenkoam @allure.label.page:/{org}/{repo}/labels Feature: Labels @critical @allure.label.jira:AE-2 Scenario: Create new label for authorized user When I open labels page And I create label with title "hello" Then I should see label with title "hello"
Java
import io.cucumber.java.en.Given; import io.qameta.allure.Allure; public class CucumberSteps { @When("^I open labels page$") public void openLabelsPage() { Allure.label("layer", "web"); Allure.label("owner", "eroshenkoam"); Allure.label("page", "/{org}/{repo}/labels"); Allure.label("severity", "critical"); Allure.label("jira", "AE-2"); // ... } }

Organize tests

Allure facilitates enhanced navigation in test reports by allowing tests to be organized into hierarchical structures. As explained in the section on Improving navigation in your test report, Allure Cucumber-JVM supports this feature.

To define a test's position within the behavior-based hierarchy:

Gherkin
@allure.label.epic:Web Feature: Labels Scenario: Create new label for authorized user When I open labels page And I create label with title "hello" Then I should see label with title "hello"
Java
import io.cucumber.java.en.Given; import io.qameta.allure.Allure; public class CucumberSteps { @When("^I open labels page$") public void openLabelsPage() { Allure.epic("Web"); // ... } }

To specify a test's location in the suite-based hierarchy:

Gherkin
@allure.label.parentSuite:Cucumber @allure.label.suite:Labels @allure.label.subSuite:Authorized Feature: Labels Scenario: Create new label for authorized user When I open labels page And I create label with title "hello" Then I should see label with title "hello"
Java
import io.cucumber.java.en.Given; import io.qameta.allure.Allure; public class CucumberSteps { @When("^I open labels page$") public void openLabelsPage() { Allure.label("parentSuite", "Cucumber"); Allure.label("suite", "Labels"); Allure.label("subSuite", "Authorized"); // ... } }

A test's placement within the package-based hierarchy is determined by the location of the feature files, relative to test resources directory.

Creating Sub-Steps

Allure Cucumber-JVM enhances test reports by allowing the creation of sub-steps within a single Gherkin test step, which can be particularly useful in cases where you have a set of actions that are:

  • Sufficiently cohesive to be described as one step in a Gherkin scenario.
  • Intricate enough that they involve several potential failure points.

Allure provides three methods for defining sub-steps:

  1. Annotated Sub-Steps: Defined using @Step annotation.
  2. Lambda Sub-Steps: Utilizing lambda expressions for more concise and inline step definitions.
  3. No-Op Sub-Steps: Employed as placeholders or for structuring steps without additional functionality.

For detailed guidance on implementing each type of sub-step, consult the test steps reference.

Java
import io.cucumber.java.en.Then; import io.qameta.allure.Step; public class CucumberSteps { @When("^I open labels page$") public void openLabelsPage() { subStep1(); subStep2(); } @Step("Sub-step 1") public void subStep1() { subSubStep1(); subSubStep2(); } @Step("Sub-sub-step 1") public void subSubStep1() { // ... } @Step("Sub-sub-step 2") public void subSubStep2() { // ... } @Step("Sub-step 2") public void subStep2() { // ... } }
Java
import io.cucumber.java.en.Then; import io.qameta.allure.Allure; public class CucumberSteps { @When("^I open labels page$") public void openLabelsPage() { Allure.step("Sub-step 1", (step) -> { // ... Allure.step("Sub-sub-step 1"); // ... Allure.step("Sub-sub-step 2"); }); Allure.step("Sub-step 2", (step) -> { // ... }); } }

Support for Scenario Outlines

Allure Cucumber-JVM provides complete support for Scenario Outlines, a feature of Cucumber-JVM that allows for parametrized tests. No special setup is needed to take advantage of this capability within Allure.

Gherkin
Feature: Parameters Scenario Outline: test authorization as "<login>" Then authorize as "<login>" Examples: | login | | johndoe | | [email protected] |
Java
import io.cucumber.java.en.Then; import io.qameta.allure.Allure; public class CucumberSteps { @Then("authorize as {string}") public void testAuthentication(String login) { // ... } }

Furthermore, Allure's Runtime API can be utilized to include extra parameters in the test report, offering enhanced documentation capabilities and greater detail for each test scenario:

Java
import io.cucumber.java.en.Then; import io.qameta.allure.Allure; public class CucumberSteps { @Then("authorize as {string}") public void testAuthentication(String login) { Allure.parameter("Login", login); // ... } }

Attaching screenshots and other files

In Allure reports, you have the ability to attach various types of files, which can greatly enhance the comprehensibility of the report. A common practice is to attach screenshots that capture the state of the user interface at specific moments during test execution.

Allure Cucumber-JVM offers multiple methods for creating attachments, whether from pre-existing files or from content generated on-the-fly. For detailed instructions on how to implement attachments, refer to the attachments section in the Allure Cucumber-JVM reference.

Java
import io.cucumber.java.en.Then; import io.qameta.allure.Allure; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; public class CucumberSteps { @When("^I open labels page$") public void openLabelsPage() throws IOException { Allure.attachment("data.txt", "This is the file content"); try (InputStream is = Files.newInputStream(Paths.get("/path/img.png"))) { Allure.attachment("image.png", is); } // ... } }

Selective test run (Allure TestOps)

In Allure TestOps, you have the option to rerun only a subset of tests by utilizing the same Continuous Integration (CI) job that was used for the full test run. The capability to perform selective test runs varies depending on the test runner you are using:

  • JUnit Platform Runner: This runner supports selective test runs natively, allowing you to easily specify which tests to include in the rerun without any additional setup.
  • JUnit 4 Runner: For Gradle users, JUnit 4 also supports selective test runs right out of the box. However, Maven users might encounter some issues with this functionality due to limitations within the Maven Surefire plugin.
  • TestNG Runner: To use selective test runs with TestNG, you must be using TestNG version 7.8 or later. Additionally, some extra configurations might be needed to enable this feature.

TestNG

For integrating selective test runs in a TestNG project with Allure TestOps, follow these steps:

  1. Create CucumberTestNgFilter: Implement a filter class named CucumberTestNgFilter. This class will determine which tests to run based on your criteria. Place this class within your project's source directory, ensuring it's correctly packaged.
  2. Register CucumberTestNgFilter: In your project, create a file named META-INF/services/org.testng.ITestNGListener in your test resources. Inside this file, write the fully qualified name of your CucumberTestNgFilter class. This step is crucial as it registers your filter as a TestNG listener, which TestNG will invoke during the test run.
Java
// Step 1: Create CucumberTestNgFilter.java package com.yourproject.filters; import io.cucumber.testng.FeatureWrapper; import io.cucumber.testng.Pickle; import io.cucumber.testng.PickleWrapper; import org.testng.IDataProviderInterceptor; import org.testng.IDataProviderMethod; import org.testng.ITestContext; import org.testng.ITestNGMethod; import java.nio.file.Paths; import java.util.ArrayList; import java.util.Iterator; import java.util.List; public class CucumberTestNgFilter implements IDataProviderInterceptor { private static final String CUCUMBER_WORKING_DIR = Paths.get("").toUri().toString(); private static final String CLASSPATH_PREFIX = "classpath:"; private static final String CUCUMBER_RUN_METHOD = "io.cucumber.testng.AbstractTestNGCucumberTests.runScenario"; @Override public Iterator<Object[]> intercept( final Iterator<Object[]> original, final IDataProviderMethod dataProviderMethod, final ITestNGMethod method, final ITestContext iTestContext ) { if (!CUCUMBER_RUN_METHOD.equals(method.getQualifiedName())) { return original; } final List<Object[]> filtered = new ArrayList<>(); original.forEachRemaining(objects -> { if (objects.length != 2) { filtered.add(objects); } final PickleWrapper first = (PickleWrapper) objects[0]; final FeatureWrapper second = (FeatureWrapper) objects[1]; final Pickle pickle = first.getPickle(); final String fullName = String.format("%s:%d", getUri(pickle), pickle.getLine() ); System.out.println(fullName); filtered.add(new Object[]{first, second}); }); return filtered.iterator(); } private String getUri(final Pickle pickle) { final String testCaseUri = pickle.getUri().toString(); if (testCaseUri.startsWith(CUCUMBER_WORKING_DIR)) { return testCaseUri.substring(CUCUMBER_WORKING_DIR.length()); } if (testCaseUri.startsWith(CLASSPATH_PREFIX)) { return testCaseUri.substring(CLASSPATH_PREFIX.length()); } return testCaseUri; } }

For Step 2, the structure of the project should include:

Text
src/ └── test/ └── resources/ └── META-INF/ └── services/ └── org.testng.ITestNGListener

Environment information

For the main page of the report, you can collect various information about the environment in which the tests were executed.

For example, it is a good idea to use this to remember the OS and Java versions. This may help the future reader investigate bugs that are reproducible only in some environments.

Allure Report Environments Widget

To provide environment information, put a file named environment.properties into the allure-results directory after running the tests. See the example in Environment file.

Note that this feature should be used for properties that do not change for all tests in the report. If you have properties that can be different for different tests, consider using Parametrized tests.

Powered by
logo

Join our newsletter

Join our community

We aim to make Allure Report as reliable and user-friendly as possible, and together with the community, we're here to help when problems arise.

© 2024 Qameta Software Inc. All rights reserved.