Allure Cucumber-JVM
Generate beautiful HTML reports using Allure Report and your Cucumber-JVM tests.
How to start
This guide is tailored exclusively for Cucumber-JVM 7. While Allure maintains compatibility with older versions of Cucumber-JVM (version 4 onwards), configurations for these versions are not included to maintain simplicity in this documentation. The general configuration steps remain similar for older versions, but you will need to adjust the dependency and plugin names accordingly. For example, when working with Cucumber-JVM 6, the dependency would be
allure-cucumber6-jvmand the plugin to use would beio.qameta.allure.cucumber7jvm.AllureCucumber7Jvm.
To integrate Allure into an existing Cucumber-JVM project, you need to:
- Add Allure dependencies to your project.
- Activate the Allure Cucumber-JVM plugin.
- Set up AspectJ for
@Stepand@Attachmentannotations support. - Designate a location for Allure results storage.
The specific implementation of these steps varies based on how your Cucumber-JVM is set up. Cucumber-JVM can be used with three different types of runners:
- JUnit Platform (recommended)
- TestNG
- JUnit 4 (deprecated)
Using JUnit Platform
JUnit Platform is the recommended runner for Cucumber-JVM tests. Full configuration demonstrated in this example.
Add Allure dependencies
Enable Allure Cucumber-JVM plugin
Using TestNG
To ensure compatibility with Cucumber-JVM 7, you must use TestNG 7.8 or higher, which in turn requires Java 11 or newer. However, if you are also using Allure TestNG, which is compatible with TestNG 6, you need to be careful with the order of dependencies in your project. Declare the
io.qameta.allure:allure-testngdependency after theio.cucumber:cucumber-testngdependency, or any explicitly stated TestNG dependency, to avoid conflicts. This sequence is important to ensure that the correct version of TestNG is used.
Add Allure dependencies
Enable Allure Cucumber-JVM plugin
Add io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm plugin to your Cucumber Options:
Java@Test
@CucumberOptions(
plugin = {
"io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"
}
)
public class CucumberTest extends AbstractTestNGCucumberTests {
}Using JUnit 4
Add Allure dependencies
Enable Allure Cucumber-JVM plugin
Add io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm plugin to your Cucumber Options:
Java@RunWith(Cucumber.class)
@CucumberOptions(
plugin = {
"io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"
}
)
public class CucumberTest {
}Configure AspectJ
Allure leverages AspectJ for the functionality of @Step and @Attachment annotations.
Additionally, certain frameworks, such as JUnit 4 or AssertJ, rely on AspectJ integration to
function correctly. These frameworks will require appropriate configuration to ensure
they work as intended.
Specifying Allure Results location
Allure, by default, saves test results in the project's root directory. However, it is recommended to store your test results in the build output directory.
To configure this, create an allure.properties file and place it in the test resources directory of
your project, which is typically located at src/test/resources:
Running tests
Execute your Cucumber-JVM tests as you normally would. Below are the commands for Gradle and Maven users:
For Gradle:
For Maven:
After running the tests, Allure will gather the test execution data and store it in the allure-results directory.
You can then generate an HTML report from these results using Allure's reporting tools.
Writing tests
The Allure Cucumber-JVM adapter extends the standard reporting features of Cucumber-JVM by providing additional capabilities for crafting more informative and structured tests. This section highlights key enhancements that can be utilized:
- Metadata Annotation: Enhance test reports with descriptions, links, and other metadata.
- Test Organization: Structure your tests into clear hierarchies for better readability and organization.
- Step Division: Break down tests into smaller test steps for easier understanding and maintenance.
- Parametrized Tests: Clearly describe the parameters for parametrized tests to specify different scenarios.
- Attachments: Automatically capture screenshots and other files during test execution.
- Test Selection: Use a test plan file to select which tests to run, allowing for flexible test execution.
- Environment Details: Include comprehensive environment information to accompany the test report.
Adding Metadata
Allure allows you to enrich your reports with a variety of metadata. This additional information provides context and details for each test, enhancing the report's usefulness. Refer to the metadata reference section for an exhaustive list of what can be added.
You can assign metadata to a Cucumber-JVM features in two ways:
- Using Gherkin tags: Use a naming convention to add metadata to test scenarios.
- Runtime API: Use a method call within the body of a test step method to dynamically assign metadata during test runtime. This approach allows for the construction of strings and values on-the-fly.
When assigning metadata to your tests using Allure's Runtime API, it's crucial to consider the balance between control and potential risks. While the Runtime API offers more flexibility in handling metadata dynamically, it also comes with certain drawbacks. For instance, if a test fails before all the dynamic metadata is applied, the resulting report may lack important information, leading to reduced clarity and completeness. Therefore, it's essential to ensure that metadata is set early in the test to mitigate this risk.
To use Gherkin tags for assigning data, be sure to upgrade to Allure Report 2.27 or higher or consider uploading your test results to Allure TestOps.
Organize tests
Allure facilitates enhanced navigation in test reports by allowing tests to be organized into hierarchical structures. As explained in the section on Improving navigation in your test report, Allure Cucumber-JVM supports this feature.
To define a test's position within the behavior-based hierarchy:
To specify a test's location in the suite-based hierarchy:
A test's placement within the package-based hierarchy is determined by the location of the feature files, relative to test resources directory.
Creating Sub-Steps
Allure Cucumber-JVM enhances test reports by allowing the creation of sub-steps within a single Gherkin test step, which can be particularly useful in cases where you have a set of actions that are:
- Sufficiently cohesive to be described as one step in a Gherkin scenario.
- Intricate enough that they involve several potential failure points.
Allure provides three methods for defining sub-steps:
- Annotated Sub-Steps: Defined using
@Stepannotation. - Lambda Sub-Steps: Utilizing lambda expressions for more concise and inline step definitions.
- No-Op Sub-Steps: Employed as placeholders or for structuring steps without additional functionality.
For detailed guidance on implementing each type of sub-step, consult the test steps reference.
Support for Scenario Outlines
Allure Cucumber-JVM provides complete support for Scenario Outlines, a feature of Cucumber-JVM that allows for parametrized tests. No special setup is needed to take advantage of this capability within Allure.
GherkinFeature: Parameters
Scenario Outline: test authorization as "<login>"
Then authorize as "<login>"
Examples:
| login |
| johndoe |
| [email protected] |Javaimport io.cucumber.java.en.Then;
import io.qameta.allure.Allure;
public class CucumberSteps {
@Then("authorize as {string}")
public void testAuthentication(String login) {
// ...
}
}Furthermore, Allure's Runtime API can be utilized to include extra parameters in the test report, offering enhanced documentation capabilities and greater detail for each test scenario:
Javaimport io.cucumber.java.en.Then;
import io.qameta.allure.Allure;
public class CucumberSteps {
@Then("authorize as {string}")
public void testAuthentication(String login) {
Allure.parameter("Login", login);
// ...
}
}Attaching screenshots and other files
In Allure reports, you have the ability to attach various types of files, which can greatly enhance the comprehensibility of the report. A common practice is to attach screenshots that capture the state of the user interface at specific moments during test execution.
Allure Cucumber-JVM offers multiple methods for creating attachments, whether from pre-existing files or from content generated on-the-fly. For detailed instructions on how to implement attachments, refer to the attachments section in the Allure Cucumber-JVM reference.
Javaimport io.cucumber.java.en.Then;
import io.qameta.allure.Allure;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
public class CucumberSteps {
@When("^I open labels page$")
public void openLabelsPage() throws IOException {
Allure.attachment("data.txt", "This is the file content");
try (InputStream is = Files.newInputStream(Paths.get("/path/img.png"))) {
Allure.attachment("image.png", is);
}
// ...
}
}Selective test run (Allure TestOps)
In Allure TestOps, you have the option to rerun only a subset of tests by utilizing the same Continuous Integration (CI) job that was used for the full test run. The capability to perform selective test runs varies depending on the test runner you are using:
- JUnit Platform Runner: This runner supports selective test runs natively, allowing you to easily specify which tests to include in the rerun without any additional setup.
- JUnit 4 Runner: For Gradle users, JUnit 4 also supports selective test runs right out of the box. However, Maven users might encounter some issues with this functionality due to limitations within the Maven Surefire plugin.
- TestNG Runner: To use selective test runs with TestNG, you must be using TestNG version 7.8 or later. Additionally, some extra configurations might be needed to enable this feature.
TestNG
For integrating selective test runs in a TestNG project with Allure TestOps, follow these steps:
- Create
CucumberTestNgFilter: Implement a filter class namedCucumberTestNgFilter. This class will determine which tests to run based on your criteria. Place this class within your project's source directory, ensuring it's correctly packaged. - Register
CucumberTestNgFilter: In your project, create a file namedMETA-INF/services/org.testng.ITestNGListenerin your test resources. Inside this file, write the fully qualified name of yourCucumberTestNgFilterclass. This step is crucial as it registers your filter as a TestNG listener, which TestNG will invoke during the test run.
Java// Step 1: Create CucumberTestNgFilter.java
package com.yourproject.filters;
import io.cucumber.testng.FeatureWrapper;
import io.cucumber.testng.Pickle;
import io.cucumber.testng.PickleWrapper;
import org.testng.IDataProviderInterceptor;
import org.testng.IDataProviderMethod;
import org.testng.ITestContext;
import org.testng.ITestNGMethod;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class CucumberTestNgFilter implements IDataProviderInterceptor {
private static final String CUCUMBER_WORKING_DIR
= Paths.get("").toUri().toString();
private static final String CLASSPATH_PREFIX = "classpath:";
private static final String CUCUMBER_RUN_METHOD
= "io.cucumber.testng.AbstractTestNGCucumberTests.runScenario";
@Override
public Iterator<Object[]> intercept(
final Iterator<Object[]> original,
final IDataProviderMethod dataProviderMethod,
final ITestNGMethod method,
final ITestContext iTestContext
) {
if (!CUCUMBER_RUN_METHOD.equals(method.getQualifiedName())) {
return original;
}
final List<Object[]> filtered = new ArrayList<>();
original.forEachRemaining(objects -> {
if (objects.length != 2) {
filtered.add(objects);
}
final PickleWrapper first = (PickleWrapper) objects[0];
final FeatureWrapper second = (FeatureWrapper) objects[1];
final Pickle pickle = first.getPickle();
final String fullName = String.format("%s:%d",
getUri(pickle),
pickle.getLine()
);
System.out.println(fullName);
filtered.add(new Object[]{first, second});
});
return filtered.iterator();
}
private String getUri(final Pickle pickle) {
final String testCaseUri = pickle.getUri().toString();
if (testCaseUri.startsWith(CUCUMBER_WORKING_DIR)) {
return testCaseUri.substring(CUCUMBER_WORKING_DIR.length());
}
if (testCaseUri.startsWith(CLASSPATH_PREFIX)) {
return testCaseUri.substring(CLASSPATH_PREFIX.length());
}
return testCaseUri;
}
}For Step 2, the structure of the project should include:
src/ └── test/ └── resources/ └── META-INF/ └── services/ └── org.testng.ITestNGListener
Environment information
For the main page of the report, you can collect various information about the environment in which the tests were executed.
For example, it is a good idea to use this to remember the OS and Java versions. This may help the future reader investigate bugs that are reproducible only in some environments.
To provide environment information, put a file named environment.properties into the allure-results directory after
running the tests. See the example in Environment file.
Note that this feature should be used for properties that do not change for all tests in the report. If you have properties that can be different for different tests, consider using Parametrized tests.
