Getting started with Allure xUnit.net

Generate beautiful HTML reports using Allure Report and your xUnit.net tests.

INFO

Check out the example project at github.com/allure-examples/allure-xunit to see Allure xUnit.net in action.

Setting up

1. Check the prerequisites

1. Make sure the Allure Report command-line tool is installed. If it's not, see the installation instructions.

2. Check the .NET SDK against which your test project is built. Allure xUnit.net supports the following SDKs:

   • .NET Core 3.1
   • .NET 5.0 and later

   Visit the official documentation for more info about installing or upgrading .NET SDK.

3. Make sure your test project references a compatible version of xUnit.net. Currently, we support xUnit.net v2 starting from 2.4.1.

   The support for xUnit.net v3 will be added in the future (you may track this task in this issue).

   INFO

   Allure xUnit.net can't work with version 3 of xunit.runner.visualstudio due to changes in how the runner discovers custom reporters. You should downgrade to version 2.8.2 or earlier to keep using it with Allure xUnit.net.

2. Prepare your project

1. Add Allure.Xunit to your project's dependencies using your IDE or the command line.

   For example, here is how the dependency can be added to a project with the dotnet CLI:

   dotnet add ⟨PATH TO PROJECT⟩ package Allure.Xunit

2. Explicitly select allure as the xUnit.net runner reporter.

   This step is only necessary if you have two or more “environmentally enabled” xUnit.net reporters, which means that the test runner may or may not use the Allure xUnit.net reporter each time by default. This is often the case when running tests on a CI platform, such as Microsoft Azure or TeamCity, because a reporter that sends test results to the CI platform might be enabled in addition to Allure xUnit.net in such environment.

   The way to select the reporter depends on the test runner that you use in the project. For example, if you use xUnit.net's default xunit.runner.visualstudio runner, you need to modify the ReporterSwitch setting:

   In the .runsettings file

   <?xml version="1.0" encoding="utf-8"?>
   <RunSettings>
     <RunConfiguration>
       <ReporterSwitch>allure</ReporterSwitch>
     </RunConfiguration>
   </RunSettings>

   In the “dotnet test” arguments

   dotnet test -- RunConfiguration.ReporterSwitch=allure

   TIP

   By default, Allure xUnit.net will also try to run another environmentally enabled reporter, if it is present. This behavior can be changed using the xunitRunnerReporter configuration option.

3. Run tests

Run your xUnit.net tests the same way as you would run them usually. For example:

dotnet test

This will save necessary data into allure-results or other directory, according to the allure.directory setting. If the directory already exists, the new files will be added to the existing ones, so that a future report will be based on them all.

4. Generate a report

Finally, convert the test results into an HTML report. This can be done by one of two commands:

• allure generate processes the test results and saves an HTML report into the allure-report directory. To view the report, use the allure open command.

  Use this command if you need to save the report for future reference or for sharing it with colleagues.

• allure serve creates the same report as allure generate but puts it into a temporary directory and starts a local web server configured to show this directory's contents. The command then automatically opens the main page of the report in a web browser.

  Use this command if you need to view the report for yourself and do not need to save it.

Writing tests

The Allure xUnit.net adapter extends the standard reporting features of xUnit.net by providing additional capabilities for crafting more informative and structured tests. This section highlights key enhancements that can be utilized:

• Metadata Annotation: Enhance test reports with descriptions, links, and other metadata.
• Test Organization: Structure your tests into clear hierarchies for better readability and organization.
• Test statuses distinction: Distinguish between failed assertions and unhandled exceptions with failed and broken statuses.
• Step Division: Break down tests into smaller test steps for easier understanding and maintenance.
• Constructors and Dispose methods: Include the initialization and finalization of shared contexts into the report.
• Parametrized Tests: Clearly describe the parameters for parametrized tests to specify different scenarios.
• Attachments: Automatically capture screenshots and other files during test execution.
• Test Selection: Use a test plan file to select which tests to run, allowing for flexible test execution.
• Environment Details: Include comprehensive environment information to accompany the test report.

In most cases, Allure xUnit.net provides two different ways to use a feature: the Attributes API and the Runtime API.

• Attributes API: add a C# attribute to a test method or a whole class to add certain data to the test result. When using this approach, the data is guaranteed to be added regardless of how the test itself runs.

• Runtime API: use Allure's functions to add certain data to the test result during its execution. This approach allows for constructing the data dynamically.

  Note that it is recommended to call the Allure's functions as close to the beginning of the test as possible. This way, the data will be added even if the test fails early.

Specify description, links and other metadata

There is a lot of metadata you can add to each test so that it would appear in the report. See the reference for more details.

Attributes API

using Allure.Net.Commons;
using Allure.Xunit.Attributes;
using Xunit;

public class TestLabels
{
    [Fact]
    [AllureSeverity(SeverityLevel.critical)]
    [AllureOwner("John Doe")]
    [AllureLink("Website", "https://dev.example.com/")]
    [AllureIssue("UI-123")]
    [AllureTms("TMS-456")]
    public void TestCreateLabel()
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Xunit;

public class TestLabels
{
    [Fact]
    public void TestCreateLabel()
    {
        AllureApi.SetSeverity(SeverityLevel.critical);
        AllureApi.SetOwner("John Doe");
        AllureApi.AddLink("Website", "https://dev.example.com/");
        AllureApi.AddIssue("UI-123");
        AllureApi.AddTmsItem("TMS-456");
        // ...
    }
}

Organize tests

As described in Improving navigation in your test report, Allure supports multiple ways to organize tests into hierarchical structures. Allure xUnit.net provides the API to assign the relevant fields to tests either by adding attributes or “dynamically” (same as for the metadata fields).

To specify a test's location in the behavior-based hierarchy:

Attributes API

using Allure.Xunit.Attributes;
using Xunit;

[AllureEpic("Web interface")]
[AllureFeature("Essential features")]
public class TestLabels
{
    [Fact]
    [AllureStory("Labels")]
    public void TestCreateLabel()
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Xunit;

public class TestLabels
{
    [Fact]
    public void TestCreateLabel()
    {
        AllureApi.AddEpic("Web interface");
        AllureApi.AddFeature("Essential features");
        AllureApi.AddStory("Labels");
        // ...
    }
}

To specify a test's location in the suite-based hierarchy:

Attributes API

using Allure.Xunit.Attributes;
using Xunit;

[AllureParentSuite("Web interface")]
[AllureSuite("Essential features")]
public class TestLabels
{
    [Fact]
    [AllureSubSuite("Labels")]
    public void TestCreateLabel()
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Xunit;

public class TestLabels
{
    [Fact]
    public void TestCreateLabel()
    {
        AllureApi.AddParentSuite("Web interface");
        AllureApi.AddSuite("Essential features");
        AllureApi.AddSubSuite("Labels");
        // ...
    }
}

A test's location in the package-based hierarchy is defined by the fully qualified names of the classes they are declared in, with common prefixes shown as parent packages.

Distinguish between test statuses

When a test throws an exception, Allure xUnit.net reflects that by assigning an appropriate test status:

• If the exception class belongs to the Xunit.Sdk namespace, the test is considered Failed.
• With any other exception, the test is considered Broken.

If you want your test with unhandled exceptions to be Failed instead of Broken, there are multiple approaches to achieve that. For example, in an assertion (e.g., in Assert.Equal), make sure that evaluating the arguments (e.g., accessing an element of a list) cannot cause an exception.

Alternatively, consider wrapping the code that may throw in a try..catch block and fail the test explicitly.

Failed test

using Xunit;

public class TestTryCatch
{
    [Fact]
    public void ThisTestWillBeFailed()
    {
        string value = GetValueOrFail();
        Assert.Equal("foo", value);
    }

    static string GetValueOrFail()
    {
        string value = default;
        try
        {
            value = GetValue();
        }
        catch (Exception e)
        {
            Assert.Fail(e.Message);
        }

        return value;
    }

    static string GetValue()
    {
        throw new Exception();
    }
}

Broken test

using Xunit;

public class TestBroken
{
    [Fact]
    public void ThisTestWillBeBroken()
    {
        string value = GetValue();
        Assert.Equal("foo", value);
    }

    static string GetValue()
    {
        throw new Exception();
    }
}

INFO

When an exception is thrown from within a step, the step status is selected according to the allure.failExceptions setting.

Divide a test into steps

Allure xUnit.net provides three ways of creating steps and sub-steps: “attribute-based steps”, “lambda steps” and “no-op steps”, see the reference.

Attribute-based steps

using Allure.Net.Commons;
using Allure.Xunit.Attributes.Steps;
using Xunit;

public class TestLabels
{
    [Fact]
    public void TestCreateLabel()
    {
        TestDomain("https://domain1.example.com/");
        TestDomain("https://domain2.example.com/");
    }

    [AllureStep("Test {url}")]
    private void TestDomain([Name("Webpage URL")] string url)
    {
        AllureLifecycle.Instance.UpdateStep(stepResult =>
            stepResult.parameters.Add(
                new Parameter {
                    name = "Started at",
                    value = DateTime.Now.ToString()
                }
            )
        );
        OpenBrowser();
        GoToWebpage(url);
        CloseBrowser();
    }

    [AllureStep("Open web browser")]
    private void OpenBrowser()
    {
        // ...
    }

    [AllureStep("Visit {url}")]
    private void GoToWebpage([Skip] string url)
    {
        // ...
    }

    [AllureStep("Close web browser")]
    private void CloseBrowser()
    {
        // ...
    }
}

Lambda steps and no-op steps

using Allure.Net.Commons;
using Xunit;

public class TestLabels
{
    [Fact]
    public void TestCreateLabel()
    {
        string[] urls =
        {
            "https://domain1.example.com/",
            "https://domain2.example.com/"
        };

        foreach (string url in urls)
        {
            AllureApi.Step($"Test {url}", () =>
            {
                AllureLifecycle.Instance.UpdateStep(stepResult =>
                {
                    stepResult.parameters.Add(
                        new Parameter { name = "Webpage URL", value = url }
                    );
                });

                AllureApi.Step("Opening web browser...");
                // ...

                AllureApi.Step($"Visiting {url}...");
                // ...

                AllureApi.Step("Closing web browser...");
                // ...
            });
        }
    }
}

Describe constructors and Dispose methods

By default, a test report does not include information about constructors and Dispose methods in xUnit.net test classes. To include it, add the [AllureBefore] and [AllureAfter] annotations to the methods that setup and clean a test context.

The methods will be displayed as special kinds of steps.

using Allure.Xunit.Attributes.Steps;
using Xunit;

public class TestLabels : IDisposable
{
    [AllureBefore("Setup test context")]
    public TestLabels()
    {
        // ...
    }

    [Fact]
    public void TestCreateLabel()
    {
        // ...
    }

    [AllureAfter("Clean test context")]
    public void Dispose()
    {
        // ...
    }
}

Describe parametrized tests

When using the parametrized tests pattern, use [Theory] and other xUnit.net's attributes or or the AddTestParameter() function, see the reference.

Attributes API

using Xunit;

public class TestLabels
{
    [Theory]
    [InlineData("johndoe", "qwerty")]
    [InlineData("[email protected]", "qwerty")]
    public void TestCreateLabel(string login, string password)
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Xunit;

public class TestLabels
{
    [Fact]
    public void TestAuthenticationWithUsername()
    {
        AllureApi.AddTestParameter("login", "johndoe");
        AllureApi.AddTestParameter("password", "qwerty", ParameterMode.Masked);
        // ...
    }

    [Fact]
    public void TestAuthenticationWithEmail()
    {
        AllureApi.AddTestParameter("login", "[email protected]");
        AllureApi.AddTestParameter("password", "qwerty", ParameterMode.Masked);
        // ...
    }
}

Attach screenshots and other files

You can attach any sorts of files to your Allure report. For example, a popular way to make a report easier to understand is to attach a screenshot of the user interface at a certain point.

Allure xUnit.net provides various ways to create an attachment, both from existing files or generated dynamically, see the reference.

Runtime API

using System.IO;
using System.Text;
using Allure.Net.Commons;
using Xunit;

public class TestLabels
{
    [Fact]
    public void TestCreateLabel()
    {
        // ...
        AllureApi.AddAttachment(
            "data.txt",
            "text/plain",
            Encoding.UTF8.GetBytes("This is the file content.")
        );
        AllureApi.AddAttachment(
            "image1.png",
            "image/png",
            File.ReadAllBytes("/path/to/image1.png")
        );
        AllureApi.AddAttachment(
            "image2.png",
            "image/png",
            "/path/to/image2.png"
        );
    }
}

Select tests via a test plan file

If the ALLURE_TESTPLAN_PATH environment variable is defined and points to an existing file, xUnit.net will only run tests listed in this file.

Here's an example of running tests according to a file named testplan.json:

MacOS/Linux

export ALLURE_TESTPLAN_PATH=testplan.json
dotnet test

Windows

$Env:ALLURE_TESTPLAN_PATH = "testplan.json"
dotnet test

Environment information

For the main page of the report, you can collect various information about the environment in which the tests were executed.

For example, it is a good idea to use this to remember the OS version and .NET version. This may help the future reader investigate bugs that are reproducible only in some environments.

To provide environment information, put a file named environment.properties into the allure-results directory after running the tests. See the example in Environment file.

Note that this feature should be used for properties that do not change for all tests in the report. If you have properties that can be different for different tests, consider using Parametrized tests.