Getting started with Allure NUnit

Generate beautiful HTML reports using Allure Report and your NUnit tests.

INFO

Check out the example project at github.com/allure-examples/allure-nunit to see Allure NUnit in action.

Setting up

1. Prepare your project

1. Install the Allure Report command-line tool, if it is not yet installed in your operating system. Note that Allure Report requires Java, see the installation instructions.

2. Make sure that you have a compatible .NET version.

   Allure NUnit requires a framework that implements .NET Standard version 2.0. You can find the full list of compatible frameworks on Allure.NUnit's page in NuGet.

3. Add Allure.NUnit to your project's dependencies using your IDE or the command line.

   For example, here is how the dependency can be added to a project with the dotnet CLI:

   dotnet add ⟨PATH TO PROJECT⟩ package Allure.NUnit

4. Add the [AllureNUnit] attribute to all test classes in your project. For example:

   using Allure.NUnit;
   using NUnit.Framework;
   
   [AllureNUnit]
   class TestLabels
   {
       [Test]
       public void TestCreateLabel()
       {
           // ...
       }
   }

   TIP

   If you have a common base class for all your tests, you can just add the [AllureNUnit] attribute to that class.

2. Run tests

Run your NUnit tests the same way as you would run them usually. For example:

dotnet test

This will save necessary data into allure-results or other directory, according to the allure.directory setting. If the directory already exists, the new files will be added to the existing ones, so that a future report will be based on them all.

3. Generate a report

Finally, convert the test results into an HTML report. This can be done by one of two commands:

• allure generate processes the test results and saves an HTML report into the allure-report directory. To view the report, use the allure open command.

  Use this command if you need to save the report for future reference or for sharing it with colleagues.

• allure serve creates the same report as allure generate but puts it into a temporary directory and starts a local web server configured to show this directory's contents. The command then automatically opens the main page of the report in a web browser.

  Use this command if you need to view the report for yourself and do not need to save it.

Writing tests

The Allure NUnit adapter extends the standard reporting features of NUnit by providing additional capabilities for crafting more informative and structured tests. This section highlights key enhancements that can be utilized:

• Metadata Annotation: Enhance test reports with descriptions, links, and other metadata.
• Test Organization: Structure your tests into clear hierarchies for better readability and organization.
• Test statuses distinction: Distinguish between failed assertions and unhandled exceptions with failed and broken statuses.
• Step Division: Break down tests into smaller test steps for easier understanding and maintenance.
• Test Fixtures: Include the setup and teardown methods into the report.
• Parametrized Tests: Clearly describe the parameters for parametrized tests to specify different scenarios.
• Attachments: Automatically capture screenshots and other files during test execution.
• Test Selection: Use a test plan file to select which tests to run, allowing for flexible test execution.
• Environment Details: Include comprehensive environment information to accompany the test report.

In most cases, Allure NUnit provides two different ways to use a feature: the Attributes API and the Runtime API.

• Attributes API: add a C# attribute to a test method or a whole class to add certain data to the test result. When using this approach, the data is guaranteed to be added regardless of how the test itself runs.

• Runtime API: use Allure's functions to add certain data to the test result during its execution. This approach allows for constructing the data dynamically.

  Note that it is recommended to call the Allure's functions as close to the beginning of the test as possible. This way, the data will be added even if the test fails early.

Specify description, links and other metadata

There is a lot of metadata you can add to each test so that it would appear in the report. See the reference for more details.

Attributes API

using Allure.Net.Commons;
using Allure.NUnit;
using Allure.NUnit.Attributes;
using NUnit.Framework;

[AllureNUnit]
class TestLabels
{
    [Test]
    [AllureSeverity(SeverityLevel.critical)]
    [AllureOwner("John Doe")]
    [AllureLink("Website", "https://dev.example.com/")]
    [AllureIssue("UI-123")]
    [AllureTms("TMS-456")]
    public void TestCreateLabel()
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestLabels
{
    [Test]
    public void TestCreateLabel()
    {
        AllureApi.SetSeverity(SeverityLevel.critical);
        AllureApi.SetOwner("John Doe");
        AllureApi.AddLink("Website", "https://dev.example.com/");
        AllureApi.AddIssue("UI-123");
        AllureApi.AddTmsItem("TMS-456");
        // ...
    }
}

Organize tests

As described in Improving navigation in your test report, Allure supports multiple ways to organize tests into hierarchical structures. Allure NUnit provides the API to assign the relevant fields to tests either by adding attributes or “dynamically” (same as for the metadata fields).

To specify a test's location in the behavior-based hierarchy:

Attributes API

using Allure.NUnit;
using Allure.NUnit.Attributes;
using NUnit.Framework;

[AllureNUnit]
[AllureEpic("Web interface")]
[AllureFeature("Essential features")]
class TestLabels
{
    [Test]
    [AllureStory("Labels")]
    public void TestCreateLabel()
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestLabels
{
    [Test]
    public void TestCreateLabel()
    {
        AllureApi.AddEpic("Web interface");
        AllureApi.AddFeature("Essential features");
        AllureApi.AddStory("Labels");
        // ...
    }
}

To specify a test's location in the suite-based hierarchy:

Attributes API

using Allure.NUnit;
using Allure.NUnit.Attributes;
using NUnit.Framework;

[AllureNUnit]
[AllureParentSuite("Web interface")]
[AllureSuite("Essential features")]
class TestLabels
{
    [Test]
    [AllureSubSuite("Labels")]
    public void TestCreateLabel()
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestLabels
{
    [Test]
    public void TestCreateLabel()
    {
        AllureApi.AddParentSuite("Web interface");
        AllureApi.AddSuite("Essential features");
        AllureApi.AddSubSuite("Labels");
        // ...
    }
}

A test's location in the package-based hierarchy is defined by the fully qualified names of the classes they are declared in, with common prefixes shown as parent packages.

Distinguish between test statuses

A test may be interrupted for various reasons. Allure NUnit reflects that by assigning an appropriate test status:

• Failed for a failed assertion, e.g., Assert.That,
• Skipped for failed assumptions, e.g., Assume.That,
• Broken for unhandled exceptions.

Note that if the test uses techniques like Assert.Multiple, it will not necessarily stop after the first Assert.That failure. This means that it may end up with a list of multiple failed assertions. For such a test, Allure NUnit will still assign the Failed status if the failed assertions are the only reason the test did not succeed, but Broken if it eventually stopped due to an unhandled exception.

If you want your test with unhandled exceptions to be Failed instead of Broken, there are multiple approaches to achieve that. For example, in an Assert.That, make sure that evaluating the first argument (e.g., accessing an element of a list) cannot cause an exception. If necessary, move some logic to a custom constraint to avoid repetitive assertions in your code.

Alternatively, consider wrapping the code that may throw in some construct that takes care of exceptions. It can be an assertion or assumption with the Throws.Nothing constraint or just a try..catch block.

Failed (with Throws.Nothing)

using System;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
public class TestThrowsNothing
{
    [Test]
    public void ThisTestWillBeFailed()
    {
        string value = GetValueOrFail();
        Assert.That(value, Is.EqualTo("foo"));
    }

    static string GetValueOrFail()
    {
        string value = default;
        Assert.That(() => { value = GetValue(); }, Throws.Nothing);
        return value;
    }

    static string GetValue()
    {
        throw new Exception();
    }
}

Failed (with try..catch)

using System;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
public class TestTryCatch
{
    [Test]
    public void ThisTestWillBeFailed()
    {
        string value = GetValueOrFail();
        Assert.That(value, Is.EqualTo("foo"));
    }

    static string GetValueOrFail()
    {
        string value = default;

        try
        {
            value = GetValue();
        }
        catch (Exception e)
        {
            Assert.Fail(e.Message);
        }

        return value;
    }

    static string GetValue()
    {
        throw new Exception();
    }
}

Broken test

using System;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
public class TestBroken
{
    [Test]
    public void ThisTestWillBeBroken()
    {
        string value = GetValue();
        Assert.That(value, Is.EqualTo("foo"));
    }

    static string GetValue()
    {
        throw new Exception();
    }
}

INFO

When an exception is thrown from within a step, the step status is selected according to the allure.failExceptions setting.

Divide a test into steps

Allure NUnit provides three ways of creating steps and sub-steps: “attribute-based steps”, “lambda steps” and “no-op steps”, see the reference.

Attribute-based steps

using System;
using Allure.Net.Commons;
using Allure.NUnit;
using Allure.NUnit.Attributes;
using NUnit.Framework;

[AllureNUnit]
class TestLabels
{
    [Test]
    public void TestCreateLabel()
    {
        TestDomain("https://domain1.example.com/");
        TestDomain("https://domain2.example.com/");
    }

    [AllureStep("Test {url}")]
    private void TestDomain([Name("Webpage URL")] string url)
    {
        AllureLifecycle.Instance.UpdateStep(stepResult =>
            stepResult.parameters.Add(
                new Parameter
                {
                    name = "Started at", value = DateTime.Now.ToString()
                }
            )
        );
        OpenBrowser();
        GoToWebpage(url);
        CloseBrowser();
    }

    [AllureStep("Open web browser")]
    private void OpenBrowser()
    {
        // ...
    }

    [AllureStep("Visit {url}")]
    private void GoToWebpage([Skip] string url)
    {
        // ...
    }

    [AllureStep("Close web browser")]
    private void CloseBrowser()
    {
        // ...
    }
}

Lambda steps and no-op steps

using Allure.Net.Commons;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestMyWebsite
{
    [Test]
    public void TestVisitPages()
    {
        string[] urls =
        {
            "https://domain1.example.com/",
            "https://domain2.example.com/"
        };

        foreach (string url in urls)
        {
            AllureApi.Step($"Test {url}", () =>
            {
                AllureLifecycle.Instance.UpdateStep(stepResult =>
                {
                    stepResult.parameters.Add(
                        new Parameter { name = "Webpage URL", value = url }
                    );
                });

                AllureApi.Step("Opening web browser...");
                // ...

                AllureApi.Step($"Visiting {url}...");
                // ...

                AllureApi.Step("Closing web browser...");
                // ...
            });
        }
    }
}

Describe test fixtures

By default, a test report does not include information about fixtures in NUnit test classes. To include it, add the [AllureBefore] and [AllureAfter] annotations to the fixture methods.

The methods will be displayed as special kinds of steps.

Attributes API

using Allure.NUnit;
using Allure.NUnit.Attributes;
using NUnit.Framework;

[AllureNUnit]
public class TestLabels
{

    [OneTimeSetUp]
    [AllureBefore("Start the test server")]
    public void StartServer()
    {
        // ...
    }

    [SetUp]
    [AllureBefore("Open the browser")]
    public void StartBrowser()
    {
        // ...
    }

    [Test]
    public void TestCreateLabel()
    {
        // ...
    }

    [TearDown]
    [AllureAfter("Close the browser")]
    public void CloseBrowser()
    {
        // ...
    }

    [OneTimeTearDown]
    [AllureAfter("Shutdown the test server")]
    public void StopServer()
    {
        // ...
    }
}

Attributes API + Runtime API

using Allure.Net.Commons;
using Allure.NUnit;
using Allure.NUnit.Attributes;
using NUnit.Framework;

[AllureNUnit]
public class TestFixtures
{
    [OneTimeSetUp]
    [AllureBefore]
    public void StartServer()
    {
        AllureApi.SetFixtureName("Start the test server");
        // ...
    }

    [SetUp]
    [AllureBefore]
    public void StartBrowser()
    {
        AllureApi.SetFixtureName("Open the browser");
        // ...
    }

    [Test]
    public void Test_SetUp()
    {
        // ...
    }

    [TearDown]
    [AllureAfter]
    public void CloseBrowser()
    {
        AllureApi.SetFixtureName("Close the browser");
        // ...
    }

    [OneTimeTearDown]
    [AllureAfter]
    public void StopServer()
    {
        AllureApi.SetFixtureName("Shutdown the test server");
        // ...
    }
}

Describe parametrized tests

When using the parametrized tests pattern, use the NUnit's attributes or the AddTestParameter() function, see the reference.

Attributes API

using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestLabels
{
    [TestCase("johndoe", "qwerty")]
    [TestCase("[email protected]", "qwerty")]
    public void TestAuthentication(string login, string password)
    {
        // ...
    }
}

Runtime API

using Allure.Net.Commons;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestAuthenticaton
{
    [Test]
    public void TestAuthenticationWithUsername()
    {
        AllureApi.AddTestParameter("login", "johndoe");
        AllureApi.AddTestParameter("password", "qwerty", ParameterMode.Masked);
        // ...
    }

    [Test]
    public void TestAuthenticationWithEmail()
    {
        AllureApi.AddTestParameter("login", "[email protected]");
        AllureApi.AddTestParameter("password", "qwerty", ParameterMode.Masked);
        // ...
    }
}

Attach screenshots and other files

You can attach any sorts of files to your Allure report. For example, a popular way to make a report easier to understand is to attach a screenshot of the user interface at a certain point.

Allure NUnit provides various ways to create an attachment, both from existing files or generated dynamically, see the reference.

using System.IO;
using System.Text;
using Allure.Net.Commons;
using Allure.NUnit;
using NUnit.Framework;

[AllureNUnit]
class TestLabels
{
    [Test]
    public void TestCreateLabel()
    {
        // ...
        AllureApi.AddAttachment(
            "data.txt",
            "text/plain",
            Encoding.UTF8.GetBytes("This is the file content.")
        );
        AllureApi.AddAttachment(
            "image1.png",
            "image/png",
            File.ReadAllBytes("/path/to/image1.png")
        );
        AllureApi.AddAttachment(
            "image2.png",
            "image/png",
            "/path/to/image2.png"
        );
    }
}

Select tests via a test plan file

If the ALLURE_TESTPLAN_PATH environment variable is defined and points to an existing file, NUnit will only run tests listed in this file.

Here's an example of running tests according to a file named testplan.json:

MacOS/Linux

export ALLURE_TESTPLAN_PATH=testplan.json
dotnet test

Windows

$Env:ALLURE_TESTPLAN_PATH = "testplan.json"
dotnet test

Environment information

For the main page of the report, you can collect various information about the environment in which the tests were executed.

For example, it is a good idea to use this to remember the OS version and .NET version. This may help the future reader investigate bugs that are reproducible only in some environments.

To provide environment information, put a file named environment.properties into the allure-results directory after running the tests. See the example in Environment file.

Note that this feature should be used for properties that do not change for all tests in the report. If you have properties that can be different for different tests, consider using Parametrized tests.