Xray integration Allure 3

Allure Report 3 can update the statuses of test runs in Xray for Jira Server and Jira Data Center. Xray Cloud is not supported. The xray plugin that does this is included in the allure package.

For each test in the report that is linked to an Xray Test, Allure finds the test runs of this Xray Test in the Test Executions that you specify, and sets their status to PASS, FAIL or TODO. If the address of the report is known, it then adds a comment with a link to the report to each Test Execution.

The plugin doesn't create Tests, Test Executions or test runs, and doesn't import test results to Xray.

Prerequisites

• Jira Server or Jira Data Center with the Xray app installed.
• The Test Executions already exist and contain the Tests whose runs you want to update.
• The Jira user, or the owner of the personal access token, can browse the Test Executions, edit their test runs, and add comments.

Link tests to Xray Tests

To link a test to an Xray Test, add a link of the tms type to the test, with the Jira key of the Xray Test as the link name, for example, XT-1. If the name is not a Jira key, Allure takes the first key from the link URL.

The link must have a URL: Allure Report 3 ignores links without one. In the Allure integrations for Java, a link added with @TmsLink gets a URL only if the TMS link pattern is configured, so set it in the allure.properties file:

allure.link.tms.pattern=https://jira.example.com/browse/{}

@TmsLink("XT-1")
@Test
void shouldLogIn() {
    // ...
}

In other Allure integrations, see the documentation of your integration for how to add links.

If several test results are linked to the same Xray Test, for example, a parameterized test, Allure combines their statuses:

Results linked to the Xray Test	Status in Xray
At least one failed or broken	FAIL
At least one passed, none failed or broken	PASS
Only skipped or unknown	TODO

For a test that was retried, only the latest attempt counts.

Configure the plugin

Add the xray plugin to the plugins section of the configuration file, next to the plugin of your report, and set the Jira address and the Test Executions to update:

allurerc.mjs

import { defineConfig } from "allure";

export default defineConfig({
  plugins: {
    awesome: {},
    xray: {
      options: {
        endpoint: "https://jira.example.com",
        executions: ["XT-6", "XT-7"],
      },
    },
  },
});

allurerc.json

{
  "plugins": {
    "awesome": {},
    "xray": {
      "options": {
        "endpoint": "https://jira.example.com",
        "executions": ["XT-6", "XT-7"]
      }
    }
  }
}

allurerc.yaml

plugins:
  awesome: {}
  xray:
    options:
      endpoint: https://jira.example.com
      executions: [XT-6, XT-7]

With this configuration, after generating the report, Allure updates the test runs in the Test Executions XT-6 and XT-7 in Jira at https://jira.example.com. The credentials are not in the configuration file: Allure takes the personal access token from the ALLURE_XRAY_TOKEN environment variable, which you can set, for example, as a secret of your CI pipeline.

Allure updates only the test runs whose status differs from the result. Test runs of Xray Tests that no test in the report links to are left as they are.

Every option can also be provided via an environment variable. If an option is set in the configuration file, the environment variable is not used. The table below lists all the options.

Options

Option	Environment variable	Description	Default
endpoint	ALLURE_XRAY_ENDPOINT	Jira address. Required
token	ALLURE_XRAY_TOKEN	Jira personal access token. Required, unless username and password are set. If set, username and password are not used
username	ALLURE_XRAY_USERNAME	Jira user name, used with password instead of a token
password	ALLURE_XRAY_PASSWORD	Jira password, used with username instead of a token
executions	ALLURE_XRAY_EXECUTIONS	Keys of the Test Executions to update. Required. In the environment variable, a comma-separated list, for example, XT-6,XT-7
statuses	ALLURE_XRAY_STATUS_<STATUS>	Xray status to set for each Allure test status, see Change the status mapping	Default mapping
comment	ALLURE_XRAY_COMMENT	Whether to add a comment with a link to the report, see Comment with a link to the report	true

Change the status mapping

By default, Allure sets these statuses in Xray:

Allure status	Xray status
Passed	PASS
Failed	FAIL
Broken	FAIL
Skipped	TODO
Unknown	TODO

To change the mapping, set the statuses option. It only needs the statuses you want to change, the others keep their default mapping. For example, this configuration sets FAIL in Xray for skipped tests:

allurerc.mjs

import { defineConfig } from "allure";

export default defineConfig({
  plugins: {
    awesome: {},
    xray: {
      options: {
        endpoint: "https://jira.example.com",
        executions: ["XT-6", "XT-7"],
        statuses: { skipped: "FAIL" },
      },
    },
  },
});

allurerc.json

{
  "plugins": {
    "awesome": {},
    "xray": {
      "options": {
        "endpoint": "https://jira.example.com",
        "executions": ["XT-6", "XT-7"],
        "statuses": { "skipped": "FAIL" }
      }
    }
  }
}

allurerc.yaml

plugins:
  awesome: {}
  xray:
    options:
      endpoint: https://jira.example.com
      executions: [XT-6, XT-7]
      statuses:
        skipped: FAIL

You can also use environment variables: ALLURE_XRAY_STATUS_PASSED, ALLURE_XRAY_STATUS_FAILED, ALLURE_XRAY_STATUS_BROKEN, ALLURE_XRAY_STATUS_SKIPPED and ALLURE_XRAY_STATUS_UNKNOWN, for example, ALLURE_XRAY_STATUS_SKIPPED=FAIL.

Only the PASS, FAIL and TODO statuses are supported.

Comment with a link to the report

After updating the test runs of a Test Execution, Allure adds a comment with a link to the report to the Test Execution. The comment is added only if the address of the report is known, for example, from historyBaseUrl, and only if all test runs of the Test Execution were read and updated successfully.

The comment is added on every report generation. To turn it off, set comment: false or ALLURE_XRAY_COMMENT=false.

Troubleshooting

If a request to Jira fails, Allure prints the error to the console and continues with the next request. The report is generated anyway. For example:

[Allure Xray Plugin] failed to read test runs of XT-7: Request failed with status code 404 (HTTP 404)

An HTTP 401 or 403 error usually means missing credentials or permissions.

If the Jira address, the credentials or the Test Executions are not set, or a status in the mapping is not supported, Allure prints an error and doesn't update anything.

If a test run isn't updated, check that:

• the test has a link of the tms type with a URL;
• the link name, or the link URL, contains the key of the Xray Test;
• one of the Test Executions contains a test run of this Xray Test.