Getting started with Allure Tinybench

Record the results of Tinybench benchmarks as performance metrics in your Allure Report.

Setting up

1. Prepare your project

1. Make sure Node.js is installed.

2. Make sure you have Tinybench installed. Allure Tinybench supports Tinybench 6.

3. Make sure Allure Report 3 is installed. If it's not, follow the installation instructions.

   Alternatively, install Allure Report 3 as a local npm package:

   npm install --save-dev allure

4. Allure Tinybench does not run on its own: you call it inside a test that runs with an Allure integration for JavaScript, such as Allure Vitest. Install it together with the integration.

   For example, to use it with Vitest:

   npm

   npm install --save-dev allure-tinybench allure-js-commons tinybench allure-vitest vitest

   yarn

   yarn add --dev allure-tinybench allure-js-commons tinybench allure-vitest vitest

   pnpm

   pnpm add -D allure-tinybench allure-js-commons tinybench allure-vitest vitest

   Then configure the integration as described in its documentation, for example, Getting started with Allure Vitest.

5. In a test, run the benchmark with runWithAllure() instead of bench.run():

   import { runWithAllure } from "allure-tinybench";
   import { Bench } from "tinybench";
   import { test } from "vitest";
   
   test("parse benchmark", async () => {
     const bench = new Bench({ time: 100 });
   
     bench.add("json.parse.simple", () => JSON.parse('{"status":"passed"}'));
     bench.add("json.parse.nested", () => JSON.parse('{"a":{"b":{"c":[1,2,3]}}}'));
   
     await runWithAllure(bench);
   });

   By default, Tinybench runs each task for about a second, which can exceed the timeout of your test runner, for example, 5 seconds in Vitest. Set a shorter time for the benchmark, as in the example, or increase the test timeout.

2. Run tests

Run your tests the same way you would normally. For example, with Vitest:

npm

npx vitest run

yarn

yarn vitest run

pnpm

pnpm vitest run

For each benchmark task, a performance metric is saved to allure-results or whatever directory your integration is configured to use. The task name becomes the metric key, and the mean duration of the task, in milliseconds, becomes the value. See Allure Tinybench reference for details.

3. Generate a report

Finally, convert the test results into an HTML report with the allure generate command. It saves the report into the allure-report directory. To view the report, use the allure open command.

The benchmark results are shown in the Metrics section of the report, see View metrics in the report.

Configure the metrics

To give the metrics readable titles and units, and to group them, add the performance section to the Allure configuration file:

allurerc.mjs

import { defineConfig } from "allure";

export default defineConfig({
  performance: {
    groups: {
      json: { title: "JSON parsing" },
    },
    metrics: {
      "json.parse.simple": {
        title: "Parse a simple object",
        unit: "ms",
        better: "lower",
        group: "json",
      },
      "json.parse.nested": {
        title: "Parse a nested object",
        unit: "ms",
        better: "lower",
        group: "json",
      },
    },
  },
});

allurerc.json

{
  "performance": {
    "groups": {
      "json": { "title": "JSON parsing" }
    },
    "metrics": {
      "json.parse.simple": {
        "title": "Parse a simple object",
        "unit": "ms",
        "better": "lower",
        "group": "json"
      },
      "json.parse.nested": {
        "title": "Parse a nested object",
        "unit": "ms",
        "better": "lower",
        "group": "json"
      }
    }
  }
}

allurerc.yaml

performance:
  groups:
    json:
      title: JSON parsing
  metrics:
    json.parse.simple:
      title: Parse a simple object
      unit: ms
      better: lower
      group: json
    json.parse.nested:
      title: Parse a nested object
      unit: ms
      better: lower
      group: json

You can also track the metrics across runs and check them with quality gates, see Performance Metrics.