Merging your Playwright sharded reports into a single HTML report

profile
Tim Deschryver
timdeschryver.dev

In a previous blog post, "Using Playwright test shards in combination with a job matrix to improve your CI speed" we've seen how to split our Playwright test suite into multiple shards to speed-up the build pipeline.

The setup presented in that blog post created multiple test reports, one for each shard. The only downside with this, is that we didn't have an overview of all the tests, which made it a little bit harder to find a test report that you were looking for. For example, when a test failed you first needed to know the shard to open the corresponding test report.

Playwright v1.37.0 changes that by providing the ability to merge the different test reports into one.

The result is a beatiful looking pipeline, with one single test report as result thanks to the new merge-reports command.

The GitHub Workflow visualization of the pipeline has an additional job to merge the individual test reports for each test shard.

Upgrade Playwright to >= v1.37 link

The first step is upgrading Playwright to at least v1.37.0.

Changing the test reporter link

Next, the reporter needs to be changed. The reporter currently outputs the test results into an HTML file, but these cannot be combined into a single report. Instead of exporting the test results into an HTML report, we want it to be a file format that can later be processed and merged.

To update the reporter, modify the reporter within the playwright config to create a blob instead of an html file. With this change the output directory containing the test result files is also different, and is changed from playwright-report to blob-report.

This results in a zip file containing a JSON file with the test output.

The change above only makes use of the blob format in a continuous integration environment (using the environment variable process.env.CI), in other words within your build pipeline. You can also enable this for your local environment if that's desired, e.g. when you're also making use of test sharding when running your test suite locally.

Creating a test report from the blob files link

To create a HTML test report the last step is to merge the different blob files. Luckily we don't need to do this manually, but we can use the new merge-reports command.

Pass it the location where your blob files are located.

Updating the CI pipeline (GitHub Actions) link

Now that we got that working, it's time to update our pipeline and make the above changes to it.

Beause the output of the test results is changed to blob-report, we need to update the artifact upload step. A side note: while updating the step, you can also lower the retention time because these files will be replaced by the single output file that will be created next.

This change still results in multiple output files. To merge the files we use the merge-reports command in an additional job. This job downloads all the blob reports created by the sharded jobs, which all upload their individual reports as an artifact. This needs to be an extra job because the test shards are run in parallel, each in its own isolated environment.

The last step within the create-report job above uploads the bundled report as a pipeline artifact.

Result link

These changes result in the following pipeline:

The GitHub Workflow visualization of the pipeline has an additional job to merge the individual test reports for each test shard.

The complete GitHub workflow file looks as follows:

You can also take a look at this commit for the changes that we've covered in this blog post.

Outgoing links

Feel free to update this blog post on GitHub, thanks in advance!

Join My Newsletter (WIP)

Join my weekly newsletter to receive my latest blog posts and bits, directly in your inbox.

Support me

I appreciate it if you would support me if have you enjoyed this post and found it useful, thank you in advance.

Buy Me a Coffee at ko-fi.com PayPal logo

Share this post

Twitter LinkedIn