In a previous blog post, "Using Playwright test shards in combination with a job matrix to improve your CI speed" we've seen how to split our Playwright test suite into multiple shards to speed-up the build pipeline.
The setup presented in that blog post created multiple test reports, one for each shard. The only downside with this, is that we didn't have an overview of all the tests, which made it a little bit harder to find a test report that you were looking for. For example, when a test failed you first needed to know the shard to open the corresponding test report.
Playwright v1.37.0 changes that by providing the ability to merge the different test reports into one.
The result is a beatiful looking pipeline, with one single test report as result thanks to the new
The first step is upgrading Playwright to at least v1.37.0.
Next, the reporter needs to be changed. The reporter currently outputs the test results into an HTML file, but these cannot be combined into a single report. Instead of exporting the test results into an HTML report, we want it to be a file format that can later be processed and merged.
To update the reporter, modify the reporter within the playwright config to create a
blob instead of an
With this change the output directory containing the test result files is also different, and is changed from
This results in a zip file containing a JSON file with the test output.
The change above only makes use of the blob format in a continuous integration environment (using the environment variable
process.env.CI), in other words within your build pipeline.
You can also enable this for your local environment if that's desired, e.g. when you're also making use of test sharding when running your test suite locally.
To create a HTML test report the last step is to merge the different blob files.
Luckily we don't need to do this manually, but we can use the new
Pass it the location where your blob files are located.
Now that we got that working, it's time to update our pipeline and make the above changes to it.
Beause the output of the test results is changed to
blob-report, we need to update the artifact upload step.
A side note: while updating the step, you can also lower the retention time because these files will be replaced by the single output file that will be created next.
This change still results in multiple output files.
To merge the files we use the
merge-reports command in an additional job.
This job downloads all the blob reports created by the sharded jobs, which all upload their individual reports as an artifact.
This needs to be an extra job because the test shards are run in parallel, each in its own isolated environment.
The last step within the
create-report job above uploads the bundled report as a pipeline artifact.
These changes result in the following pipeline:
The complete GitHub workflow file looks as follows:
You can also take a look at this commit for the changes that we've covered in this blog post.
Feel free to update this blog post on GitHub, thanks in advance!
I appreciate it if you would support me if have you enjoyed this post and found it useful, thank you in advance.