Faster Code Coverage

January 14, 2019

Murali Nandigama once wrote:

"Knowing what should be tested is beautiful, and knowing what is being tested is beautiful."

Test-Driven Development helps with the former and the Code Coverage software metric helps with the latter.

While it runs a software's test suite, PHPUnit can keep track of which parts of that software's source code are executed. In other words, we can measure to which degree the tests cover the source code of the software.

Code Coverage found its way into the PHP world back in 2003 when Derick Rethans started to add support for it to Xdebug. It would take until 2007, though, for this feature to make it into a stable release, Xdebug 2.0.

Xdebug's code coverage functionality was originally rather slow. For instance, back in 2006 it took hours to execute the test suite of the eZ Components project with code coverage enabled compared to minutes without code coverage. A slowdown in (test) execution speed when code coverage information is collected is normal and nothing specific to PHP. The massive slowdown incurred by Xdebug was not expected, though. And it rendered using code coverage with PHPUnit in everyday development unfeasible.

The release of Xdebug 2.0.1 improved code coverage performance dramatically: running the tests for the eZ Components project with code coverage now only took a couple of minutes more than running them without code coverage.

Xdebug 2.6

When you want to generate a code coverage report using PHPUnit then you need to configure a list of source code files that you want to see in the report. After all, you are not interested in the code coverage of the third-party librarries you use in your project, for instance.

By default, Xdebug collects coverage information for all code that is executed between a call to xdebug_start_code_coverage() and a call to xdebug_stop_code_coverage() . PHPUnit calls these functions at the beginning and at the end of a test's execution, respectively. As a result, Xdebug collects code coverage information for the test's code, the code being tested, PHPUnit's own code, third-party code used by PHPUnit, and third-party code used by the code being tested. PHPUnit then filters the collected code coverage information based on the configured list of files.

Of course, collecting and processing code coverage information you are not interested in only to filter it away – which in itself is yet another expensive operation – is a waste of resources. This is why Xdebug 2.6 introduced filter capabilities for code coverage and other features such as function tracing, for instance. The xdebug_set_filter() function can be used to prevent Xdebug from collecting and processing code coverage information for source code files you are not interested in. This saves time and memory.

Intermediate Solution

Holger Woltersdorf was one of the first to tweet about a bootstrap script for PHPUnit that could speed up test runs by using xdebug_set_filter() .

While this already showed the great potential of the native filtering provided by Xdebug, it could not realize its full potential as the source code of PHPUnit itself as well as that of its dependencies is already loaded when the test suite's bootstrap script is included. This is a problem because Xdebug can only apply its native filtering for source code files that are loaded after xdebug_set_filter() was called.

During FrOSCon 2018 in August, we developed a proof-of-concept implementation for generating a bootstrap script that configures Xdebug's native filtering based on PHPUnit's XML configuration file.

We liked how the proof-of-concept worked and decided to develop it further into a proper feature a couple of weeks later at the PHPUnit Code Sprint in Munich.

PHPUnit 7.4

During the PHPUnit Code Sprint in Munich in September, we developed the ideas from the intermediate solutions mentioned above into features offered by PHPUnit out-of-the-box.

First, we implemented a new command-line option, --dump-xdebug-filter , that instructs PHPUnit to load the phpunit.xml configuration file for a project and then dump the configured list of files to a PHP script. This script contains the PHP code that is required to configure Xdebug's native filtering mechanism.

This generated script needs to be loaded as early as possible to have any impact on the resource consumption related to code coverage data collection. Loading this script during the bootstrap of the test suite (configured through either the --bootstap command-line option or the bootstrap= configuration directive in phpunit.xml ) is too late.

We solved this problem with the implementation of another new command-line option, --prepend , that is interpreted by PHPUnit as early as possible.

Real-World Benchmark

Here is a real-world benchmark from the test suite for the code base of kartenmacherei.de:

Benchmark

It is interesting to note that the performance improvement depends on how PHPUnit is installed and used:

When PHPUnit is installed using Composer, running the tests and generating a code coverage report takes 40% less time when an Xdebug filter script is loaded using --prepend (75.96 seconds versus 126.29 seconds)
When PHPUnit is used from a PHP Archive (PHAR), running the tests and generating a code coverage report takes 60% less time when an Xdebug filter script is loaded using --prepend (33.72 seconds versus 86.29 seconds)
Running the tests and generating a code coverage report takes 55% less time when PHPUnit is used from a PHP Archive (PHAR) than when PHPUnit is installed using Composer (33.72 seconds versus 75.96 seconds)

We did not investigate this further but assume the differences are related to the fact that the phpunit.phar does not use class auto-loading for the source code files of PHPUnit itself as well as its dependencies.

Usage Example

By now you must be wondering: how do I use these new features to speed up the code coverage reporting for my test suite? Don't worry, we'll get to that now. The examples in this section assume that you have downloaded phpunit.phar to tools/phpunit . Replace ./tools/phpunit with ./vendor/bin/phpunit in the examples in case you use Composer to install PHPUnit.

Your phpunit.xml configuration file should look similar to the one shown below. For the purpose of this article, we're only interested in the <filter> section:

<?xml version = " 1.0 " encoding = " UTF-8 " ?>

< phpunit bootstrap = " vendor/autoload.php "

forceCoversAnnotation = " true "

beStrictAboutCoversAnnotation = " true "

beStrictAboutOutputDuringTests = " true "

beStrictAboutTodoAnnotatedTests = " true "

verbose = " true " >

< testsuites >

< testsuite name = " default " >

< directory suffix = " Test.php " > tests </ directory >

</ testsuite >

</ testsuites >

< filter >

< whitelist processUncoveredFilesFromWhitelist = " true " >

< directory suffix = " .php " > src </ directory >

</ whitelist >

</ filter >

</ phpunit >

By the way: the phpunit.xml file shown above configures PHPUnit using best practice defaults. You do not even have to copy and paste it: just run ./tools/phpunit --generate-configuration to launch a "wizard" that will ask you three simple questions before it generates a configuration file like that for your project.

Back to speeding up your code coverage report: the first step is to generate the filter script for Xdebug using the --dump-xdebug-filter option:

$ ./tools/phpunit --dump-xdebug-filter build/xdebug-filter.php
PHPUnit 7.5.1 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.3.1 with Xdebug 2.7.0beta1
Configuration: /workspace/project/phpunit.xml

Wrote Xdebug filter script to build/xdebug-filter.php

The generated filter script looks like this:

<?php declare ( strict_types = 1 ) ;

if ( ! \function_exists ( 'xdebug_set_filter' ) ) {

return ;

}

\xdebug_set_filter (

\XDEBUG_FILTER_CODE_COVERAGE ,

\XDEBUG_PATH_WHITELIST ,

[

'/workspace/project/src'

]

) ;

Now we can use the --prepend option to load the Xdebug filter script as early as possible when we want to generate a code coverage report:

$ ./tools/phpunit --prepend build/xdebug-filter.php \
                  --coverage-html build/coverage-report

What's next?

The performance of the collection and processing of code coverage data can be significantly improved when PHPUnit 7.4 (or later) and Xdebug 2.6 (or later) is used. For now, this requires PHPUnit to be invoked with --dump-xdebug-filter to generate a PHP script with configuration code for Xdebug any time the file inclusion list is changed in phpunit.xml . PHPUnit then needs to be invoked with --prepend to load that generated PHP script before any other code is loaded.

We are thinking about how we could improve the developer experience and make it easier to benefit from the performance improvements made to the collection and processing of code coverage data.

One approach could be changing PHPUnit to automatically dump the configured file list to a PHP script named .phpunit.xdebug.filter and load this file when it exists and Xdebug 2.6 (or later) is used as early as possible. This would allow us to first deprecate and then remove the --dump-xdebug-filter and --prepend command-line options.

Update (June 22, 2021): Since this article was written, the --dump-xdebug-filter and --prepend options discussed here were removed from PHPUnit because they were no longer needed thanks to improvements in php-code-coverage . Furthermore, most developers use the PCOV extensions these days to collect line coverage information and Xdebug 3 is a lot faster than Xdebug 2.