The story of code coverage for PHP starts with Xdebug 1.2 in April 2003, but actually it starts earlier, because Derick Rethans had to develop the feature first before it could find its way into a release. Back then, collecting code coverage data with Xdebug was very slow and the initial version of the report that PHPUnit generated from this data was very ugly.
From April to December 2006, I lived in Norway and wrote my thesis at eZ Systems, where Derick was also working at the time. We worked together on the Open Source project eZ Components, for which there were of course tests. Without code coverage analysis, we could run these tests in a few minutes. With code coverage analysis, it took half a day. Unfortunately, this was far too slow to be of any real practical use.
Xdebug 2
With the release of Xdebug 2 in 2007, code coverage analysis became significantly faster and therefore more useful. Since then, Derick has continued to improve the code coverage functionality. For example, Xdebug 2.6 introduced a filter mechanism that speeds up the code coverage analysis.
In a previous article , I already wrote about this filter mechanism, which incidentally has become superfluous due to other developments.
I presented the improvements we have made to PHPUnit to generate code coverage reports faster in another article .
PCOV
Of course, the performance of a functionality such as code coverage analysis could always be better: developers want to spend less time waiting for the tests to run and it is also better for the planet if the CPU consumes less power in the process.
This was also the motivation for Joe Watkins, who introduced PCOV in 2019 as an alternative for collecting code coverage data. Since then, PHPUnit has supported PCOV in addition to Xdebug.
Unfortunately, PCOV is currently not maintained. The last version was released in 2021, but can be used with current PHP versions. However, it cannot be compiled without a patch for the upcoming PHP 8.4.
In contrast to Xdebug, PCOV only supports line coverage. This software metric measures whether every executable line has been executed. Since version 2.3, Xdebug also supports path coverage. This measures whether every possible execution path in a function or method was executed during the execution of the test suite.
For everyday use, line coverage is completely sufficient for me. In recent years, I have used PCOV for this because it was significantly faster than Xdebug, especially compared to Xdebug 2.
I do not use path coverage regularly when I run my tests by hand during programming, nor in my CI pipelines. I only use it when I want to look at the execution paths of a specific method in detail. Therefore, in recent years I have only ever loaded Xdebug when I wanted to look at the path coverage.
Back to the question ...
As I was happy with PCOV, I didn't really give Xdebug 3, which was released at the end of 2020, a chance in daily use. I have now made up for this, inspired by the question mentioned at the beginning, which I have been asked more frequently recently.
Sebastian, which PHP extension do you use to collect code coverage data?
Now I can answer that question!
After five years of PCOV, I am returning to Xdebug. There are two reasons for this: the configuration setting xdebug.mode
introduced with Xdebug 3 and the currently uncertain future of PCOV.
Xdebug 3
How Xdebug works can be configured since version 3 via the setting xdebug.mode
or the environment variable XDEBUG_MODE
.
I have set xdebug.mode=off
in my PHP configuration file. This means that Xdebug is only loaded, but does nothing by default. There is no overhead just because Xdebug is loaded.
If I want to have a code coverage report, I invoke PHPUnit with XDEBUG_MODE=coverage ./tools/phpunit
instead of ./tools/phpunit
.
I advise against activating the code coverage functionality of Xdebug in the PHP configuration file with xdebug.mode=coverage
. The following benchmark will hopefully make it clear why.
I ran this benchmark with PHP 8.3.4, Xdebug 3.3.1, PCOV 1.0.11 and PHPUnit 11.0.7 on an Apple MacBook Air M3 (2024) using the test suite of my raytracer project.
If neither Xdebug nor PCOV are loaded, the tests are run in 13 seconds. If Xdebug is loaded but deactivated with xdebug.mode=off
, there is no measurable overhead.
xdebug.mode=debug
activates the step debugger of Xdebug. In this mode, the tests are run in 33 seconds. The setting xdebug.mode=develop
activates Xdebug's Development Helpers. In this mode, the tests are run within 35 seconds.
xdebug.mode=coverage
activates the code coverage functionality of Xdebug. In this mode, the tests are run in 50 seconds. It does not seem to matter whether PHPUnit instructs Xdebug to collect code coverage data or not. If PHPUnit requests the collection of path coverage data, running the tests takes 127 seconds.
With PCOV, the tests are run in 15 seconds. Again, it does not seem to matter whether PHPUnit instructs PCOV to collect code coverage data or not.
Generating a code coverage report with line coverage is still faster with PCOV (15 seconds) than with Xdebug 3 (50 seconds). However, Xdebug is being actively developed by Derick and often provides better data than PCOV, especially in edge cases.