PHP Code Coverage Tools

Published On13 Oct 2020

PHP Code Coverage tools
Code Coverage tools help discover which parts of an application get executed during a test. The most common use case is to measure how much of the code base gets executed, and thus "covered" and "tested" during a test. While 100% of code coverage during a test does not mean the application will be bug-free, it is a measurable indication on how thorough the unit tests are.

Code Coverage reports can reveal how effective those tests are, and tools such as PHPUnit can generate meaningful reports using the code coverage data.

A screenshot of PHPUnit code coverage HTML report
PHPUnit has support various formats to report code coverage, including a simple CLI output option.

Code Coverage report only showing line coverage

Code Coverage tools are not perfect. Some tools can report specific lines that were executed, and some tools can go as far as showing which specific branches of code were executed.

Line Coverage Dead-Code Indication Path Coverage
Code Coverage report only showing line coverage Code Coverage report showing dead code Code Coverage report showing individual code paths

Generating Code Coverage Reports

PHPUnit, the popular PHP Unit Testing framework supports Xdebug, PCOV, and phpdbg for code coverage reports. sebastianbergmann/php-code-coverage is the package used by PHPUnit for its code coverage functionality. It's possible to use this library as an abstraction layer for various code coverage tools.

In PHPUnit 9.x series, Code Coverage can be enabled from the phpunit.xml(.dist) configuration file, with a setup similar to this:

<coverage>
    <include>
      <directory suffix=".php">src/</directory>
    </include>
    <report>
      <html outputDirectory="build/coverage"/>
      <text outputFile="build/coverage.txt"/>
    </report>
</coverage>

Using code coverage data from the tool used (such as PCOV or Xdebug), PHPUnit can generate code coverage reports in various formats that are human-readable and machine-readable for further processing.

If path coverage is desired, that can be specified from the XML configuration file as well:

- <coverage>
+ <coverage pathCoverage="true" cacheDirectory="build/phpunit/cache">

PHP Code Coverage Tools

PHP Code Coverage feature is often provided as part of a bigger tooling the PHP developers would use otherwise.

  • Xdebug, a PHP extension project started back in 2002 by Derick Rethans is the de-facto tool that helps in step debugging, profiling, and of code in code coverage.

  • PCOV, a recent PHP extension by Joe Watkins aims to be a minimal and fast solution for code coverage, which solves the often slow Xdebug version 2 series.

  • phpdbg is a PHP Debugger shipped with PHP itself. It has features similar to Xdebug, albeit relatively less adoption in community and IDEs. phpdbg is implemented as a server API, which limits its usefulness, but it provides an easy approach to code coverage without having to install PECL extensions.

Xdebug

Xdebug is a mature and actively developed PHP extension with a profiler, step debugger, and a code coverage tool. Due to the complex architecture and rich feature set, Xdebug is often slow, and using Xdebug often results in the tests to run considerably slow.

Code Coverage data from Xdebug provides information about dead code (code that will never be executed), and it also supports code path coverage, although it is significantly slower in practice. Xdebug is the most accurate and precise code coverage tool, as it even has custom handling when the Zend/Opcode does not provide enough information to infer accurate code coverage.

Xdebug version 3 is active development, and had its first beta release recently. For the benchmark and comparison purposes, Xdebug 2.9.8 and the latest code from master branch are used for the upcoming version 3.

Xdebug 3 splits the functionality to "modes". Unlike Xdebug 2 that had toggles for individual features (code coverage, step debugger, and profiler), Xdebug 3 simplifies the configuration and code with only one mode active at a time. Xdebug 3 in coverage mode is provides the same code coverage functionality, but it is significantly faster in Xdebug 3.

Code Coverage report only showing line coverage
Xdebug provides dead-code information. In code coverage reports, they can help discover snippets that have no code paths leading to them.

Xdebug also supports path coverage, which means it can detect execution of individual branches in a statement.

Code Coverage report only showing line coverage

Xdebug version 3

Xdebug 3 is the upcoming version of Xdebug. Xdebug 3 is not released as a stable version yet, but beta versions are available, and compiles just fine on PHP 8 that already has its API/ABI frozen.

Xdebug 3 introduces "modes". In contrast to various toggles for Xdebug features, it now only functions in different modes, be it step debugger, profiler, or a code coverage. This gives a significant performance improvement over the previous version.

PCOV

PCOV is a PECL extension designed for code coverage and code coverage alone. It does not support path coverage, nor dead-code analysis, but it is the fasted (benchmarks below) extension approach to PHP code coverage.

It is not possible to run Xdebug and PCOV at the same time, but PCOV provides an INI toggle to keep the extension loaded, but the functionality disabled.

PCOV has the advantage of being fast and light-weight because it is not a debugger. It is light-weight code base, without the sophisticated extra complexity Xdebug carries as Xdebug has custom functionality to provide more accurate results. PCOV code coverage data are comparable to Xdebug's and except for a few edge cases, the code coverage results would be the same.

phpdbg

phpdbg is a PHP Debugger shipped with PHP itself, since PHP 5.6. It is a feature-full debugger that is implemented as a Server API.

First developed by Joe Watkins, the author of PCOV, and then phpdbg was later merged to PHP core itself. It has code coverage support among its other features.

phpdbg does not offer path coverage. phpdbg being a separate SAPI, it is easy to invoke phpdbg without toggling any extensions.

Code Coverage report only showing line coverage
phpdbg is called with phpdbg in terminal, and it will run standard PHP CLI with phpdbg features (including code coverage) enabled.

PHP Code Coverage Benchmark

Using a code coverage tool can slow down the execution time of unit tests. This benchmark compares the relative execution time with each code coverage tool.

CommonMark from The PHP League supports PHPUnit 9.4, and contains more than 3,000 tests. Comparing the execution time of the test suite with each code coverage tool reveals the impact of each tool.

Comparison of execution times for PHP Code Coverage
The values are averages of five runs on PHP 7.4.11.

Despite the minor differences in how PCOV reports coverage information, PCOV is extremely fast. Xdebug 3 is significant improved compared to Xdebug 2. Note that Xdebug 3 is still in beta.

The same test was done on PHP 8 development branch, with PCOV and Xdebug 3 compiled against it.

Comparison of execution times for PHP Code Coverage - PHP 8

Path Coverage

Not all PHP Code Coverage tools support path coverage. Xdebug 2 and 3 support path coverage. PCOV and phpdbg, combined with PHPUnit do not support code coverage.

Running path coverage can significantly slow down the execution.

Comparison of execution times for PHP Path Coverage - PHP 7

Summary of PHP Code Coverage Tools

Xdebug 2

  • Debugger with many features; Slow.
  • Supports path coverage.
  • Mature and actively developed.

Xdebug 3

  • Fully-featured debugger, but with modes, code coverage is significantly faster than Xdebug 2.
  • Supports path coverage
  • Actively developed.
  • No stable releases yet.

PCOV

  • Dedicated code coverage tool, and is extremely fast.
  • Actively maintained.
  • Less granular code coverage in certain cases.
  • Does not support path coverage when used with PHPUnit.

phpdbg

  • Debugger with many features; Slow.
  • Implemented as a separate SAPI, rather than a PHP extension.
  • Less granular code coverage data.
  • Does not support path coverage.

Recent Articles on PHP.Watch

All Articles • Feed 
How to fix PHP Curl HTTPS Certificate Authority issues on Windows

How to fix PHP Curl HTTPS Certificate Authority issues on Windows

On Windows, HTTPS requests made with the Curl extension can fail because Curl has no root certificate list to validate the server certificates. This article discusses the secure and effective solutions, and highlights bad advice that can leave PHP applications insecure.
AEGIS Encryption with PHP Sodium Extension

AEGIS Encryption with PHP Sodium Extension

The Sodium extension in PHP 8.4 now supports AEGIS-128L and AEGIS256 Authenticated Encryption ciphers. They are significantly faster than AES-GCM and CHACHA20-POLY1305. This article benchmarks them and explains how to securely encrypt and decrypt data using AEGIS-128L and AEGIS256 on PHP.
How to Install/Upgrade PHP 8.3 on MacOS with Homebrew

How to Install/Upgrade PHP 8.3 on MacOS with Homebrew

Install PHP 8.3 and PHP extensions on MacOS with Homebrew.
Subscribe to PHP.Watch newsletter for monthly updates

You will receive an email on last Wednesday of every month and on major PHP releases with new articles related to PHP, upcoming changes, new features and what's changing in the language. No marketing emails, no selling of your contacts, no click-tracking, and one-click instant unsubscribe from any email you receive.