In this article I would like to talk about how a change made in PHP 7.4 suddenly impacted the way PHPUnit's PHAR works due to a change that was recently made in PHP 8.1 .
This recent change added more type information for built-in types such as Iterator
and IteratorAggregate
. Here is a selection of issues that show how this change sent a wave of ripples through the PHP ecosystem:
- composer/composer#10008
- humbug/php-scoper#530
- symfony/symfony#42199
- symfony/symfony#42231
- justinrainbow/json-schema#668
- sebastianbergmann/php-file-iterator@1d83c072
- sebastianbergmann/php-token-stream@9c1da832
I am happy to see more type information being added to built-in types and gladly perform the chores of making my software compatible with PHP 8.1. The improved type safety is well worth the effort.
I was confused, though, by the fact that these changes caused the PHPUnit PHAR to stop working :
Fatal error: Could not check compatibility between
PharIo\Manifest\AuthorCollection::getIterator(): PharIo\Manifest\AuthorCollectionIterator
and IteratorAggregate::getIterator(): Traversable, because class
PharIo\Manifest\AuthorCollectionIterator is not available in ...
I had already fixed the PHP 8.1 compatibility issue
in the phar-io/manifest
package. Why was this coming up again?
Then I read the error message again and realized that this was a different issue entirely: PHP's compiler could not see the class AuthorCollectionIterator
while compiling the sourcecode file that declares the AuthorCollection
class.
Type relationships can be tricky
When a class implements an interface, then for each method that is declared by that interface and implemented in this class the compatibility of that method's signature has to be checked. This also applies to methods that are overwritten in a class that extends another class.
A method from an interface that is implemented in a class (or overwritten in a class that extends another class) may be contravariant and accept a more general parameter as well as covariant and return a more specific type :
| class AuthorCollection implements IteratorAggregate |
| { |
| // ... |
| public function getIterator ( ) : AuthorCollectionIterator |
| { |
| return new AuthorCollectionIterator ( $this ) ; |
| } |
| } |
| class AuthorCollectionIterator implements Iterator |
| { |
| public function __construct ( AuthorCollection $authors ) |
| { |
| $this -> authors = $authors -> getAuthors ( ) ; |
| } |
| // ... |
| } |
The PHPUnit PHAR stopped working because parameter type contravariance and return type covariance are only fully supported when all interfaces and classes involved are available before they are referenced.
The compiler works on AuthorCollection.php
, sees that the class AuthorCollection
implements the interface IteratorAggregate
, sees that AuthorCollection::getIterator()
returns AuthorCollectionIterator
, wants to check whether AuthorCollection::getIterator()
is compatible with IteratorAggregate::getIterator()
, but cannot perform this method compatibility check because AuthorCollectionIterator
has not been compiled yet.
At this point you probably wonder: why does the PHP compiler not interupt the compilation of AuthorCollection.php
and uses autoloading to trigger the compilation of AuthorCollectionIterator
? Well ...
Under the hood of PHPUnit's PHAR
To avoid problems that occur when the code under test shares dependencies with PHPUnit but requires different versions than the ones bundled in the PHAR, a couple of measures have been implemented.
Most units of code bundled in PHPUnit’s PHAR distribution, including all dependencies such as vendor directories, are moved to a new and distinct namespace
, for instance. Classes such as PHPUnit\Framework\TestCase
that are part of PHPUnit’s public API are exempt from this.
Furthermore, PHPUnit’s PHAR does not use dynamic autoloading to load the bundled units of code. Instead, all units of code bundled in the PHAR are loaded on startup. Until recently, this was implemented using a long list of require
statements:
| // ... |
| require 'phar://phpunit.phar' . '/AuthorCollection.php' ; |
| require 'phar://phpunit.phar' . '/AuthorCollectionIterator.php' ; |
| // ... |
The paths in the code shown above have been shortened for legibility.
The require
statements were generated using static analysis and a topological sort of implements
, extends
, and uses
relationships between the interfaces, classes, and traits.
This worked fine as long as the sourcecode of PHPUnit and its dependencies only contained trivial sub-type relationships between code units. This changed when PHP 8.1 added return types for Iterator
and IteratorAggregate
.
Problems wherever you look
My first idea was to use dynamic autoloading instead of static preloading and to error out when the PHAR is used and PHPUnit is also installed using Composer . The latter is necessary because Composer prepends the autoloader it generates to the front of the autoloader chain. This leads to conflicts that can only be detected by PHPUnit, but cannot be resolved by PHPUnit.
However, this turned out to cause new problems. Here is what Juliette Reinders Folmer commented :
Most projects I work on have PHPUnit in their Composer require-dev, but as a large part of my work is focused on PHP cross-version compatibility, I run tests locally using a Phar for whichever PHP version I need to test at that time.
One way to work around this could have been to configure Composer to not register its autoloader with a higher priority than that of the autoloader registered by PHPUnit's PHAR
. Unfortunately, though, this has to be done in the composer.json
file which, in most cases, is not under Juliette's control so this approach does not help her.
Another way could have been to register PHPUnit's autoloader again after the configured bootstrap script has been loaded. This did not work as expected, though.
I was not satisfied with this situation. On the one hand, I could not bring myself to revert the change to error out when the PHAR is used and PHPUnit is also installed using Composer . On the other hand, I felt like I was letting Juliette alone with the problem that this change caused for her.
Making things better, together
In a conversation with Arne Blankerts , I explained:
Due to this change in PHP 8.1, we can no longer use static preloading. We have to use dynamic autoloading. But this causes other problems ...
It was then and there that Arne interrupted me and said:
Hold on! You are forgetting that you can, and in this case should, combine static preloading and dynamic autoloading.
This was a great idea and rather easy to implement .
Instead of replacing static preloading with dynamic autoloading, we now combine the best of both worlds. The PHAR still loads all sourcecode files bundled in it on startup, but is assisted by a dynamic autoloader so that PHP's compiler can find code units that have not been loaded yet for performing method compatibility checks, for instance.
I am thankful to Juliette for being so persistent in spuring me on to find a better solution. And, of course, I am thankful to Arne for having the right idea at the right time. To me, this is what Open Source is all about: working together to make our software better for everyone involved.
How does this affect you?
You may be wondering: how does this affect me? Surely, my software does not have to work around such weird edge cases.
Well, I do not consider this code to be an edge case:
| class A |
| { |
| public function method ( ) : B |
| { |
| } |
| } |
| class B extends A |
| { |
| public function method ( ) : C |
| { |
| } |
| } |
| class C extends B |
| { |
| } |
In his answer to a question of mine with regards to static preloading and method compatibility checks, PHP core developer Nikita Popov provided the example shown above . He explained:
There is no legal ordering of these classes that does not involve autoloading during inheritance.
I would not blame you if you get the impression that the technical details discussed in this article are not relevant to your work. But they are relevant and important if you want to use the preloading feature that was introduced in PHP 7.4 . Go ahead, follow that link, and read what we wrote in our eBook PHP 7 Explained about improving your application's performance with preloading.
What I wrote back in 2016 when the active support for PHP 5 ended still holds true:
Existing systems primarily age because technology evolves. In the long run, the safe, secure, and smooth operation of software is only possible on up-to-date systems. [...] Upgrading the version of PHP you use must not be a rare event you are afraid of. You must not think of upgrading your PHP stack as a "special project". You need to make upgrading the PHP version you use part of your normal operational procedure and align the upgrade cycle of your PHP stack with the release cycle of the PHP project.
The development of PHP 8.1 is now focused on stabilization. Now is the the time to test your software against the beta versions and release candidates of PHP 8.1. There is a good chance that this will be rather boring: your software just continues to work with PHP 8.1. Great! But if your software requires changes to work with PHP 8.1, the sooner you know, the sooner you can adapt.