thePHP.cc Logo Deutsch Contact
Help! My tests stopped working

Help! My tests stopped working

Running tests at the touch of a button is a great thing. Updating the test tool at the push of a button also sounds promising - or maybe not?

Following the release of PHPUnit 8, some developers tweeted about changes that they hate in this new version. To a certain extent, I understand these problems and I can certainly relate to the frustration these developers experience. This article provides guidance on how to avoid such frustration in the future.

PHPUnit 8 was released on February 1, 2019. As laid out in its release process documentation , a new major version of PHPUnit is released each year on the first Friday of February.

A new major provides an annual opportunity for cleanup. Most new features will be implemented in minor versions, PHPUnit 8.1 through PHPUnit 8.5, that are due on the first Friday of April, June, August, October, and December, respectively.

PHP 7.2 Requirement

PHPUnit 8 requires PHP 7.2 (or newer). This is because active support for PHP 7.1 by the PHP project ended on December 1, 2018. As of today, the only actively supported versions of PHP are PHP 7.2 and PHP 7.3. PHPUnit 8 is supported on PHP 7.2 and PHP 7.3, it will also be supported on PHP 7.4.

If you are still on PHP 7.1 then you should start to migrate your software to a supported PHP version, ideally PHP 7.3, as security support for PHP 7.1 by the PHP project will end on December 1, 2019. As a long-term goal, you must make PHP upgrades a part of your normal operational procedure and align the upgrade cycle of your PHP stack with the release cycle of the PHP project .

If you cannot use PHP 7.2 yet then, of course, you can also not use PHPUnit 8 right away. As PHPUnit 7 is supported until February 7, 2020 , there is no immediate problem. You can stay with PHP 7.1 and PHPUnit 7 for now but, of course, you will miss out on any improvements made in PHP 7.3 and PHPUnit 8.

Return Type of Template Methods

The template methods of PHPUnit\Framework\TestCase , setUpBeforeClass() , setUp() , assertPreConditions() , assertPostConditions() , tearDown() , tearDownAfterClass() , and onNotSuccessfulTest() , now have a void return type declaration. Implementations of these methods now must be declared void , too, otherwise you will get a compiler error:

<?php  declare ( strict_types = 1 ) ;
use   PHPUnit\Framework\TestCase ;
 
final   class   MyTest   extends   TestCase
{
     protected   function   setUp ( )
     {
     }
 
     // ...
}

Trying to run the tests of the MyTest test case class shown above with PHPUnit 8 will result in the compiler error shown below:

$ ./tools/phpunit MyTest PHP Fatal error: Declaration of MyTest::setUp() must be compatible with PHPUnit\Framework\TestCase::setUp(): void in ...

All other methods of PHPUnit that do not return a value have a void return type declaration since PHPUnit 7. Because of the aforementioned compiler error, the void return type declaration was not added to the template methods of PHPUnit\Framework\TestCase in PHPUnit 7 as this would have resulted in a break of backward compatibility without giving developers enough time for preparation. Instead, the release announcement for PHPUnit 7 contained the information that this change would happen a year later in PHPUnit 8. Additionally, the following advice was given:

Please declare your methods that overwrite the [template methods] void now so you are not affected by this backward compatibility break.

The void return type is used to express that a method does not return anything. If the compiler sees a method that has a void return type declaration and the method returns a value then the compiler raises an error:

<?php  declare ( strict_types = 1 ) ;
final   class   Example
{
     public   function   doSomething ( ) :   void
     {
         return   false ;
     }
}

Trying to execute the code shown above will lead to the compiler error shown below:

$ php Example.php PHP Fatal error: A void function must not return a value in ...

This is useful as it can make programming mistakes obvious.

Before they had a void return type declaration, the template methods of PHPUnit\Framework\TestCase could be implemented in such a way that they returned a value. PHPUnit never used that value. Over the years, though, I have seen implementations of setUp() , for instance, that had a return value. Where this was the case, the setUp() method was not only invoked automatically by PHPUnit before the execution of each test method of the test case class in question but also manually through code in test methods.

Thanks to the void return type declaration, the fact that PHPUnit does not expect a return value from a template method is now made explicit in code. This information is used by PHP's compiler and runtime as well as static analysis tools. Either way, this type information is vital for finding problems as early as possible.

You may disagree with my opinion that more type information leads to code that is easier to understand, by both humans and static analysis tools alike. I will not dictate how you should write your code and neither will PHPUnit. It is only at the interface between PHPUnit's code and your code, for instance when you extend a class such as TestCase and implement a template method such as setUp() , that you must follow PHPUnit's lead. Actually, most third-party software that you use in your projects encourage best practices. Frameworks do that, as do tools such as Composer or PHPUnit.

I believe that there is a lot of value in leveraging the improvements made to the type system in PHP 7. This is why I spent a lot of time and effort on modernizing PHPUnit's codebase to make better use of strict interpretation of scalar type declarations and return type declarations. Explicit parameter and return types also makes that code a lot easier to understand for everyone who contributes to PHPUnit. Thanks to more type information quite a few coding mistakes can be prevented, or at least become a lot easier to debug and fix.

Deprecations

Software keeps growing over time, especially when the maintainers keep adding new features without ever removing code. It is not prudent, especially in an Open Source project where all work is done by volunteers, to waste valuable development time on maintaining features that are no longer needed, rarely used, or problematic. This is why I regularly remove old functionality from PHPUnit. I believe that your project should do the same, but, of course, this is entirely up to you.

There is a chance that you will see deprecation warnings when you run your tests with PHPUnit 8:

$ ./tools/phpunit MyTest PHPUnit 8.0.2 by Sebastian Bergmann and contributors. W 1 / 1 (100%) Time: 33 ms, Memory: 4.00MB There was 1 warning: 1) MyTest::testSomething assertInternalType() is deprecated and will be removed in PHPUnit 9. Refactor your test to use assertIsArray(), assertIsBool(), assertIsFloat(), assertIsInt(), assertIsNumeric(), assertIsObject(), assertIsResource(), assertIsString(), assertIsScalar(), assertIsCallable(), or assertIsIterable() instead. WARNINGS! Tests: 1, Assertions: 1, Warnings: 1.

In case you see deprecation warnings such as the one shown above: do not panic.

Deprecated assertions continue to work. A warning is only generated for those tests that use a deprecated assertion but would otherwise be reported as successful. This means that a deprecation warning does not hide information about a test error or failure.

PHPUnit behaves differently compared to PHP and how it reports the usage of deprecated functionality. When a deprecated feature of PHP is used then a message of type E_DEPRECATED is created. This message, however, is only shown when the developer has configured PHP's error_reporting setting accordingly. In my experience, developers either do not have error_reporting configured correctly (it should be set to -1 for development so that you see all error messages) or never bother to look at error messages that get buried in some log file. This is why quite a few developers are surprised that their software stops working when a major version of PHP is released that removes previously deprecated functionality.

PHPUnit reports a test that uses deprecated functionality with a warning because I know how developers use, or rather not use, PHP's E_DEPRECATED messages. You cannot opt out of getting this information from PHPUnit.

By default, PHPUnit's command-line test runner exits with shell exit code 0 when the use of a deprecated feature is reported. This shell exit code is used to indicate that no error occurred. This information is used by continuous integration environments, for instance, to decide whether or not the build was successful. If you want your build to fail because the tests use deprecated functionality from PHPUnit, configure failOnWarning="true" in phpunit.xml . This instructs PHPUnit to exit with shell exit code 1 when deprecated assertions are used.

assertEquals()

Without a doubt, assertEquals() is the assertion method that is used the most. Over the years, many optional parameters were added to its API. Some of these cannot even be used together which is a constant cause for edge-case bugs in the underlying implementation. The main problem with long lists of optional parameters is that you have to specify the first four if you want to use the fifth, for example. And honestly, who can remember what these parameters are for? At least I cannot.

To remedy this, specialized alternatives to assertEquals() were introduced in PHPUnit 7.5. This is why in PHPUnit 8, the following optional parameters of assertEquals() are deprecated:

These parameters will be removed in PHPUnit 9.

The problems with assertEquals() and its optional parameters are similar to the situation we had with getMock() in the past. createMock() , createPartialMock() , etc. were created to replace the confusing API of getMock() with clean, explicit, and separate methods.

For the sake of completeness it should be mentioned that everything I said above also applies to assertNotEquals() which is the inverse of assertEquals() .

assertContains()

Over the years, optional parameters were also added to the assertContains() method. Furthermore, its scope was expanded from just working on arrays and objects that implement the Iterator interface to also work on strings. Time and time again this lead to problems because different, sometimes even conflicting, use cases were implemented in the same unit of code.

To fix these problems, specialized alternatives that only operate on strings were introduced in PHPUnit 7.5. In PHPUnit 8, the optional parameters $checkForObjectIdentity , $checkForNonObjectIdentity , and $ignoreCase are now deprecated. Furthermore, using assertContains() with string haystacks is also deprecated. Tests that use assertContains() with string haystacks should be refactored to use assertStringContainsString() instead.

These parameters as well as the ability to use assertContains() with string haystacks will be removed in PHPUnit 9.

While cleaning up assertContains() , I made a mistake which made it impossible to assert that an object is contained in an iterable while using == instead of === . This mistake is corrected through the introduction of assertContainsEquals() and assertNotContainsEquals() in PHPUnit 8.0.2. Thanks to Rathes Sachchithananthan for bringing this to my attention.

All this also applies to assertNotContains() which is the inverse of assertContains() .

assertInternalType()

The assertInternalType() method can be used to assert that a variable contains a value of a specific type that is not user-defined. Here is an example that shows asserting that a variable contains a value of type array :

$this->assertInternalType('array', $variable);

A significant aspect of this assertion's intent is hidden in a parameter, 'array' . This is problematic as an IDE, for instance, cannot help with autocomplete on the word array . Furthermore, no real safeguard against a typo in 'array' is possible as it is, after all, just a string.

As you may have guessed, to address this problem, specialized alternatives to assertInternalType() were introduced in PHPUnit 7.5. These new methods do not hide important information in a parameter but make explicit what they are about in their name:

$this->assertIsArray($variable);

A more explicit API means that there are more methods but each method is simpler as it only has to implement one specific use case. This leads to code that is easier to read and understand. It is also easier to write because the IDE can autocomplete now for the entire intent and not just a part of it.

Everything above also applies to assertNotInternalType() which is the inverse of assertInternalType() .

assertArraySubset()

The assertArraySubset() method has been a constant source of confusion and frustration as can be seen here , here , here , here , here , here , or here .

The reason for this situation is that I merged the original pull request after only a cursory review. I did not have a use case for it but its implementation looked, at least at first glance, like it would serve the use case of the developer who proposed this new functionality.

Over the years, changes that I perceived to be bugfixes for edge cases lead to behavior that was confusing because, again, conflicting use cases were handled. Removing support for any of these new use cases now would constitute a break of backward compatibility.

While working towards PHPUnit 8, I came to the conclusion that assertArraySubset() cannot be fixed, at least not without breaking backward compatibility. This is why I decided to deprecate assertArraySubset() now and to remove it from PHPUnit 9 next year. This results in a break of backward compatibility that is explicit and obvious. Changing how assertArraySubset() works yet again would have been more problematic.

Anybody who thinks that this functionality is useful is more than welcome to take the code, put it into a separate project, and package it as an extension for PHPUnit.

Assertions and Non-Public Properties

Objects that have problematic dependencies and are too large in general are common in legacy code. Objects like that may in fact require indirect testing through the inspection of non-public state even though this has never been a suggested best practice. PHPUnit provides quite a few assertions that operate on non-public properties such as assertAttributeEquals() . These assertions were originally implemented to make testing of legacy code easier.

Unfortunately, these assertions were often abused to test new code. The mistake made by developers is wanting to assert the contents of a private or protected property. Rather than going down this road, they should focus more on the behavior of objects rather than asserting their state.

It turns out that providing functionality such as assertAttributeEquals() out-of-the-box encourages bad testing practices. This is why I decided to deprecate all assertions as well as utility methods that operate on non-public properties in PHPUnit 8. In PHPUnit 9 they will be removed.

Remember: methods that are not public must not be tested directly by invoking them through the Reflection API. The private state of an object must also not be asserted in a test. Doing so creates a tight coupling between your test code and the production code you're testing. You are testing implementation, not API. As soon as the implementation changes, though, the test will break because it has relied on private implementation details.

Expecting Exceptions

Almost to the day three years ago, I wrote about best practices for expecting exceptions with PHPUnit . Back then I suggested to use $this->expectException(MyException::class); inside a test method over using an @expectedException MyException annotation on the test method. The reasoning behind that suggestion is explained in detail in the referenced article. Here is the rundown:

On the one hand, PHP does not have native support for annotations. Instead, annotations are expressed inside code comments. In a code comment you cannot use an unqualified class name, Example for instance instead of vendor\project\Example , that you would be able to use in code when that class is in or imported into the current namespace.

On the other hand, when the @expectedException annotation is used then PHPUnit will consider the test successful if the specified exception is raised at any point in time during the execution of the test method. This may not always be what you want.

I think today that it does more harm than good to support multiple ways of expecting exceptions in PHPUnit. Using @expectedException and related annotations is deprecated in PHPUnit 8 and support for these annotations will be removed in PHPUnit 9.

Upgrading to PHPUnit 8

PHPUnit follows semantic versioning . In a nutshell, this means that you can rely on the following three simple rules for interpreting PHPUnit's version numbers:

While we are on the topic of semantic versioning, you may be interested in an article I wrote on how to correctly depend on PHPUnit when you use Composer to manage your project's dependencies.

A tool such as PHPUnit must not be upgraded haphazardly from one major version to another. After all, you also do not upgrade to a new major version of PHP, without reading up on the changes made to the language no less, and expect your code to continue to work without adapting it to the new version. Or do you? The same applies to the framework your application is built upon and the libraries you use.

When I hear complaints about tests no longer working or requiring changes only one business day after a major version such as PHPUnit 8 has been released then I can only assume that developers made the upgrade in a way that was completely unplanned.

You should not upgrade to a new major version of PHPUnit without reading up on the breaking changes it introduces. You should also not start migrating your project to a new version of your test framework when any of your tests currently fail.

Run your tests, only locally at first and not in continuous integration, with the new PHPUnit version. Take note of warnings, especially with regards to deprecated functionality, and errors. Then read up on the relevant changes in the release notes and learn how you need to adapt your code to the new version. Make the required changes exemplarily for one or two tests of each error category. You should then be able to project how much time is required to make all necessary changes.

A lot of these changes can be automated. php-cs-fixer is the go-to tool for quickly and reliably making changes to an entire code base.

For instance, php-cs-fixer can automatically add a void return type declaration to all functions and methods of a code base that do not have a return statement in their body. Let us assume we have the following code in a tests/MyTest.php source file:

<?php  declare ( strict_types = 1 ) ;
use   PHPUnit\Framework\TestCase ;
 
class   MyTest   extends   TestCase
{
     protected   function   setUp ( )
     {
     }
}

We can use php-cs-fixer to add the missing void return type declaration:

$ php-cs-fixer fix --allow-risky=yes --rules void_return tests Loaded config default. Using cache file ".php_cs.cache". 1) tests/MyTest.php Fixed all files in 0.003 seconds, 10.000 MB memory used

tests/MyTest.php now looks like this:

<?php  declare ( strict_types = 1 ) ;
use   PHPUnit\Framework\TestCase ;
 
class   MyTest   extends   TestCase
{
     protected   function   setUp ( ) :   void
     {
     }
}

php-cs-fixer will eventually be able to automatically migrate code such as assertInternalType('array', $variable) to assertIsArray($variable) , for instance. The work on supporting developers that migrate their tests from PHPUnit 7 to PHPUnit 8 is tracked here , here , and here .

Rector is another tool that is worth a look. It supports many "instant upgrade" refactorings such as changing assertEquals() calls that use optional method parameters or assertInternalType() calls to the respective alternatives.

Conclusion

The frustration experienced by the developers that expressed their dislike for PHPUnit 8 is not rooted in technical changes of that new version but rather in the lack of a proper process for upgrading the dependencies in a project.

The fact that developers need to add a void return type declaration to template methods such as setUp() in their test case classes was announced a year ago when PHPUnit 7 was released. The necessary changes could have been made at any point in time between the release of PHPUnit 7 and the release of PHPUnit 8. This would not have caused a problem and it would not have cost a lot of time. Especially not when a tool such as php-cs-fixer would have been used to automatically add the void return type declarations.

The release announcement for PHPUnit 8 contains similar information for PHPUnit 9 which is due in February 2020. However, I sometimes get the impression that nobody reads announcements like these.

The maintenance of a software project must not only be reactive. It must also be pro-active in the sense that developers need to be aware of changes that they need to make in next twelve months to avoid unpleasant surprises when a new major version of the programming language or a framework, library, or tool that they use is released.

At the very least this means carefully reading release announcements of new major versions and to schedule making the changes required for an upcoming sprint. Ideally, though, the developers follow the development of important third-party software packages their project depends on. In the case of PHPUnit this would mean to look at the milestone for the next major version or trying out a nightly build (available at https://phar.phpunit.de/phpunit-nightly.phar ) every once in a while. This gives them not only the opportunity to learn about upcoming changes early but also to provide feedback to the upstream project on these changes.

We can help you

Existing systems primarily age because technology evolves. In the long run, the safe, secure, and smooth operation of software is only possible on up-to-date systems. We can support you in finding a strategy and process for upgrading PHP as well as the framework, libraries, and tools you depend on.

You can find more information on how we can help you in dealing with your legacy software as well as operating your PHP applications on our website.