Acceptance Testing WordPress Plugins

In my previous post on acceptance testing, I demonstrated how I’ve been using Codeception to build a bank of acceptance tests for Gravity Flow to check for regression issues and ensure reliability between every release.

In this second post, I make the case for a deeper approach to acceptance testing and demonstrate how to create an easy-to-use, self-contained environment for running acceptance tests for any WordPress plugin that will run anywhere. I also offer some tips for writing tests and some suggestions for integrating the tests into the wider development process.

Introduction

Ever since their initial release, both Gravity Flow (2015) and Gravity Forms (2009) have had the same commitment to backwards compatibility as WordPress – it’s key to our success – and the longer that success continues, the greater the challenge becomes for us to guarantee stability and reliability for our users. It’s our biggest challenge right now.

With the looming prospect of a significant shift in technology and an ever increasing risk of regression issues, it’s vital that we find ways to mitigate that risk. The trust of our users is at stake. One of the ways we can do this is by making automated acceptance testing a more integral part of our plugin development process.

We’ve been hearing “Learn JavaScript deeply” for a long time, but perhaps we ought to be including acceptance testing in that mantra. “Learn acceptance testing deeply and then worry about rewriting everything in JavaScript”. OK, perhaps it doesn’t have the same ring to it.

After working with Codeception and understanding it better over the last year or so, I’m confident it provides the tools we need. The fact we can write the tests in PHP and use WordPress functions is helpful, and now PHPStorm has built-in Codeception support. It’s also encouraging to see it gaining traction both inside and outside the WordPress community.

The tests themselves are easy enough to write. However, the environment is a bit fiddly to set up and with so many parameters to get right, it can be a bit daunting and frustrating at first.

So, in an attempt to make acceptance testing more accessible, I set out to find a way to make the setup a bit easier.

The goals were:

1. Easy setup – as easy as writing a Hello World plugin.
2. Zero-configuration – so easy that everyone on the team can download the setup and start writing tests without having to customise any settings.
3. Cross-platform so all developers on the team can use the same setup regardless of their operating system.
4. Support for continuous integration services without modifications.

Acceptance Testing with Docker

My first attempt was a bash script which downloaded, installed and started PhantomJS and used WP-CLI to set up a PHP server. It worked well locally, but the tests would hang intermittently on CI services such as Travis CI and Codeship. Frustratingly, I never really got to the bottom of it, even with the help of their support, but I suspected an issue with PhantomJS running inside containers. I was never really that keen on using a headless browser anyway so I decided to take a different approach – one that would be more portable and less dependent on the local environment.

The solution I came up with for Gravity Flow and Gravity Forms uses Docker to set up a private network of services for the duration of the tests. The setup handles the following:

• Creates containers for each of the services:
  • WordPress(Latest) + Apache + PHP5.6 (official image)
  • MariaDB database (official image)
  • Codeception + the WPBrowser module (based on the official Codeception image)
  • Selenium standalone with Chrome plus a VNC server (official image)
• Downloads and install the latest version of Gravity Forms.
• Runs the acceptance tests for Gravity Flow.
• Generates an HTML report in the tests/acceptance-tests/_output folder.
• Makes Chrome available via VNC

Gravity Forms Add-On developers can use the same Docker configuration and just tweak the plugin slug. For other WordPress plugins – just strip away the Gravity Forms dependencies or switch them out for your own.

The configuration consists of 6 files in the project root. If you’re developing a Gravity Forms Add-On, only three files require minor edits.

Dockerfile: This file sets up the main service which will run the Codeception tests. It also includes WP-CLI to allow Gravity Forms to be installed via the Gravity Forms CLI Add-On. Once Docker has built this image for the first time it caches the image locally.

docker-entrypoint.sh: As the filename suggests, this file runs every time the containers are fired up to run the tests. This script takes care of installing the plugin dependencies – in this case, it installs Gravity Forms and verifies the checksums. If you’re developing a Gravity Forms Add-On, you don’t need to modify this file.

docker-compose.yml:This file is the Docker Compose configuration which sets up all the services in a private network. Adjust the plugin slug.

codeception.dist.yml: This is the Codeception configuration file. You only need to edit the list of plugins that WPBrowser will activate.

composer.json: This file is copied into the container created by the Dockerfile and defines the dependencies for installing Codeception. You don’t need to have Composer installed – it’s all done inside the container.

.env.sample: (Optional) You can use environment variables instead if you prefer. Copy to .env and add your Gravity Forms license key. If you have access to the Gravity Forms repository on GitHub, you can use the latest development master by specifying a GitHub token otherwise leave it blank. Add the .env file to your .gitignore.

If you’d like to try the Gravity Flow acceptance tests, install Docker, clone the Gravity Flow repository, add your Gravity Forms license key to .env and run the following command from the plugin root.

docker-compose run --rm codeception run -vvv --html


Windows users only: enable hard drive sharing in the Docker settings.

If you’re on a Mac, you can open vnc://localhost:5900 in Safari to watch the tests running in Chrome. If you’re on Windows, you’ll need a VNC client. Password: secret.

Continuous Integration

One of the goals was to include acceptance tests in the continuous integration process, so they run more frequently – not just before every release but also on every push to every branch. That way we get a heads-up much earlier if a new feature breaks existing functionality.

I think of it as development hygiene.

After evaluating a few different CI services, I decided on CircleCI. They support the latest version of Docker Compose, it’s fast, and the support for the paid levels is great. Public open source projects get 4 “containers” (virtual machines), each of which can run four more containers in parallel making a total of “16x parallelism”.

CircleCI provides a really easy way to split tests by duration and run them concurrently on all the available VMs. This is especially helpful for acceptance tests which are, by their nature, relatively slow to run.

For a few tests, it doesn’t make a big difference. However, for Gravity Forms, where the number of tests is about to race past 100, the speed is making a massive difference.

Check out the .circleci config file I’m using to split the tests across multiple containers.

You can see all the results of the Gravity Flow acceptance tests on CircleCI.

You may have noticed that I’m using CircleCI for acceptance tests and TravisCI for the unit/integration tests. Why both? It’s faster to kick off both in parallel, it’s easier to troubleshoot issues, and it’s better redundancy. Also, Travis CI is great for the build matrix, and CircleCI is great for splitting up the tests.

Acceptance Test-Driven Development

Acceptance tests do give a lot of peace of mind at release time, and they’re very useful during development and continuous integration to pick up regression issues early, but they’re also really helpful even before development starts.

This Docker setup makes acceptance test-driven development a lot more accessible for the whole team. Support engineers can write concise tests to reproduce bugs instead of writing tedious explanations, and the tests will fail until the bug is fixed. The great thing, of course, is that once the bug is fixed, it’ll stay fixed. The same goes for product managers – they can write the tests the way they expect new functionality to work.

The development processes for both Gravity Forms and Gravity Flow use this approach.

Tips for Writing Acceptance Tests

Take a look at the Gravity Flow acceptance tests and use the Codeception documentation.

One of the biggest gotchas I found was that the tests appeared to be flaky – sometimes they’d pass locally, but they’d fail during continuous integration. This is because the local services have lower latency but the networked services need a bit more leeway.  For example, after a page load use performOn() or $I->waitForText( 'Text', $timeout_in_seconds ) instead of $I->see('Text'). Note that the waitFor functions are case sensitive and don’t count towards the assertions total. The see functions are not case sensitive and do count as assertions.

Tests are not isolated, so each test will be affected by all previous tests. This makes the tests faster to run, but it also means that you need to bear it in mind if a test depends on global settings such as locale. If you’d prefer to reset the database after every test, then add the WPDB module and specify DB dump file.

Remember that the site runs in a different process to the tests, so actions and filters will not work inside tests. However, database queries, like updating options, can be very useful.

Groups are particularly useful as a quick way of running one test or just a few. Tests can be split into groups by adding the @group pseudo-annotation comment to the top of the test. This is the easiest way to run a single test when you’re working on it. Multiple groups are supported.

// @group admin
// @group form-editor

Run groups by specifying the -g option. For example:

docker-compose run -rm codeception run -g admin -g form-editor

Codeception has an interactive console that you can use to try commands before adding them to a test:

docker-compose run --rm codeception console acceptance

If there’s a test you want to commit, but it’s not ready to be included in the automated test suite then you can tell Codeception to skip it:

$scenario->skip();

Take a screenshot at any point, and it’ll appear in the _output/debug folder.

$I->makeScreenshot('My Screenshot')

Final Thoughts

The only way we’re going to be able to retain the trust of our users and at the same time meet the all the challenges that are just around the corner is by understanding the all the risks and by doing our best to mitigate them. By embracing acceptance testing deeply, I believe we can do that, and help WordPress continue its incredible growth well into the future.