tl;dr: Along with a continued focus on security, stability and scalability, 2019 will bring a more proactive approach to accessibility as integral to code quality with help from Rian Rietveld.
Now 2018 is coming to a close, I’d like to share some of my thoughts on the quality of the code in Gravity Forms over the years and how we’re going to improve it in 2019.
There were several bugs in the recent release of Gravity Forms v2.4 which needed a couple of minor releases deploying within a few days and I was asked by someone who was concerned about this if it’s possible to prevent defects completely in a release. It’s a fair question so I thought it would be helpful to write a post in case others have the same concern – is Gravity Forms making progress in this area?
Without getting into sigma levels of quality, some bugs will always be inevitable. However, all of them are ultimately my responsibility, and I still face-palm to varying degrees whenever they pop up so I always find myself striving for zero defects.
We do have systems and procedures in place that we’re continually refining. Here’s what we’ve done to improve the quality of the releases since I joined the team in 2012.
2013
In 2013, we started adding a unit tests. We began with the Gravity Forms API which is used extensively throughout all Gravity Forms products so it helped prevent a lot of regressions very quickly. We now have a lot more coverage (between 50%-100% in most of the important areas) and every time code is pushed to the repository over 70k checks run against a matrix of PHP and WP versions on Travis CI from PHP 5.2.4 to PHP 7.3 and WP going back to 4.4.
We also started enforcing WordPress coding standards. We want all WordPress developers to feel at home in Gravity Forms code so we do our best to keep WordPress as the reference point for all technical and UI decisions. Not having to reinvent the wheel has saved us countless hours over the years.
2014
In 2014, we introduced a formal system of manual user testing with an external consultant for each major release. Our Beta and RC releases still don’t get nearly enough traction so this stage is essential for picking up bugs before the final release.
2015
In 2015, we brought on Dewhurst Security, maintainers of WPVulndb.com & WPScan, to conduct security audits for all our products 5 times a year. They also do a great job of consulting on specific issues which need in-depth investigation or due diligence and we continue to learn a lot from them. This has given us a much more proactive approach to security and we now have a lot more confidence in every release.
2016
In 2016, we started writing automated browser tests which check that the PHP and JavaScript are working correctly together on the form. We’re now up to 139 test scenarios with over 2200 individual checks which run on CircleCI against the latest version of WP every time we push code to a branch. These have saved us from releasing regressions on countless occasions and, as we keep adding to them, the stability between releases continues to improve.
We also started opening up our private GitHub repository to some trusted third-party add-on developers and we continue to add more each year. This has been hugely successful and we’ve all benefited enormously from the tighter relationship with our partners in the Gravity Forms ecosystem. Developers frequently submit and review Pull Requests (changes to the codebase) and it’s a real pleasure to work with them more closely – a special shout out to the teams at GravityView and Gravity PDF – Gravity Forms is a better product for all their excellent contributions both in terms of advice and code.
2017
In 2017, we started requiring peer reviews for all Pull Requests. Previously, not all PRs were reviewed, but now every single line that’s merged has been reviewed and tested by another developer. Four eyes are always better than two and this policy has helped to prevent bugs and keep the code standards consistent in a very clear and self-evident way. I’m really pleased with how this is working. One reason this is working so well in our team is the high level of trust between the developers so the feedback is always constructive, helpful and well received. It’s important to highlight that the safe environment for developers has a direct impact on the quality of the code.
2018
In 2018, we refactored the database layer entirely for better scalability and performance. This was a huge achievement and I’m very proud of the whole team effort that managed to pull it off successfully while remaining faithful to our commitment to backwards compatibility. The whole process took about a year and we even had to start from scratch at the eleventh hour at one point, but it was absolutely necessary, not just for enterprise customers, but to ensure that Gravity Forms continues to grow as the leading form-based application platform well into the future.
Also this year, manual testing was given a significant boost by bringing the process in internally. We now have a small team dedicated to testing and they do a fantastic job of manually testing every aspect of all new add-ons and features and catching bugs before the final release.
—
So the quality of our releases has never been better.
Despite these efforts, we still ended up with some serious bugs in the initial release of 2.4, so there’s still room for improvement. The most serious issue was particularly irksome because the PR included unit tests and was reviewed not by four eyes but by six – I took the unusual step of getting a second opinion before approving it myself. The bright side was the speed at which the team got the fix out. The customer reported the bug report on 3 Dec, the support team triaged the issue quickly by the end of the following day, and we came up with a fix and got it reviewed and merged it so it could be deployed the next day on 5 Dec. It was a great example of how remote teams working in different time zones can really shine by keeping a project going round the clock. Additional tests are in place to ensure the issue can’t reoccur but the question still bothers me – how could I have avoided it in the first place?
2019
What are we going to do in 2019 to continue to improve the quality of the code in our releases?
Better Automated Testing
- We need to get better at adding more comprehensive tests along with our PRs – unit tests and browser tests for both fixes and new features. We do this frequently already, and there’s a consensus in the team that it’s definitely the right approach, but we can do better. It can be a pain to write tests, and it can slow down development which is tricky when there’s a time pressure to deliver a fix but, in the long run, it does prevent regressions and therefore saves time.
- Our test coverage is respectable for most areas, however, where time permits we’re going back over the code and aiming for 100% in all important areas of existing code – some of which was written in 2008.
- We’ll also extend automated browser testing to different browsers and operating systems. We’ve started doing this with Gravity Flow using Browserstack and it’s proving very helpful especially for automating tests on real mobile devices.
An Integrated Approach to Accessibility
Without a doubt, the most important improvement in the quality of our code in 2019 is going to be in the area of accessibility. We’ve always done our best to make Gravity Forms as accessible as possible and v2.4 contained more accessibility fixes than any previous release but our approach over the years has not been proactive enough and we’ve lacked awareness of our own weaknesses and shortcomings in this area as developers. The result has been far too much retrofitting – and not enough integration of accessibility into the development process as another aspect of code quality.
I’m thrilled that we have recently enlisted the help of Rian Rietveld, of Level Level. I’ve been following Rian’s work since her talk at WordCamp Europe in Seville in 2015 up to her recent work as the WordPress Accessibility Team lead and her work on the Gutenberg project to rebuild the WordPress editor, so I’m really excited to be able to bring her expertise to Gravity Forms.
The initial review is already underway, and the results will have a significant impact on the next major releases which will have a focus on improved styling. After the audit phase, Rian will continue to help us out by reviewing PRs, submitting patches, and generally educating us all in this area which is absolutely crucial to the continuing success of Gravity Forms.
Final Thoughts
Matt Mullenweg recently remarked that WordPress is one of the few open source products that is used by the whole spectrum of customers from individual consumers to non-profits to global organisations. We also see this unusual phenomenon in the range of Gravity Forms customers so we have to ensure that our products are both enterprise-ready and consumer-friendly. This means that we need to continue to improve the security, stability and scalability of our products. However, the size and breadth of our customer base and the magnitude of the installation base bring an even greater responsibility to ensure that accessibility is integrated more deeply and consistently into the way we gauge the quality of our code at every stage of development.
Leave a Reply