After writing a dedicated article about Automated Testing, I moved to the next phase: Visual Testing. All the hard work helped me came out with a bunch of insights you might consider if you plan to test your WordPress site with a Visual Regression tool. I highly recommend doing so.
Also, @dtbaker‘s tweet and article about Visual Regression Testing made me write about my experience with this hot topic.
Doing some Visual Regression Testing on #WordPress themes using @nightwatchjs, this was fun to setup: https://t.co/zoKl2p4vPE
— dtbaker (@dtbaker) March 2, 2017
Why testing your WordPress site with a Visual Regression tool?
Visual Regression Testing is the procedure where a software interface is tested and verified visually (with screenshots in our case) to know if it still performs correctly after some modifications.
In this article, I plan to focus only on the WHY part, so I will not include the technical details right here since I plan to continue expanding the topic in future articles tailored on the HOWs.
Bare in mind that my Visual Tests are public in a GitHub repository, so don’t hesitate to check it out. It will complete your perspective about what I’m going to write next.
Going the extra mile
Well, in case it is not yet obvious yet, testing pixel by pixel has significant advantages. Even if you may believe that you can double check your website or your digital product manually, please let me show you why you can’t achieve the same outstanding results.
Let’s take a WordPress site for instance:
- My blog is a WordPress site (proudly hosted by the Patch theme developed by Pixelgrade), and it has 21 pages and articles, which is not a lot.
- A web page contains a considerable amount of HTML elements. My homepage has 574 DOM nodes, and this doesn’t mean much as well.
- Websites are styled entirely with CSS, each HTML element from above can change with CSS properties, and there are around 340 style properties.
- Ah, we’re talking about WordPress, where you can choose to install up to 49.000+ plugins. Mine has 18 active plugins.
- Don’t forget that all the style comes from your active theme.
I don’t pretend I want to teach you Math or Statistics, but in this list, I’ve bolded a few chances concerning the way your website can change without the human eye detection. So don’t rely too much on human nature. At least not in this particular scenario.
If you still deeply believe that you can correctly maintain your website without a technical supervision, just give it a second thought. Think again. Time will pass, details will be forgotten and the theme updates will show up alongside with the ones from plugins. On top of that, WordPress itself can ask for an upgrade as well.
How to Accurately Handle Visual Regression Testing?
In the last couple of months, I discovered that maintaining 15 WordPress premium themes is a hell of a task, and trying to deliver top-notch updates require an enormous amount of hard work and obsessive attention to details.
So I started playing with the following libraries:
- Wraith
- Gemini
- BackstopJS
- Nightwatchjs
- Galen Framework
- But the list is way longer than you may think.
I don’t want to get into specific details about what I’ve learned while playing around with all of them (it might take a while). Long story short, I ended up choosing Wraith.
My choice is not linked to the fact that the other libraries were not good enough. We should all give a big round of applause to the great people who are maintaining these excellent, FREE TO USE, and useful tools.
I’ve picked Wraith because our technical requirements were fulfilled entirely by it, but also because I’ve seen a cleaner way in using Wraith than in the other libraries.
Oh, by the way, by requirements I mean that:
- I need to run visual tests with a task runner like GULP
- Automatically called by a Continuous Integration service like CircleCI
- Triggered by a GitHub commit
- Build results on a custom VPS from DigitalOcean
- In the end, send a Slack notification with the results.
Sound a bit overwhelming if you read the above sentences with a loud voice, right? Well, let’s focus on what really matters and let me introduce you into some real case studies.
Not in the mood for getting your hands dirty in terminal? Go for a premium service like diff.io ghostinspector or percy.io
Case Study #1: How it helps us in the process of maintaining a WordPress theme?
We give our best to continuously improve our products, to keep our customers happy, but we don’t want to risk their loyalty by bringing unexpected (or ugly) changes during a theme update.
To achieve great results is quite simple: keep our designer happy. I mean it. Every designer in the world develop a special relation with the products and automatically with the details that can draw a huge difference.
Losing small details === sad designer === sad customer.
Let’s take an example of how this could work out. For example, we faced some changes on our Hive theme (the desktop version of the layout), and when we checked on mobile, no one detected a little color change from gray to black on the post date:
What about that, ha?
From a designer’s point of view, this aspect is quite crucial since the hierarchy of the post’s author and other meta information is well defined by a small color variation.
These kinds of accidents can occur, we are only humans, but the way we check and handle these cases makes us better players.
Case Study #2: Check your website after moving it
You know that moment when you build a new WordPress site in a development environment like an IP or a localhost? I bet that you fear about losing small details during the transfer.
The last time I’ve changed my hosting, I’ve copied my website in a development environment like https://aldev.ro/al_test.
I still keep this site up since I may need to compare the live blog with it (you will see in the next scenario).
Even if the website moving process went smoothly, my eyes were lazy enough, so I didn’t test it with a non-logged user.
But the Wraith opened my eyes with this result:
And we were both right. My website was super ok with me logged in, but Wraith was testing my site in the incognito mood, an excellent way for catching problems like the one above. The issue was linked to handling image requests with SSL.
Case study #3: Check your site after an update
Another tricky move is updating a website or a website component. WordPress has a very simple update process: just click, and the code will be updated. However, it is the code we are talking about. No one can ensure you that the visuals won’t change and this brings us to the need of having an update flow.
Regardless of the production version of my website I always keep a test instance, either locally or on a testing server, where I firstly update WordPress before checking in production.
- Back-up the test instance.
- Update the test instance.
- Run Visual tests to compare the test instance with the production one.
- Check the results and decide if the changes are acceptable.
- If the tests fail, fix the problem, and redo the step 3.
- If the tests run successfully, you can update the production version with less stress.
This is what I’ve done with the latest update of Patch from 1.2.2 to 1.3.1. If you look at the changelog, you can agree that it is not an easy peasy task to do.
The first issue was obvious; I don’t need Wraith to catch these kinds of problems:
Well, the new version of the theme comes with an option for logo size, and in conflict with my child theme, it didn’t work that well. But is OK, some CSS tinkering and the problem is gone for the time being.
An issue which I admit I didn’t catch was related to the article’s titles:
Now the theme comes with uppercase titles; but not for me, I like my titles to be capitalized.
Case study #4: Update PHP version or other server-side tools
Recently, I updated the PHP version on my website from PHP 7.0.10 to PHP 7.1.2. I know for sure that this is a big step and that the new version of PHP may not be compatible with some extensions. But, hey who else likes taking huge risks just for having fun?
I was pretty lucky since there was no real impact regarding the visual of the website. The only visible difference between these two PHP versions is happening on the PHP info page itself.
Thanks for reaching so far!
If I convinced you about the Why, then you’d probably want to know more about the How as well. In the next article, I will dig deeper and share how I tested my blog against the changes I’ve shared in this post. Stay tuned!
If you have a question or a proposal for an interesting scenario I should test with Wraith, hit the comments section, and maybe I sort something out!
March 31, 2017 at 10:27 pm
Looks good 🙂
April 2, 2017 at 10:22 am
Very thorough explanation Andrei! Your style of writing is getting better and better 💪
July 30, 2019 at 2:10 am
The link to the Automated Testing post is now https://a.lup.dev/508/automated-testing-what-can-they-bring/#more-508. Thanks for these articles by the way, just getting started with doing WordPress testing.