Recently we needed to update our front-end website component library (the library that contains many discrete user-interface parts of website pages, which we can pull into various, or all, web pages). A straightforward example might be the navigation menu or the footer that appears on every page of vam.ac.uk. These user-interface parts, or components in the library, are discrete in that they contain all files necessary – those for the structure (HTML), visual rendering (CSS) and behaviour (Javascript) – so that the components can be used as independent modules, inserted anywhere on our website.
This particular update meant small, subtle differences to basically everything you see on vam.ac.uk. Our challenge was to check that nothing unexpected was appearing on the website.
The changes to the component library involved an upgrade to the library itself, a change of CSS processor (Dart SASS) and modifications to the stylesheets in all components so that we could introduce newer, improved features – including the way the way the files connect to each other. At the same time, the (numerous!) individual library components were not to change during the upgrade. We had two concerns regarding potential unintended changes – the CSS processor update and the component stylesheet changes.
So, on initial completion of the upgrade the component library, we had an immediate need to evaluate whether there had been inadvertent visual changes to the user-interface of each component. Rather than laboriously pursuing this manually across over a 100 components we used a visual regression tool in the form of Percy (https://percy.io/). The intention was to save time, but also remove potential errors a manual process might introduce.
Visual regression testing basics
Regression testing of digital productions is intended to ascertain that there are no unintended changes to functionality, usability or appearance during the software development or update process. It’s a quality check that existing or unmodified parts of a website still behave and appear as expected. Regression testing is generally required when: (a) new functionality is added, (b) a change is implemented or (c) a defect is corrected.
A visual regression test evaluates only the appearance of visual aspects of a website; as such it’s often implemented as part of a wider software testing strategy. Visual regression testing evaluates specifics such as typographical changes, layout changes of page components and changes in the way graphic components are rendered. Basically, only the things that a user might see – not what they might do.
Visual regression tools have developed from early open source tools such as the BBC’s Wraith, which automates the creation of website screenshots at two different stages of development and ‘diffs’ the screenshot images to indicate, at the pixel level, where the two images differ (think of a ‘Spot the Difference’ puzzle, but with any differences already highlighted). Commercial services now exist that make this comparison even easier to run as part of a CI/CD service (Continuous Integration/Delivery), so any changes made to a codebase can confirm, before deployment, that they don’t introduce any unexpected visual regressions.
Automated visual regression testing processes can be summarised as the collation of baseline screenshots of the existing user-interface as rendered by library components before any changes to the underlying code has been made, the rendering of screenshots after changes have been made and visual change detection. This is effectively comparison of computational differences between user-interface screenshots taken before and after changes to the software.
As with other test runner frameworks, Percy not only creates its baseline user-interface screenshots for subsequent automated comparisons but also generates those screenshots as rendered in several user-agents or browsers on several devices and at more than one breakpoint (screen width).
Why use automated visual regression testing?
Automated visual regression testing can be preferential to manual checks on visual differences as the latter may often fail to detect small changes and can be time consuming. In many cases it is possible for a human eye to miss very small changes, such a single pixel shift in position.
Additionally, automated testing saves time and works well where there are frequent visual changes to a stable, existing website user-interface.
Although our automated regression testing pointed out some changes, we needed a separate manual review process. The first pass of automatically detected differences raised some issues we could dismiss (either because they were ultimately false, or of no concern).
Trialing Percy locally – Fractal v.2.0
Although we had no intention of changing the appearance of any website modules within our component library, the upgrade process required us to change every stylesheet in the existing project. We knew there might be errors in in the newly implemented CSS namespace references, as well as typographical errors in the code for each component.
For this initial trial, we didn’t want to setup Percy as part of a CI/CD (and as discussed above, we are unlikely to run full visual regression tests on every build, due to the build time increase and cost this would incur). So, we undertook the process of testing the website components in manually assigned batches using the Percy command line application, and a simple shell script running the comparisons against different components – and then reviewing any detected differences in the Percy dashboard.
From the very start Percy surfaced numerous things for almost every component. Reviewing these gave us some specific concerns – but there were also some common problems.
One common issue was typographical – for many components the Percy overlay showed text differences that were not immediately explainable; in some instances, it looked as if there were changes in overall font kerning, and in others there were very slight changes to seemingly random glyphs (or characters). Once we’d checked the CSS for components showing these textual differences, we were sure these were not caused by a CSS property value set in the stylesheets.
However – on further investigation we found a truly subtle difference between the old and new stylesheets. The font-size CSS property in many cases was originally calculated as a ‘rem’ value to five decimal places. In the upgraded Fractal v.2.0, the new SASS compiler had calculated the same value, but in this case to ten decimal places. This (very) slight difference in a mathematically defined font-size was impossible to detect by eye, but Percy successfully picked it up.
One other very useful outcome of reviewing was related to colour, or – more specifically – calculated hexadecimal values. Invariably we use SASS functions to look up such values in SASS maps through a more user-friendly key value – a descriptive text string for the colour in question. So, for example, the standard key value for the darkest palette entry we is ‘vam-black’, and in the map this is the hexadecimal value #1D1E20. In several instances the difference led us to function calls that omitted to adopt the new SASS namespacing convention, and in some cases an incorrect hexadecimal value was substituted – with resultant subtle difference.
Summary
Percy is just one of several visual regression tools available, and some add further capability in, for example, the form of integrated end-to-end testing. Our trial of Visual Regression Testing, while in its infancy, is set to continue. We are still evaluating how to deploy such a tool in an ongoing production environment, but as can be seen, it has proved invaluable for finding multiple subtle differences in the upgrade path of our front-end website component library.
