Visual regression testing in Nexus StudioQA.
Visual regression testing in Nexus StudioQA compares a screenshot from each run with a baseline image you approved. The first run with --visual plants the baseline. Later runs count the pixels that moved, ignore anti-aliased edges and keep a diff image when the change passes the tolerance. By default a regression is recorded but does not fail the test.
By the Nexus StudioQA team, Verdict System LLC.
Last updated: .
What each run compares
Two kinds of checkpoint, each with its own tolerance: the share of pixels allowed to differ before the change counts as a regression.
| Checkpoint | What is photographed | Default tolerance |
|---|---|---|
| Whole page | The end-of-test screenshot of the viewport, saved under the name baseline. | 1.0% of pixels (NEXUS_VISUAL_THRESHOLD) |
| Element | One element, set inside the test with visualCheck(page, 'cart-total', page.locator('#cart-total')). | 0.1% of pixels (NEXUS_VISUAL_ELEMENT_THRESHOLD) |
How a change is decided
Baselines are kept per browser and platform
They live in
visual-baselines/<browser>@<platform>/in the project folder, so they can be versioned with your code. A Windows image is never compared with a Linux one, because fonts and anti-aliasing differ. A browser and platform with no baseline plants its own.Anti-aliased edges are not counted
A pixel that only moved because an edge was re-blended is painted amber in the diff and left out of the count. A pixel counts as changed when a color channel differs by 10 or more, and NEXUS_VISUAL_PIXEL_TOLERANCE changes that between 0 and 255.
Photos are retaken until they settle
A photo is retaken until two in a row are byte-identical, three shots at most. A page that never settles is reported as unstable instead of becoming the baseline unnoticed.
A different size is not stretched
The two images are compared at native size over the region they share, aligned top-left. Area that does not overlap counts as different.
A check that did not run fails
If the comparison cannot run, for example an image is missing or the native image library is not installed, the test fails whatever the strict setting says. A check that never happened is not reported as a pass.
The settings that matter
Each one is on the Visual Regression screen of the Studio, with the default below.
| Setting | Default | What it does |
|---|---|---|
| Compare pixels on every run | Off | Turns the comparison on for every run. --visual turns it on for one run. |
| A regression fails the test | Off | With NEXUS_VISUAL_STRICT=true, moved pixels past the tolerance fail the test. Off, the regression is recorded and the test keeps its verdict. |
| Masked on every photo | None | CSS selectors painted over in every photo, such as a clock or an ad (NEXUS_VISUAL_MASK). |
| Regression folders kept | 10 | Newest folders kept per test and checkpoint (NEXUS_VISUAL_REGRESSION_RETAIN). |
What you review and approve
Each regression is filed under visual-regression with its own files, and the Studio shows them in four views: Side by side, Swipe, Diff and Blink, next to the earlier runs of the same checkpoint.
| File | What it is |
|---|---|
baseline.png | The approved image. |
current.png | This run’s image. |
diff.png | Red for changed pixels, amber for anti-aliasing ignored, gray for unchanged. |
visual-report.md | The numbers in plain text. |
result.json | The same result as data. |
What it cannot tell you
Pixels cannot tell a planned redesign from a bug, so a person approves each change: Approve this change in the Studio replaces one checkpoint’s image, and nexus baseline --update <TC_ID> does the same from the command line. Anything that changes on its own, such as a clock, an ad or an avatar, has to be masked or it will keep turning the comparison red.
Common questions
Does a visual regression fail my test?
Not by default. The regression is recorded and the test keeps its verdict. Turn on
NEXUS_VISUAL_STRICTand a regression past the tolerance fails the test.Why is there a baseline per operating system?
The same page renders with slightly different fonts, anti-aliasing and scrollbars on Windows, Linux and macOS. Comparing across them reports a change that is only the renderer, so each browser and platform keeps its own baseline.
How do I stop a clock or an ad from failing the check?
Mask it with a CSS selector in the visual settings, or use an element checkpoint on the part of the page you care about. An element checkpoint is held to a tenth of the whole-page tolerance, so it is strict without a moving ad turning it red.
Where are the baseline images stored?
In
visual-baselinesin your project folder, not in the evidence folder, so they can be committed with the code. The regression diffs are run output and live invisual-regressionunder evidence.
Run it on your own machine.
Try every feature free, on your own machines. No card needed.