Forensic test verdicts: 15 outcomes instead of pass or fail.
Forensic test verdicts replace pass or fail with a named outcome that says whose fault a result is. Nexus StudioQA gives every run one of 15: a clean pass, a pass with a reservation, a broken application, an out-of-date test, or an environment problem such as a timeout or a bot block. Only a failure that belongs to the application is a bug report.
By the Nexus StudioQA team, Verdict System LLC.
Last updated: .
The 15 verdicts, by group
The 15 verdicts are SECURE, CHAOS_RESILIENT, AUTO_RECOVER, FLAKY, PERFORMANCE_DEGRADED, A11Y_VIOLATION, SECURITY_VIOLATION, BREACHED, BREACHED_FLAKY, CHAOS_BREACHED, TEST_DRIFT, TIMEOUT, UNREACHABLE, BLOCKED and CRASHED.
| Group | Verdicts | CI exit code | Is it a bug report? |
|---|---|---|---|
| Passed | SECURE, CHAOS_RESILIENT | 0 | No |
| Passed with a reservation | AUTO_RECOVER, FLAKY, PERFORMANCE_DEGRADED, A11Y_VIOLATION, SECURITY_VIOLATION | 0 | No, but the report and History show the reservation |
| The application broke | BREACHED, BREACHED_FLAKY, CHAOS_BREACHED | 1 | Yes |
| The test is out of date | TEST_DRIFT | 1 | Yes, against the test: repair the line |
| The environment is at fault | TIMEOUT, UNREACHABLE, BLOCKED, CRASHED | 1 | No |
How the engine picks one
Three questions, in order
Did the test produce a result at all? If it passed, did it pass cleanly? If it failed, whose fault is it: the harness, the environment or the application?
A fixed order of patterns
For a failure, the engine tests the error message and the network log against groups of patterns in a fixed order. The first group that matches decides, so the order is part of the design. Playwright and Selenium WebDriver wording are both covered.
Not everything gets a verdict
SKIPPED, INTERRUPTED and CONFIG_ERROR are not among the 15, because none of them says anything about your application. They file no evidence folder and no History row.
How an environment failure is named
| Verdict | What the engine saw | What it means |
|---|---|---|
| TIMEOUT | Playwright's Timeout 30000ms exceeded, Selenium's Wait timed out after, a TimeoutError | The run never answered in time. A slow page, not proof the application is broken. |
| UNREACHABLE | A name that did not resolve, a refused connection or an offline machine, before any page was observed | The host could not be reached at all. |
| BLOCKED | A 403 or a 429, a CAPTCHA, an access-denied page or a bot challenge | The site defended itself, and the request never reached your product. |
| CRASHED | Target closed, Page crashed, a DevTools protocol error, SessionNotCreatedError | The browser or the driver died: look at the machine and the driver. |
Where the full rules live
This page is the summary. How 15 verdicts are decided gives the full rules, in the order the engine applies them, with the limits of the method. The verdicts index answers each verdict on its own page.
Common questions
Is a pass with a reservation a pass in CI?
Yes. FLAKY, AUTO_RECOVER, PERFORMANCE_DEGRADED, A11Y_VIOLATION and SECURITY_VIOLATION exit 0, so they do not break the build. The reservation shows in the report and in History.
Does a TIMEOUT mean the application works?
No. A TIMEOUT is not proof that the application works. It says the run cannot tell, so it is not reported as a bug in the application. A test that times out on every run deserves a look.
Can accessibility or security checks change a verdict?
Only when you make them strict in Settings. Then, on a test that passed, they can turn the verdict into SECURITY_VIOLATION, A11Y_VIOLATION or PERFORMANCE_DEGRADED, in that order. A failed test is never escalated.
How reliable is the classification?
It reads error text, so a custom assertion message that contains 403 or Forbidden can read as BLOCKED. Keep custom messages specific, and open metadata.json when a verdict surprises you.
Run it on your own machine.
Try every feature free, on your own machines. No card needed.