Your tests stay Playwright, Selenium and pytest. We don't replace them. We make them enterprise-ready, and you own your code, your evidence and your history.

Forensic test verdicts: 15 outcomes instead of pass or fail.

Forensic test verdicts replace pass or fail with a named outcome that says whose fault a result is. Nexus StudioQA gives every run one of 15: a clean pass, a pass with a reservation, a broken application, an out-of-date test, or an environment problem such as a timeout or a bot block. Only a failure that belongs to the application is a bug report.

By the Nexus StudioQA team, Verdict System LLC.

Last updated: .

The 15 verdicts, by group

The 15 verdicts are SECURE, CHAOS_RESILIENT, AUTO_RECOVER, FLAKY, PERFORMANCE_DEGRADED, A11Y_VIOLATION, SECURITY_VIOLATION, BREACHED, BREACHED_FLAKY, CHAOS_BREACHED, TEST_DRIFT, TIMEOUT, UNREACHABLE, BLOCKED and CRASHED.

GroupVerdictsCI exit codeIs it a bug report?
PassedSECURE, CHAOS_RESILIENT0No
Passed with a reservationAUTO_RECOVER, FLAKY, PERFORMANCE_DEGRADED, A11Y_VIOLATION, SECURITY_VIOLATION0No, but the report and History show the reservation
The application brokeBREACHED, BREACHED_FLAKY, CHAOS_BREACHED1Yes
The test is out of dateTEST_DRIFT1Yes, against the test: repair the line
The environment is at faultTIMEOUT, UNREACHABLE, BLOCKED, CRASHED1No

How the engine picks one

  • Three questions, in order

    Did the test produce a result at all? If it passed, did it pass cleanly? If it failed, whose fault is it: the harness, the environment or the application?

  • A fixed order of patterns

    For a failure, the engine tests the error message and the network log against groups of patterns in a fixed order. The first group that matches decides, so the order is part of the design. Playwright and Selenium WebDriver wording are both covered.

  • Not everything gets a verdict

    SKIPPED, INTERRUPTED and CONFIG_ERROR are not among the 15, because none of them says anything about your application. They file no evidence folder and no History row.

How an environment failure is named

VerdictWhat the engine sawWhat it means
TIMEOUTPlaywright's Timeout 30000ms exceeded, Selenium's Wait timed out after, a TimeoutErrorThe run never answered in time. A slow page, not proof the application is broken.
UNREACHABLEA name that did not resolve, a refused connection or an offline machine, before any page was observedThe host could not be reached at all.
BLOCKEDA 403 or a 429, a CAPTCHA, an access-denied page or a bot challengeThe site defended itself, and the request never reached your product.
CRASHEDTarget closed, Page crashed, a DevTools protocol error, SessionNotCreatedErrorThe browser or the driver died: look at the machine and the driver.

Where the full rules live

This page is the summary. How 15 verdicts are decided gives the full rules, in the order the engine applies them, with the limits of the method. The verdicts index answers each verdict on its own page.

Common questions

  • Is a pass with a reservation a pass in CI?

    Yes. FLAKY, AUTO_RECOVER, PERFORMANCE_DEGRADED, A11Y_VIOLATION and SECURITY_VIOLATION exit 0, so they do not break the build. The reservation shows in the report and in History.

  • Does a TIMEOUT mean the application works?

    No. A TIMEOUT is not proof that the application works. It says the run cannot tell, so it is not reported as a bug in the application. A test that times out on every run deserves a look.

  • Can accessibility or security checks change a verdict?

    Only when you make them strict in Settings. Then, on a test that passed, they can turn the verdict into SECURITY_VIOLATION, A11Y_VIOLATION or PERFORMANCE_DEGRADED, in that order. A failed test is never escalated.

  • How reliable is the classification?

    It reads error text, so a custom assertion message that contains 403 or Forbidden can read as BLOCKED. Keep custom messages specific, and open metadata.json when a verdict surprises you.

Run it on your own machine.

Try every feature free, on your own machines. No card needed.