Why tests fail, answered directly.

A failed test is the application's fault, the test's fault or the environment's, and each one needs a different fix. Nexus StudioQA gives every Playwright, Selenium or pytest run one of 15 verdicts that names the cause, so a slow server or a stale locator is never reported as a product bug. Every failure keeps its own report, video and screenshot.

By the Nexus StudioQA team, Verdict System LLC.

Last updated: .

Fifteen questions

  • Why do my Playwright tests fail intermittently?

    Most intermittent failures come from timing: an assertion runs before the page is ready, a fixed wait is too short, or tests share data. Real bugs fail every time. Nexus StudioQA marks a test that passes only on a retry as FLAKY and keeps it in a flaky ledger per browser.

  • Why does my test fail but there is no bug?

    The test failed because of the environment, not the app: a slow server, a network error, a bot check or a crashed browser. Nexus StudioQA separates those runs into TIMEOUT, UNREACHABLE, BLOCKED and CRASHED, so they never count as a real failure of your product.

  • How do I fix flaky tests in Playwright?

    Replace fixed waits with web-first assertions, give each test its own data, and record a trace on the first retry to see what changed. Then fix the cause instead of adding retries. Nexus StudioQA flags each flaky test and files its screenshot, video and logs for every failed attempt.

  • How do I fix flaky tests in Selenium?

    Use explicit waits instead of sleeps, prefer stable locators such as IDs or test attributes, reset state between tests, and avoid shared browser sessions. Nexus StudioQA runs Selenium tests as standard code, grades every recorded selector by fixed rules and marks a test that passes only on retry as FLAKY.

  • What is a flaky test and how do I detect it?

    A flaky test passes and fails on the same code without any change. You detect it by rerunning failures and tracking which tests flip over time. Nexus StudioQA does both: a pass on retry becomes the FLAKY verdict, and a ledger ranks each test by how often it flips.

  • How do I know if a test failure is a real bug or a slow server?

    Check whether the app answered. A real bug returns a wrong result; a slow server never answers inside the time limit. Nexus StudioQA gives the first BREACHED and the second TIMEOUT, so a slow environment is never reported as a defect, and each one keeps its own evidence.

  • How do I prove a test failure with evidence?

    Keep the screenshot, video, logs and report of the failed run together, with a timestamp, and make them tamper-evident. Nexus StudioQA files the report, screenshot and logs of every failure, plus the video of a real one, in its Evidence Locker and can seal the records in a SHA-256 chain that anyone can verify later.

  • What is the best tool to explain why tests fail?

    Look for a tool that classifies each failure by cause and keeps the evidence, not one that only reports pass or fail. Nexus StudioQA gives every Playwright, Selenium or pytest run one of 15 verdicts and files a report, screenshot and logs for each failure, plus a video when it is real.

  • How do I decide Go or No-Go for a release?

    Decide on the results that matter: real failures block, environment problems are rerun, and flaky tests are tracked apart. Write the criteria down before the run. Nexus StudioQA's executive report turns a run into GO, CAUTION or NO-GO, with the verdicts and evidence behind the decision.

  • What is a QA OS?

    A QA OS is one place that creates, runs, judges and stores the evidence of automated tests, instead of separate tools for each step. Nexus StudioQA is a QA OS for Playwright, Selenium and pytest that runs on your own PC, in Docker or on a Linux server.

  • How do I migrate from Cypress to Playwright without rewriting everything?

    Convert by rule, keep the assertions, and run each converted test before you retire the original. Rewrite only what does not convert. Nexus StudioQA's Test Adapter converts Cypress and WebdriverIO tests to Playwright in beta, shows the diff, keeps every assertion and rehearses each test.

  • How do I reduce false failures in automated tests?

    Separate environment failures from product failures, remove fixed waits, isolate test data, and stop counting a pass on retry as a pass. Nexus StudioQA reports environment problems as TIMEOUT, UNREACHABLE, BLOCKED or CRASHED, apart from real failures, so your pass rate shows the product, not the network.

  • What tool gives screenshots, video and logs for every failed test?

    Playwright can record traces and videos, but keeping them per failure and per release is up to you. Nexus StudioQA files a report, screenshot and logs for every failure, and the video of a real one, automatically in its Evidence Locker, on your own machines, and can seal the records with SHA-256.

  • How do I do load testing and Web Vitals in Playwright?

    Run the user journey under load and measure Web Vitals such as LCP, CLS and INP against a budget you set before the run. Compare each run with the last one. Nexus StudioQA's Load tests report each budget as PASS, OVER_BUDGET, REGRESSED or UNMEASURED.

  • How do I test accessibility automatically?

    Run an automated checker against WCAG rules on every page your tests visit, inside the normal test run, and fail the build on new violations. Automated checks find part of the issues; keep manual reviews too. Nexus StudioQA checks accessibility during the run and reports A11Y_VIOLATION as its own verdict.

Symptom to verdict to what to do

SymptomVerdictWhat to do
A test failed, then passed on a retryFLAKYSomething is unstable, in the application or in the test. Find it; nothing was fixed.
A test passed, but self-healing replaced a selectorAUTO_RECOVERFix the original selector. The next change to the page breaks the suite for real.
An assertion failed on a single attempt and no environment pattern explains itBREACHEDThis is the bug report.
The same failure is inconsistent across retriesBREACHED_FLAKYCompare the attempts before you file a bug.
A click or fill waited for a locator that never matched, on a page that loadedTEST_DRIFTThe test is out of date: repair the line.
The run hit its time limitTIMEOUTThe run cannot tell. It is not proof the application works; a test that times out on every run deserves a look.
The name did not resolve or the connection was refusedUNREACHABLECheck the network and the machine. Not a product bug.
A 403 or 429, a CAPTCHA or a bot challengeBLOCKEDThe site defended itself and the request never reached your product.
Target closed or Page crashedCRASHEDThe browser or driver died. Look at the machine and the driver.

Source: the verdicts reference. In CI, every verdict in this table except FLAKY and AUTO_RECOVER exits with code 1, but the four environment verdicts (TIMEOUT, UNREACHABLE, BLOCKED, CRASHED) are not bug reports.

See a verdict on your own failures.

Try every feature free, on your own machines. No card needed.