Your tests stay Playwright, Selenium and pytest. We don't replace them. We make them enterprise-ready, and you own your code, your evidence and your history.

Flaky test detection that keeps the retry visible.

Flaky test detection tells a test whose attempts disagree from a real bug. Nexus StudioQA records every retry of Playwright and Selenium runs: a failure that a retry passed is FLAKY, a failure on a single attempt is BREACHED, and a failure that is itself inconsistent is BREACHED_FLAKY. A retry never turns a defect into a green run.

By the Nexus StudioQA team, Verdict System LLC.

Last updated: .

Three verdicts, one rule

FLAKY means the attempts disagreed, and nothing else. A failure on a single attempt, with zero retries, is never labeled flaky.

VerdictAttemptsLast attemptCI exit codeFailure video
FLAKYFailed, then retriedPassed0No
BREACHEDOneFailed1Kept by default
BREACHED_FLAKYFailed, then retriedFailed1Kept by default

How retries are counted

  • The attempt number decides

    A passing test with a retry number above zero is FLAKY, unless a selector was healed, in which case it is AUTO_RECOVER. A clean pass needs retry zero and no healed selector.

  • Environment first

    Before a failure can be BREACHED or BREACHED_FLAKY, the engine rules out a harness error, a crash, a block, an unreachable host and a timeout. A timeout or a CAPTCHA does not arrive as a bug report.

  • A slow first attempt is not flaky

    When the first attempt failed only on time, a closed browser or a wait for visibility, and the retry passed, the verdict is SECURE: that shows latency, not instability.

  • The count stays with the run

    Each History row records retry_attempt and attempts. The failed attempt's raw-data.json is kept, so a retry that hid the failure still leaves its log.

Spotting a test that flips

  • History trends

    History keeps one row per run for each test and browser or device. It shows trends by session, day or week: Regressed, Recovered, Still broken and Slower.

  • Risk ranking

    The ranking orders tests by how often they flip, break or cost time, with a predicted risk from a logistic regression over your history and an EWMA fallback when data is thin.

  • Nothing is flaky below five runs

    Below five runs, History and the risk ranking call nothing flaky, because one or two runs cannot show a pattern.

Common questions

  • Do retries hide real bugs?

    They can when the only states are pass and fail, because a retry that passes turns a red run green. Nexus StudioQA keeps the retry visible: a test that failed and then passed is FLAKY, not SECURE, and shows in the report and in History.

  • Does a FLAKY test break the build?

    No. FLAKY exits 0 in CI, files no Jira ticket and keeps no video, but it stays visible as a pass with a reservation.

  • Why are Playwright tests flaky?

    Usual causes: shared or leftover data, a wait on the wrong thing, an unstable locator, the environment, or a real intermittent bug. A retry is a way to see an unstable test, not a fix.

  • Does FLAKY say why a test is unstable?

    No. It says the attempts disagreed. Finding the cause in the application or in the test is still work for a person. With retries off, an intermittent failure is a BREACHED.

Run it on your own machine.

Try every feature free, on your own machines. No card needed.