Flaky test detection that keeps the retry visible.
Flaky test detection tells a test whose attempts disagree from a real bug. Nexus StudioQA records every retry of Playwright and Selenium runs: a failure that a retry passed is FLAKY, a failure on a single attempt is BREACHED, and a failure that is itself inconsistent is BREACHED_FLAKY. A retry never turns a defect into a green run.
By the Nexus StudioQA team, Verdict System LLC.
Last updated: .
Three verdicts, one rule
FLAKY means the attempts disagreed, and nothing else. A failure on a single attempt, with zero retries, is never labeled flaky.
| Verdict | Attempts | Last attempt | CI exit code | Failure video |
|---|---|---|---|---|
| FLAKY | Failed, then retried | Passed | 0 | No |
| BREACHED | One | Failed | 1 | Kept by default |
| BREACHED_FLAKY | Failed, then retried | Failed | 1 | Kept by default |
How retries are counted
The attempt number decides
A passing test with a retry number above zero is FLAKY, unless a selector was healed, in which case it is AUTO_RECOVER. A clean pass needs retry zero and no healed selector.
Environment first
Before a failure can be BREACHED or BREACHED_FLAKY, the engine rules out a harness error, a crash, a block, an unreachable host and a timeout. A timeout or a CAPTCHA does not arrive as a bug report.
A slow first attempt is not flaky
When the first attempt failed only on time, a closed browser or a wait for visibility, and the retry passed, the verdict is SECURE: that shows latency, not instability.
The count stays with the run
Each History row records retry_attempt and attempts. The failed attempt's raw-data.json is kept, so a retry that hid the failure still leaves its log.
Spotting a test that flips
History trends
History keeps one row per run for each test and browser or device. It shows trends by session, day or week: Regressed, Recovered, Still broken and Slower.
Risk ranking
The ranking orders tests by how often they flip, break or cost time, with a predicted risk from a logistic regression over your history and an EWMA fallback when data is thin.
Nothing is flaky below five runs
Below five runs, History and the risk ranking call nothing flaky, because one or two runs cannot show a pattern.
Common questions
Do retries hide real bugs?
They can when the only states are pass and fail, because a retry that passes turns a red run green. Nexus StudioQA keeps the retry visible: a test that failed and then passed is FLAKY, not SECURE, and shows in the report and in History.
Does a FLAKY test break the build?
No. FLAKY exits 0 in CI, files no Jira ticket and keeps no video, but it stays visible as a pass with a reservation.
Why are Playwright tests flaky?
Usual causes: shared or leftover data, a wait on the wrong thing, an unstable locator, the environment, or a real intermittent bug. A retry is a way to see an unstable test, not a fix.
Does FLAKY say why a test is unstable?
No. It says the attempts disagreed. Finding the cause in the application or in the test is still work for a person. With retries off, an intermittent failure is a BREACHED.
Run it on your own machine.
Try every feature free, on your own machines. No card needed.