What do flaky tests cost your team?
Enter how many flaky failures your team looks at each week, how long each one takes, and what an engineer hour costs. The calculator multiplies them and shows the engineer time and money you lose. It runs in your browser and sends nothing.
Last updated:
By the Nexus StudioQA team, Verdict System LLC. October 15, 2026.
Your numbers
Example values. Replace them with your own.
Engineer time per month:
Cost per month:
Cost per year:
What do published numbers say?
Google's testing team reported in May 2016 that about 1.5% of all its test runs gave a flaky result, that almost 16% of its tests had some flakiness, and that about 84% of the pass-to-fail transitions it saw involved a flaky test. Most red runs a team reads are noise, not a new bug.
A small rate adds up across a suite. If each test run flakes 1.5% of the time and the tests flake independently, the chance that one run of the whole suite has at least one flaky failure is 1 − 0.985N for N tests:
| Tests in the suite | Runs with at least one flaky failure |
|---|---|
| 20 | 26.1% |
| 50 | 53.0% |
| 100 | 77.9% |
| 200 | 95.1% |
| 500 | 99.9% |
So at 100 tests, about four suite runs in five need someone to look at a failure that is not a bug. Your rate is not Google's: count it from your own CI history, then put the weekly count in the calculator above. Real flakes are not independent either, since one shared cause often fails several tests at once.
How is it worked out?
Failures per week times minutes per failure, divided by 60, gives hours per week. Times 52 gives hours per year, and that divided by 12 gives hours per month. Hours times your hourly cost gives the money. There is no hidden factor.
The example values are round numbers to show the shape of the result, not a survey of any company. Count your own failures from your CI history for a few weeks before you trust the output.
Limits
- It counts only triage time. It leaves out the cost of a release that waited, of a real bug that was ignored because the suite cries wolf, and of CI minutes spent on reruns.
- Your own tests will not have the same cost for every failure. A long average hides a few failures that took a day.
- It does not say how much a tool would save you. Nexus StudioQA labels a failure that passed on a retry as FLAKY and a real defect as BREACHED, which tells you where to look first, but the time you save depends on your suite.
Where to go next
Read Is this test flaky, or is it a real bug? for how the verdicts are decided, or the hardware cost calculator to compare a cloud subscription with a machine of your own.