A flaky test is worse than no test. Delete it.
It defies common sense and picks a fight with our engineering instincts. Some coverage is better than none. A test that passes after enough retries is better than nothing, right…right?
Nope. The green checkmark tells you to stop looking at that feature. The confidence is unearned. The coverage (like the cake) is a lie. One thing is true: you have a bug (maybe feature, maybe test) hiding behind a soothing green dashboard.
The inheritance
When I took over test for VS Ultimate, I inherited literally tens of thousands of automated tests. Many of them were flaky, a problem the team had "solved" with automatic retries built into the test runner.
Any test that stayed red after several retries got handed to a team of contract testers in China to investigate. The debugger's "nightly" run took days, and each pass generated around a thousand investigations. The yield: roughly one real product bug for every forty hours of that work. A full engineer-week per bug, most of it spent chasing ghosts.
Two rules
So I implemented two simple rules: set retries to zero; delete any test that fails for any reason other than a real product regression.
Why not quarantine? Because quarantine creates an airport's worth of psychic baggage the team carries forever. Flaky tests (even in quarantine) steal focus and waste resources for no benefit. Deletion brought focus back. Each time an engineer deleted a test, they had to evaluate the coverage needs for that area. They triaged, prioritized, and made deliberate choices about where their time bought the most quality.
What it bought
With the flakiness gone, the runs got fast. The "nightly" run finally earned its name: trustworthy feedback every morning. Our engineers stopped maintaining test runners and suites and started testing new features. Everyone talks about shifting quality left. We did it.
Green is a status, not a destination.
Measure twice, cut once. The aphorism leaves out the most important part: measure the right thing.
Every org has a dashboard, and someone whose job is keeping it green. But staying green chases the grade, not the information. (And believe me, I know about chasing grades.) Red, real red, is information you can act on.