Killing Flaky Tests: A Practical Playbook
Flaky tests quietly destroy trust in a test suite. A concrete playbook for finding the flakes, diagnosing the real cause, and stopping them from coming back.
A flaky test — one that passes and fails without any code change — is worse than no test at all. It trains the team to ignore red, and an ignored suite is a dead suite. Here is the playbook I use to kill flakes for good.
The moment people start saying “just re-run it,” the suite has lost its authority. Every flake you tolerate makes the next real failure easier to dismiss.
Step 1 — Make flakes visible
You cannot fix what you cannot see. Before anything else, start tracking which tests fail intermittently.
- Record every test failure with its test name and commit
- Flag any test that has both passed and failed on the same commit
- Keep a running “flaky list” the whole team can see
A test that fails 1 in 20 runs is invisible day to day but poisons confidence over a month. Data makes it undeniable.
Step 2 — Quarantine, don’t ignore
A known flake blocking the pipeline pressures everyone to bypass CI. So move it out of the blocking path — but into a visible holding area, not oblivion.
- Tag the test as quarantined so it runs but doesn’t fail the build
- Give every quarantined test an owner and a ticket
- Cap the quarantine — if the list grows unbounded, that is its own alarm
Quarantine is a hospital, not a graveyard. A test that sits quarantined for a month with no investigation should be deleted — a test nobody will fix is just noise.
Step 3 — Diagnose the real cause
Flakiness almost always traces to a handful of root causes. Work through them:
- Timing / async — asserting before the app is ready. The number-one cause.
- Test interdependence — one test leaks state that another depends on.
- Shared/real data — tests fighting over the same records or environment.
- Non-determinism — random data, time-of-day, ordering, timezones.
- Resource contention — parallel tests hitting limits (connections, ports).
Step 4 — Fix the cause, not the symptom
The tempting “fix” is a retry or a sleep. Both hide the flake instead of removing it.
// Symptom-hiding: brittle and slow
await page.waitForTimeout(3000);
await expect(page.getByRole('heading')).toBeVisible();
// Cause-fixing: wait for the actual condition
await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
The second version waits for the state you care about, not an arbitrary duration. It is both faster and reliable. Apply the same idea everywhere: replace fixed waits with explicit waits, isolate test data, and reset state between tests.
Step 5 — Prevent regression
Once a flake is fixed, keep it dead:
- The test isolates its own data and cleans up after itself
- No fixed
sleep/waitForTimeout— only condition-based waits - Tests pass when run in a random order
- Tests pass under parallel execution
- A quarantined test has an owner and a deadline
Why it’s worth the effort
A reliable suite is a force multiplier: green means ship, red means stop, and nobody argues. A flaky suite is a tax on every merge and a slow erosion of the one thing automation exists to provide — trust. Killing flakes is not cleanup work; it is protecting the value of everything else you automated.
