Stop retrying flaky tests and start reading them

The quickest way to make a flaky suite look green is to set retries to three. The quickest way to ship a race condition to production is the same setting. These are the same action with two different names.

A flaky test fails sometimes and passes sometimes against unchanged code. When that happens, the cause is one of three things: the test is wrong, the environment is wrong, or the product has a real timing bug that only shows up under the right scheduling. A blanket retry treats all three identically, by hiding them. Two of the three can be hidden for a while without much harm. The third is the expensive one, and it is the one retries bury most effectively, because a race in the product looks exactly like a race in the test when all you see is a red run that goes green on the second try.

Measure it before you argue about it

Teams waste a lot of breath debating whether a particular test “is flaky” from memory. Stop debating and start counting. The signal you want is simple: for each test, how often does it fail and then pass on rerun, tracked over time rather than per run.

Playwright gives you the raw material. When a test fails and then passes on retry, the report labels it flaky, distinct from passed and failed. Turn retries on in CI specifically so you can see that label, and record it. A test that flakes once in two hundred runs is noise. A test that flakes one run in five is a defect with a countdown attached, and you want to know which is which before someone turns the number into an argument.

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,   // see flakiness in CI, feel it locally
  reporter: [['html'], ['json', { outputFile: 'results.json' }]],
  use: { trace: 'on-first-retry' },   // a trace for exactly the runs you need to debug
});

Setting local retries to zero is deliberate. On your own machine a flaky test should fail and annoy you, because that is when you are holding the context to fix it. trace: 'on-first-retry' captures a full trace only for the reruns, so you get the network log, the DOM snapshots and the timeline for exactly the runs that are hard to reproduce, and nothing for the runs that passed first time.

A triage flow that is not “add a retry”

Once a test is flagged, run it through the same path every time, so the response is a procedure rather than a mood.

1. 01Detectthe CI report labels a fail-then-pass run flaky; record it against the test
2. 02Reproducerun it in isolation with –repeat-each until it fails on demand
3. 03Classifytest wrong, environment wrong, or a real product race
4. 04Fix or quarantinefix in the hour you have, or quarantine with an owner and a date
5. 05Reviewconfirm the fix with repeated runs, then check the quarantine list weekly

When you open the trace from a first-retry failure, you are looking for one of two shapes. If the action fired before the thing it needed was ready, the DOM snapshot at the moment of failure shows the element missing or still in its previous state, and the network panel shows the response that had not arrived yet. That is a test-timing problem you fix with a proper assertion. If the DOM and the network both look correct and the app still produced the wrong result, you are looking at the product, and the trace is now a bug report. Reading the trace is what lets you classify in minutes instead of guessing, and it is the reason trace: 'on-first-retry' earns its place in the config.

Reproduction is the step people skip and the one that saves the day. A fix you cannot prove is a guess. Playwright’s --repeat-each runs a test repeatedly in one command, so npx playwright test the-spec --repeat-each=50 gives you fifty attempts to make the failure show up before you touch anything, and then fifty more to prove the fix holds.

The usual suspects, with symptom and fix

Most flakiness traces back to a short list of causes, and the fixes are concrete enough to put in a table your team can work from.

Root cause	Symptom	Fix
Fixed sleeps (waitForTimeout)	passes on fast runners, fails on slow ones	a web-first assertion that retries (toHaveText, toBeVisible)
Shared test data	passes alone, fails in parallel	unique data per test, created in setup
Order dependence	passes in one order, fails in another	no cross-test state; each test sets up and tears down its own
Unsettled animations	assertion fires mid-transition	disable animations in the test environment
Real third-party calls	fails when the dependency has a bad second	mock at the boundary so you test your code
Shared login across workers	random auth failures under parallelism	one account per parallel worker
Wall-clock dates	passes by day, fails on the overnight build	pin time with the clock API and set the zone

Fixed sleeps are the single biggest one. A waitForTimeout(2000) is a bet that two seconds is always enough and never too much, and it loses both ways. Replace it with an assertion that waits for the actual condition.

// flaky: a race against a hard-coded clock
await page.click('#save');
await page.waitForTimeout(2000);
expect(await page.textContent('.status')).toBe('Saved');

// stable: the assertion retries until the state is real or it times out
await page.click('#save');
await expect(page.locator('.status')).toHaveText('Saved');

Playwright’s web-first assertions such as toHaveText and toBeVisible retry on their own until the condition holds or the timeout expires. You almost never need an explicit wait, and every one you delete is a race you stop running in your own test.

The Riyadh midnight case

Clocks and time zones deserve their own section because they catch teams in a way that looks like infrastructure flakiness and is not. A test that builds a date on the runner in UTC and asserts against an app that renders in Riyadh time, UTC+3, will flip around midnight. It passes all afternoon, when the UTC date and the Riyadh date agree, and fails on the overnight build in the three-hour window where they do not. The failure correlates with the time the pipeline happened to run, which is why it reads as random until someone notices every failure is between 9pm and midnight UTC.

The fix is to stop reading the wall clock and pin both the instant and the zone inside the test. Playwright’s clock API controls time for the page under test, so you can freeze the instant and make the run identical at two in the afternoon and two in the morning. The clock alone is not enough, though: the browser still renders local time in the runner’s zone, so set the context’s timezoneId as well, or a UTC runner will show the 9th no matter what instant you freeze.

import { test, expect } from '@playwright/test';

// Render the page in Riyadh time regardless of where the runner lives.
test.use({ timezoneId: 'Asia/Riyadh' });

test('statement shows the Riyadh business day', async ({ page }) => {
  // Freeze time so the test is identical on the afternoon and the overnight build.
  // Asia/Riyadh is UTC+3, so this instant is 10 March in Riyadh and 9 March in UTC.
  await page.clock.install({ time: new Date('2026-03-10T00:30:00+03:00') });
  await page.goto('/statements');
  await expect(page.getByTestId('business-date')).toHaveText('2026-03-10');
});

Be explicit about the zone in the configuration itself rather than in a comment, and set it the same way everywhere the test fleet runs. A suite that assumes the runner’s zone is portable until the day you move CI to a region that is not on UTC, and then it is a two-day investigation into “random” overnight failures that were never random.

Two more sources of time-and-locale flakiness are specific to the products we test here. Bilingual apps that render right-to-left in Arabic and left-to-right in English can flake on layout-sensitive assertions when the test does not pin the locale, because the element you are asserting on shifts position depending on direction. And apps that show Hijri dates next to Gregorian ones carry a second calendar whose conversion can differ by a day between libraries and calendar variants, Umm al-Qura among them, so an assertion that hard-codes a converted date breaks the day someone upgrades the date library. Pin the locale and the calendar in the fixture the same way you pin the clock, and both classes of failure stop looking random.

Quarantine with a name and a date

When a test is confirmed flaky and you cannot fix it in the hour you have, the wrong moves are deleting it or wrapping the suite in more retries. Deleting it loses the coverage silently. More retries spreads the blindness to every other test. Quarantine instead: move the test out of the blocking set so it stops failing the build, but keep it running and keep it visible.

• Every quarantined test has a named owner, a person and not a team.
• Every quarantined test has an expiry date by which it is fixed or deleted on purpose.
• The quarantine list is reviewed weekly, in the same meeting where you look at the pipeline.
• A test that has sat quarantined for a month is a decision someone makes out loud, not a default.
• Quarantined tests still run and still report, so the coverage loss is never silent.

A quarantine list with no dates becomes a graveyard that grows until nobody trusts the suite at all. The dates are what keep it a holding pen rather than a dumping ground.

Parallelism is where latent flakiness becomes visible, so expect that adding workers or sharding across CI machines will surface tests that passed for years. A suite that was reliable at one worker and flakes at eight almost always has hidden shared state: a global fixture, a single seeded user, a database row that two tests both write. Resist the urge to drop back to one worker to make it green, because that shared state is a real defect in the suite that will bite again under production-like concurrency. Isolate the state instead, and keep the parallelism that makes the suite fast enough to run on every merge.

Telling a product race from a test race

Here is the case that justifies the whole discipline. Sometimes the test is written correctly and the environment is fine, and the product still fails intermittently. The product has a real race: a request that can arrive before the record it depends on is committed, a double-submit the UI does not guard, a cache read that can beat its own write. Under the scheduling of a loaded CI runner, that race shows up as a flaky test. Under the scheduling of a loaded production server on the day salaries land, it shows up as a customer seeing the wrong balance.

The way to tell the two apart is to fix the test first and see if the flakiness survives. Replace the sleeps with proper web-first assertions. Give the test its own data. Pin the clock. Mock the third parties. If a correctly written, fully isolated test still fails intermittently against the product, you are no longer looking at a test problem. You have a product defect, and the right move is to open a bug and point an engineer at the server-side timing, not to quarantine the test as “flaky.” Quarantining it there would be the exact mistake: silencing the one warning you have that the product has a race customers will hit.

If you have already retried that kind of test green, you have shipped the bug and muted the alarm. That is why “just add retries” deserves more suspicion than it usually gets. Whatever the config calls the setting, it is a decision to not know which of the three causes you have.

Tracking flakiness over time

Per-run flakiness is noise; the trend is the signal. The JSON reporter writes a status for every test, one of expected, unexpected, flaky or skipped, so you can aggregate across many CI runs and rank tests by how often they flake. A small script over the stored reports turns the gut-feel list into a real backlog.

// tally-flakes.js: aggregate flaky outcomes from one Playwright JSON report
import { readFileSync } from 'node:fs';

const report = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const flaky = [];

function walk(suite) {
  for (const spec of suite.specs ?? []) {
    for (const t of spec.tests ?? []) {
      if (t.status === 'flaky') flaky.push(`${spec.file} :: ${spec.title}`);
    }
  }
  for (const child of suite.suites ?? []) walk(child);
}
report.suites.forEach(walk);
console.log(flaky);

Run that against each build’s results.json, append the output to a store keyed by test, and compute a flake rate per test over a rolling window of the last few weeks, flaky runs divided by total runs. Rank by that rate. The test at the top of the list, flaking one run in four over a month, is where your time goes first, and the long tail that flaked once and never again can wait. The ranked list is almost always shorter and more honest than the one living in people’s heads, and it gives the weekly review something to argue about that is backed by counts instead of memory.

What you do with the ranked list matters as much as having it. Set a line: any test above a flake rate you choose, say five percent over the trailing month, is fixed or quarantined this week and not discussed again next week. Tests below the line stay on the board but do not block work. Without a line, the list becomes a standing agenda item that generates discussion and no decisions, which is how a team ends up with the same flaky tests a year later and a document proving they talked about them the whole time.

Turn on the flaky label in your CI report and let it run for a week before you change anything else. The list it produces is your actual starting point.

Sources

1. Playwright: retries, Playwright, 2025.
2. Playwright: assertions, Playwright, 2025.
3. Playwright: trace viewer, Playwright, 2025.
4. Playwright: clock, Playwright, 2025.
5. Playwright: command line, Playwright, 2025.