DORA metrics for quality leaders

Most QA status reports lead with the wrong numbers. Tests automated this sprint, percentage of coverage, pass rate on the last run. A CTO reads those and learns almost nothing about whether the product ships faster, breaks less, or recovers quicker when it does break. The numbers go up and to the right, the release still slips, and the meeting ends with everyone slightly less sure what quality engineering is for.

The fix is to report the same delivery metrics the rest of engineering already reports, and to show how your work moves them. DORA’s software delivery performance model uses five metrics as of its 2024 report, and every one of them is pushed or dragged by things a quality team controls directly: how fast the suite runs, how often it lies, how long a test environment takes to get, and what the release gate actually blocks.

Why these and not coverage

Coverage and test counts measure activity. They answer “how much testing did we do,” which is a fair question in isolation and a misleading one in a leadership report, because activity has no fixed relationship to outcome. A team can push coverage from sixty to eighty percent and ship exactly as slowly, with exactly as many rollbacks, as before. The extra tests might be on code that never fails. Some of them might be flaky and actively making the pipeline worse. Nothing in the coverage number tells you which.

DORA’s metrics measure outcomes instead. They come out of a decade of research into what separates high-performing software teams from the rest, and they describe the delivery system end to end rather than one activity inside it. When you report change lead time dropping and change fail rate holding steady, you are telling leadership that the product reaches customers faster without breaking more often, which is the actual goal that testing serves. The reframe also changes how your team is seen. A QA function that reports coverage looks like a cost center counting its own output. One that reports delivery metrics looks like part of the system that ships the product.

The five metrics and the two groups

DORA sorts the metrics into two groups, throughput and instability, and the grouping is the point. Throughput tells you how quickly change moves through the system, including how quickly you get back on your feet when a change goes wrong. Instability tells you how often deployments go wrong in the first place. A report that shows only throughput is dangerous, because the fastest way to deploy more often is to test less, and the instability numbers are what stop that from looking like a win.

Throughput covers three metrics: change lead time, the time from a commit to that commit running in production; deployment frequency, how often you release to production; and failed deployment recovery time, how long it takes to restore service after a deployment fails and needs urgent intervention. Instability covers two: change fail rate, the share of deployments that need urgent intervention such as a rollback or a hotfix, and deployment rework rate, the share of deployments that are unplanned and happen because of an incident in production. Rework rate is a direct read on how much of your release activity is cleaning up after itself.

One piece of history matters for how you measure the recovery number. In 2023 DORA replaced “time to restore service” with “failed deployment recovery time” so the measure focuses on failures caused by changes rather than outages from external events like a cloud provider going down. When you compute it, count the clock only on incidents a deployment caused. A regional network outage that had nothing to do with your release does not belong in this number, and including it makes your delivery system look worse than it is.

How quality work moves each metric

This is the part that makes the metrics yours to report. Every one of them has a lever a quality team holds.

Change lead time is dominated, in most teams we see, by waiting rather than working. The commit is done in an hour; it then sits for most of a day waiting for a CI run that is slow or has to be retried because the suite is flaky, waiting for a shared test environment that another team is using, and waiting for a manual release approval. A flaky suite is a lead-time tax paid on every pull request, because a retry is dead time and a false failure sends an engineer back to investigate code that was fine. Cutting suite runtime and killing flakiness shows up directly as lead time falling.

Deployment frequency is throttled by whatever makes a release feel risky or expensive. If every deploy needs a manual regression pass that takes two days, you cannot deploy daily, full stop. Reliable automated regression, fast enough to run on every merge, is what lets a team raise deployment frequency without raising fear. Change fail rate is the gate’s job. A release gate that runs the right checks and blocks on real failures keeps bad changes out of production and holds the fail rate down; a gate full of flaky tests either gets ignored or gets bypassed, and both outcomes let failures through. Failed deployment recovery time leans on things like health checks, smoke tests that run against production right after a deploy, and the rollback mechanism being tested rather than assumed. Deployment rework rate falls when the testing before release catches the user-facing problems that would otherwise force an unplanned fix deploy the next morning.

Computing each from data you already have

None of this needs a survey. The data lives in your version control, your deployment pipeline, and your incident tracker. The table is the formula sheet, including the mistake that most often corrupts each number.

Metric	Data source	Formula (report the median)	Common mistake
Change lead time	Git plus deployment logs	Time from a commit merged to that commit live in production	Starting the clock at ticket creation or PR open, which mixes in backlog time
Deployment frequency	Release pipeline events	Count of production deployments per day or week	Counting builds or staging deploys, not production releases
Change fail rate	Deployment logs plus incident and rollback records	Deployments causing a failure needing remediation, divided by all deployments	Counting every incident, including ones no deployment caused
Failed deployment recovery time	Incident tracker plus deployment logs	Time from a change-caused failure starting to service restored	Including outages from external events, which the 2023 rename exists to exclude
Deployment rework rate	Deployment logs tagged by purpose	Unplanned deployments made because of a production incident, divided by all deployments	Conflating it with change fail rate; rework counts the fix deploys, not the failures

Lead time carries a measurement trap worth naming on its own. The clean definition is commit to production, but the word commit is ambiguous when a branch lives for two weeks before it merges. Measure from the commit that reaches the main line, not the first keystroke on the branch, because the time a branch spends in development is a planning signal rather than a delivery one. Folding it in makes the metric jump every time someone keeps a long-lived branch, and you lose the ability to tell a slow pipeline from a slow backlog. If you want the development time too, track it as its own number.

If you have to phase the work, start with the two throughput metrics, because they come almost for free from data you already keep. Deployment frequency is a count of production releases, which your pipeline already emits. Lead time is two timestamps you already have, the merge and the deploy. The instability metrics need a little more plumbing, because they depend on reliably linking an incident back to the deployment that caused it, which is the main reason tagging deploys by purpose at the source pays off so quickly.

Two practical notes. Report the median, not the mean, for the time-based metrics, because one bad week of a stuck release will drag a mean into fiction. And tag deployments at the source. If your pipeline writes a deploy event with a field for purpose (planned release, hotfix, rollback), the rework rate and change fail rate compute themselves from pipeline data, and you avoid the monthly archaeology of asking people which of last month’s deploys were cleanups.

A worked example

Take an illustrative team to make the numbers concrete. The figures below are invented to show the shape of the analysis, not measurements of any real service.

This team deploys five times a week, with a median change lead time of eighteen hours, a change fail rate of twelve percent, a median failed deployment recovery time of two and a half hours, and a deployment rework rate of nine percent. The throughput looks fine. The instability is the story: better than one deploy in eight causes a failure, and nearly one deploy in ten is an unplanned fix.

Breaking down where the eighteen hours of lead time actually go points at the fix.

CI run and flaky reruns7h

Waiting for a shared test environment6h

Manual release approval window4h

Deploy and verify1h

Most of the lead time is waiting, and two of the three waits are quality problems. Seven hours lost to CI reruns says the suite is flaky and slow. Six hours waiting for a shared environment says test environments are a bottleneck. If this team spent a quarter killing flakiness and giving each service an ephemeral environment, lead time could fall toward six or seven hours without touching the approval step at all. The change fail rate of twelve percent, meanwhile, points at the gate letting failures through, which is a different fix: find what class of bug keeps reaching production and add a check that catches it before the merge. Reporting all five together is what makes that diagnosis possible. The lead-time number alone would have sent everyone to argue about the approval window.

Pitfalls that turn the metrics into theater

The metrics are easy to corrupt, and a few failure modes show up again and again.

Do not compare teams against each other. A payments service with a slow, deliberate release process and a marketing site that deploys thirty times a day are measuring different realities, and ranking them teaches the payments team to cut corners to win a chart. Use the metrics to track one team’s trend against its own past.

Do not let them drive individual performance reviews. The moment deployment frequency shows up in someone’s appraisal, they split one real deploy into five trivial ones, and the number climbs while nothing improves. The same incentive makes people argue that a real incident “was not really a change failure” to protect their change fail rate. This is Goodhart’s law in its plainest form: once a number decides someone’s pay, it stops telling you about the system and starts telling you about the person’s ingenuity.

The cleanest way to show a quality team’s contribution is to overlay suite reliability on the DORA trend. When flaky-test rate falls and median CI duration falls, watch change lead time follow within a sprint or two, and point at that correlation in the review. It is the most direct evidence you will get that the unglamorous work of fixing tests and speeding up the pipeline is delivery work rather than housekeeping, and it reframes the suite from a cost the team tolerates into a lever the team pulls.

Do not report throughput without instability. Speed with no stability signal is a team shipping faster and breaking more, presented as success. The pair exists precisely so one cannot be improved by secretly sacrificing the other. If you report lead time dropping, report change fail rate in the same breath, so everyone can see the drop did not come from skipping the tests.

A first quarter

Rolling this out well takes a quarter, and the mistake to avoid is setting targets in week one. You do not yet know your own baseline, and a target set on a guess becomes the thing people game.

1. 01Weeks 1 to 2Instrument deploys and incidents; write down each definition so everyone counts the same thing
2. 02Weeks 3 to 6Report all five metrics together, as a baseline, with no targets attached
3. 03Weeks 7 to 10Find the biggest constraint, usually flaky CI or slow environments
4. 04Weeks 11 to 13Fix that one constraint, show the metric move, and review the pair as a team

The Saudi context sharpens why the instability pair matters. Banking and government platforms here often sit under change freezes around salary-day peaks and around regulatory reporting, which compresses deployments into the windows on either side of the freeze. A team that reports only deployment frequency looks worse during a freeze through no fault of its own, while a team that reports the pair can show it held change fail rate low through the freeze and shipped safely the moment it lifted. The freeze also makes recovery time concrete: during a freeze even a rollback can need an emergency change approval, so a change-caused failure that slipped in just before it tends to take longer to fix than the same failure on an ordinary Tuesday. That is the strongest argument for spending the quality effort before a freeze rather than after it.

The written definitions in the first two weeks matter more than they sound. Most disagreement about DORA numbers is really disagreement about what counts as a deployment or a failure, and settling that on paper before anyone reports a number prevents a quarter of arguments later.

• A deployment is defined, in writing, as a release to production (not staging, not a build).
• A change failure is defined as one needing remediation such as a rollback or hotfix.
• Failed deployment recovery time counts only change-caused failures, not external outages.
• Deploy events are tagged by purpose so rework rate and fail rate compute from pipeline data.
• Metrics are reported as a team trend, never as a league table or an individual score.

The first time you show leadership that lead time fell by a third because the team stopped retrying a flaky suite, and change fail rate held steady while it happened, the conversation about what quality engineering is worth tends to end.

Sources

1. DORA: software delivery performance metrics, DORA, 2024.
2. DORA: history of the metrics, DORA, 2024.