Skip to content

Measuring Delivery

We measure the health of our delivery using the four DORA metrics, the industry-standard measures of software delivery performance:

  • Deployment frequency: how often we ship to production
  • Lead time for changes: how long a change takes to get from a commit to running in production
  • Change failure rate: what proportion of production changes cause a defect or incident
  • Time to restore service: how quickly we put things right when production breaks

These metrics describe the delivery system, not the people in it. We use them to spot trends, have better conversations with clients, and decide where to improve. We never use them to rank individuals or set league tables between teams, because the fastest way to ruin a useful measure is to turn it into a target.

Metrics are a by-product of good delivery

We don't ask teams to fill in spreadsheets or report their own numbers. The metrics are collected automatically from the delivery platform, which means they're only as trustworthy as the everyday working habits they're read from.

Four conventions make the four metrics measurable. None of them is extra work. They're the habits of a well-run delivery anyway, and the metrics fall out for free.

1. Production deployments happen through a pipeline environment

Deployment frequency needs every production release to leave a record. In Azure DevOps, that means the deployment job in the pipeline declares an environment: whose name identifies production (Production, or MyService-PROD for multi-service projects). It's one line of YAML, and it gives every release a timestamped, auditable record of what went where.

Deploying by hand, or through a pipeline that never touches an environment, makes releases invisible. If a step is important enough to do, it's important enough to leave a trace.

2. Deployments trace back to commits

Lead time is the gap between writing a change and customers benefiting from it. When the pipeline builds and deploys straight from the repository, every deployment already knows exactly which commit it shipped, so this convention comes free with the first one. What breaks the chain is deploying artefacts that have lost their link to source control.

3. Defects found in production are tagged

Change failure rate depends on knowing which defects customers actually saw. When a defect is raised against something already live, tag the work item Production at the point it's created. One tag, applied consistently, is the whole convention. Defects found in testing or UAT need no tag; they're the process working as intended.

4. Defects are resolved, not abandoned

Time to restore service is only measurable if defects move through the workflow. When a fix ships, the work item moves to Resolved or Closed, and that transition records when service was put right. A board where defects sit in New forever tells us nothing, and usually tells the client nothing either.

Adopting this on an existing engagement

Going forward, the conventions cost almost nothing. Historical data needs a little care:

  • Backfilling the Production tag onto old defects is safe, and improves the historical picture.
  • Bulk-closing old work items is not. Closing a defect stamps it as resolved today, however long ago the fix actually shipped, which corrupts the time-to-restore measure for months.

So tidy the board by all means, but agree the date the metrics start counting from first, and clean up before it.

Starting from a clean date

When a team adopts these conventions partway through an engagement, we measure from the adoption date rather than pretending the history is complete. Honest partial data beats impressive-looking noise.

Other platforms

The conventions transfer to whatever platform an engagement uses. On GitHub, environments and the deployments API play the pipeline environment's role, and labels stand in for tags. The principles are the same everywhere: production releases leave a record, defects say where they were found, and the board reflects reality.