Mean time to detect down from forty minutes to under three

Meridian run a national freight operation with a control room that is staffed around the clock. Their monitoring was a hosted service billed per host, and the bill had grown faster than the estate — but the more immediate problem was that it was not telling them anything useful.

The control room found out about most incidents from drivers phoning in. Measured over the preceding quarter, mean time to detect was just over forty minutes.

What we did

The estate review found 214 hosts, of which 60 were emitting no telemetry at all and another 40 were emitting metrics nobody had ever looked at. That distinction shaped the whole engagement.

We deployed Grafana with Prometheus and Loki into their own infrastructure, with retention sized to their actual data volume rather than to a vendor’s pricing tier.

Dashboards were built in the control room, with the people who would use them at three in the morning. Two of the three dashboards originally specified were abandoned during that process because nobody could say what they would do differently on seeing them.

Alert design was the most contested part and the most valuable. We removed roughly half the alerts that had been proposed, on the grounds that an alert nobody acts on trains people to ignore all of them. What remained has an owner, a runbook and a threshold someone defended.

The outcome

Mean time to detect is now under three minutes, measured the same way as the baseline. The control room finds out before the drivers do.

The £71,000 annual SaaS bill was removed. Infrastructure and support cost a fraction of it, and the saving was not the point — the detection time was.

Meridian’s own team have since built eleven further dashboards without our involvement, which is the outcome we were actually aiming for.