The interesting thing about incident response costs is that nobody tracks them. Your monitoring bill shows up in an invoice. Your engineers’ time debugging at 2am doesn’t.
But it adds up. And once you see the numbers, the case for automation gets obvious.
Where the time goes
A typical incident has three phases: detection, investigation, and resolution. Detection is usually automated (your monitoring fires an alert). Resolution is usually fast (a config change, a restart, a rollback). Investigation is where engineers lose their evenings.
The average investigation takes 30-45 minutes. That’s SSH into the server, pull logs, grep for errors, check upstream services, cross-reference timestamps, maybe check deploy history. The fix itself takes 5 minutes once you know what’s wrong.
If your team handles 15-20 incidents a month (not unusual for a 50-person engineering org), that’s 7.5 to 15 hours of investigation time per month. Just finding root cause. Not fixing anything.
The costs nobody talks about
Engineer time at 2am is expensive
An engineer making $150k/year costs roughly $75/hour during business hours. At 2am, after being woken up, their productivity is maybe half that. The real cost isn’t the hourly rate. It’s the next day: they’re tired, they shipped nothing, and you’re paying a full salary for someone running at 60%.
If you page someone 4 times a month and each incident takes 45 minutes of investigation plus the recovery time the next day, you’re looking at $3,000-5,000/month in lost productivity per on-call engineer. Most of that is the investigation phase.
Knowledge concentration is a liability
When only 2-3 engineers know how to diagnose a specific failure mode, you have a bus factor problem. Those engineers get paged disproportionately. They burn out faster. When they leave, the institutional knowledge goes with them and the next person takes even longer per incident.
This doesn’t show up in any dashboard, but it’s the most expensive ops cost most teams have.
MTTR compounds
Slow investigation means longer outages. Longer outages mean more customer impact. More customer impact means support tickets, churn risk, and the occasional postmortem meeting that eats half a day from 8 people. A 30-minute incident that could have been 10 minutes costs far more than the 20-minute difference.
Where automation pays off
Not everything is worth automating. The highest-ROI targets for ops automation:
The investigation phase is the 30-45 minute bottleneck. If you can cut that from 40 minutes to 10, you save 30 minutes per incident. At 15 incidents a month, that’s 7.5 hours back. At $75/hour, that’s $562/month per engineer. For a team of 5 on-call engineers, $2,800/month or $33,750/year.
Routine ops work (patching, restarts, user changes) is the next target. These aren’t hard but they take time and they queue up. If 20% of your ops team’s week goes to routine work that could run automatically with the right guardrails, that’s a full day per person per week.
Then there’s runbook execution. If your team runs the same diagnostic checklist every time a specific alert fires, that checklist should be a runbook that executes itself and interprets the output. Building the runbook takes 30 minutes. Running it manually takes 30 minutes every time. Running it automatically takes zero.
What doesn’t pay off
Some things aren’t worth automating yet:
- Novel incidents that haven’t happened before (you need a human the first time)
- Cross-team coordination during major outages (still a people problem)
- Architecture decisions after a postmortem (automation won’t restructure your services)
Automation handles the repeatable parts. Humans handle the judgment calls.
The math for BitSentry
We sell two products with very different price points. SuperTerminal Pro is a $149 lifetime deal for the desktop runbook executor. Dashboard is $7,200/year flat for the continuous ingestion plus 24/7 background worker.
If SuperTerminal saves one on-call engineer 30 minutes per incident across 15 incidents a month, that’s 7.5 hours/month, which at $75/hour is $6,750/year in recovered time from a single engineer. At $149 once, the lifetime payback is roughly two weeks of incident savings.
With a 5-person on-call rotation, the recovered time is closer to $33,750/year. Dashboard’s 24/7 background worker catches issues before they page anyone, which means some of those incidents never interrupt someone’s sleep at all. At $7,200/year, the math pays back inside the first quarter for most mid-sized on-call teams.
Try it
SuperTerminal is free while in beta. Start with one engineer, one runbook, one incident. See how long the investigation takes compared to manual SSH. Get started here.