Back to BitSentry Desktop
Diagnostic Runbook Read-only diagnosis ~5 min

Diagnose API 500s
after a deploy.

A full diagnostic runbook walkthrough. Real commands, real output, AI interpretation at each step, evidence-backed root cause. This is the canonical incident BitSentry Desktop was built for.

Incoming alert

Checkout API 500 rate spike

Source Sentry
Started at 2026-04-12 02:14 UTC
Error count 1,247 errors in 4 minutes
Endpoint POST /api/v2/checkout/confirm
Recent deploy abc123 — payment-service v3.18.0 @ 2026-04-12 02:11 UTC (3 min before spike)

The runbook

5 steps. ~5 minutes. Read-only.

Every step is a deterministic command you wrote. The AI reads the output and tells you what it means. Nothing mutates production.

1

Check recent deploys

read-only

Command 

kubectl rollout history deployment/payment-service -n prod | head -10

Output

deployment.apps/payment-service
REVISION  CHANGE-CAUSE
17        kubectl set image deployment/payment-service ...:v3.17.4 (2026-04-10 14:22)
18        kubectl set image deployment/payment-service ...:v3.18.0 (2026-04-12 02:11)

AI interprets

A new revision (v3.18.0) was rolled out at 02:11 UTC — exactly 3 minutes before the alert fired. Previous revision was stable for 36 hours. Strong correlation with deploy.

2

Tail service logs

read-only

Command 

kubectl logs -n prod -l app=payment-service --since=10m --tail=200 | grep -iE 'error|exception|timeout'

Output

[02:14:11] ERROR HikariPool-1 - Connection is not available, request timed out after 30000ms
[02:14:11] ERROR HikariPool-1 - Connection is not available, request timed out after 30000ms
[02:14:12] ERROR PaymentController - Failed to acquire DB connection within 30s
[02:14:12] ERROR HikariPool-1 - Connection is not available, request timed out after 30000ms
... (truncated, 1,184 similar errors)

AI interprets

Every error is a HikariCP connection-pool timeout. The application can't get a DB connection for 30 seconds. This is not a code bug — it's a resource exhaustion symptom.

3

Inspect DB connection pool

read-only

Command 

ssh ops@db-prod-1 'psql -U postgres -d payments -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"'

Output

   state          | count
------------------+-------
 active           |    98
 idle             |     2
 idle in transaction |   100
(3 rows)

max_connections = 200

AI interprets

200 of 200 connections are in use. 100 are stuck in 'idle in transaction' — they've started a transaction but never committed. Pool is exhausted because connections aren't being released.

4

Check what changed in v3.18.0

read-only

Command 

git -C /repos/payment-service log --oneline v3.17.4..v3.18.0 -- 'src/payment/'

Output

abc123 feat(payment): batch confirmation flow for high-volume merchants
def456 refactor(db): switch to manual transaction control for confirmations
ghi789 chore: bump dependencies

AI interprets

Commit def456 'switch to manual transaction control' is the smoking gun. The new code begins transactions manually but likely has a path that returns early without committing or rolling back — leaving connections in 'idle in transaction' state.

5

Confirm the leak path

read-only

Command 

ssh ops@db-prod-1 'psql -U postgres -d payments -c "SELECT pid, age(clock_timestamp(), xact_start), query FROM pg_stat_activity WHERE state = ''idle in transaction'' ORDER BY xact_start LIMIT 3;"'

Output

  pid  |      age        |              query
-------+-----------------+----------------------------------
 88421 | 00:11:32.4      | UPDATE payments SET status='confirmed' ...
 88433 | 00:11:18.1      | UPDATE payments SET status='confirmed' ...
 88447 | 00:10:54.7      | UPDATE payments SET status='confirmed' ...

AI interprets

All leaked connections are mid-UPDATE on the payments table. They've been open for 10+ minutes. This matches the new manual-transaction path in def456 — the batch confirmation flow opens a transaction per row but error-paths don't roll back.

Root cause hypothesis

DB connection pool exhausted by leaked transactions in deploy abc123.

The batch confirmation flow introduced in v3.18.0 (commit def456) begins transactions manually but lacks COMMIT/ROLLBACK on the error path. Each errored row leaks one connection. Under normal traffic this exhausts the 200-connection pool in ~3 minutes — which matches the observed alert lag.

Evidence

Time correlation: Deploy at 02:11, spike at 02:14 (3-min lag matches pod-rollout window)
Error signature: 100% HikariCP pool timeouts — not application logic errors
DB state: 200/200 connections used, 100 stuck in 'idle in transaction'
Suspect commit: def456 'switch to manual transaction control' in v3.18.0
Leak pattern: Leaked connections all running UPDATE on payments — matches new batch flow

Next step

Three remediations. Pick one — humans decide.

The runbook stops here. It presents remediation options instead of applying them for you.

Rollback (fastest, lowest risk)

Mutating option

kubectl rollout undo deployment/payment-service -n prod

Hotfix forward (commit COMMIT/ROLLBACK on error paths in def456)

Mutating option

Patch the batch confirmation code, build, deploy. Higher risk under load.

Temporarily raise max_connections

Mutating option

Buys time but doesn't fix the leak. Not recommended.

Build this runbook in BitSentry Desktop.

Chain shell commands, HTTP calls, and AI steps. Pass output between them. Run against your own servers. AI calls go to your provider.

Try BitSentry Desktop Free See it for teams →