Blameless Postmortem Generator

# Postmortem: Checkout outage from a missing migration

**Date:** 2026-06-24  
**Authors:** Priya Nair, Marco Diaz  
**Status:** Blameless review

## Summary

A deploy of checkout-service shipped code that depended on a database column that had not been migrated yet. Checkout failed for about 26 minutes until we rolled back.

## Impact

Roughly 26 minutes of degraded checkout. About 4,100 checkout attempts failed. No data was lost or corrupted.

## Detection

An automated alert on the /checkout error rate fired at 14:09 and paged the on-call engineer.

## Timeline

- **14:02** - Deploy of checkout-service v3.4.0 rolled out to production.
- **14:09** - Error rate on /checkout climbs above 20%. First alert fires.
- **14:13** - On-call engineer acknowledges the page and opens an incident.
- **14:21** - Root cause narrowed to a missing database migration.
- **14:28** - checkout-service rolled back to v3.3.2. Error rate recovers.
- **14:35** - Checkout success rate confirmed back to baseline. Incident resolved.

## Root cause

checkout-service v3.4.0 read a column that a separate migration was meant to create. The migration was never run because it sat in a branch that merged after the deploy.

## Contributing factors

The deploy pipeline does not check for pending migrations. The canary stage had no end to end checkout probe, so the failure only surfaced at full rollout.

## Resolution

We rolled checkout-service back to v3.3.2, which does not depend on the new column. Error rates recovered within two minutes of the rollback.

## Action items

- [ ] **Priya (Platform)** - Add a migration gate to the deploy pipeline.
- [ ] **Marco (Checkout)** - Add a synthetic checkout probe to the canary stage.
- [ ] **On-call lead** - Document the rollback runbook for checkout-service.