DemoAcme Cloud · Retro Desktop template

Use this template

Acme Cloud

Live status for the Acme Cloud API, dashboard and data pipeline. Updates land here first, or read the incident handbook.

Fatal exception in production (Major outage)

System Monitor: Major outage
ProgramStatusHistory (90 days)Uptime (90 days)CPU (response time)Trend
Core platform
APICrashed (Down)Major incident99.78%, response 190 ms190 ms (fast)worsening
DashboardCrashed (Down)Major incident99.78%, response 334 ms334 ms (ok)worsening
AuthResponding (Operational)99.84%, response 157 ms157 ms (fast)steady
Data
Postgres primaryResponding (Operational)99.84%, response 41 ms41 ms (fast)steady
BackupsResponding (Operational)96.73% reliability—steady
Edge
CDNResponding (Operational)99.84%, response 60 ms60 ms (fast)steady
WebhooksResponding (Operational)99.84%, response 273 ms273 ms (fast)steady
Programs: 7Responding: 5

Messages

API and dashboard unavailable

InvestigatingMajor impact · Started

Affects API, Dashboard

Requests to the API and dashboard are failing. We are investigating and will post an update within 30 minutes.

  1. We are rolling back a load balancer change made at the start of the outage.

  2. Every region is affected. Background jobs are paused to protect data.

Event viewer

Event viewer
LevelTimeSourceEvent
IncidentStatus pageAPI and dashboard unavailable [Investigating]
2 automatic events
  • Dashboard went down
  • API went down
IncidentStatus pageElevated error rates [Resolved · 50 min]
2 automatic events
  • API recovered
  • API went down
Details

A bad deploy raised error rates for about 45 minutes. We rolled it back and added a canary stage.

  1. Error rates are back to normal. This incident is resolved.

  2. We found a faulty deploy and are rolling it back.