DemoAcme Cloud · Blueprint template

Use this template

Acme Cloud

Major outage

Note API and dashboard unavailable

Live status for the Acme Cloud API, dashboard and data pipeline. Updates land here first, or read the incident handbook.

Title
Acme Cloud
Dwg no.
ACME-CLOUD
Rev
4
Drawn by
Crontap
Sheet
1 of 1
Scale
90 D
Status
Fault
Date

Revisions

RevDescriptionDate
2Last released issue
3Fault on API
4Fault on Dashboard

Incidents

  1. Note 1.

    Fault on API and Dashboard. See note 2.

  2. Note 2.

    Major impact · Investigating

    API and dashboard unavailable

    Started · Affects U1 API and U2 Dashboard

    Requests to the API and dashboard are failing. We are investigating and will post an update within 30 minutes.

    1. We are rolling back a load balancer change made at the start of the outage.

    Earlier updates (1)

    1. Every region is affected. Background jobs are paused to protect data.

Subsystem A · Core platform

  • U1Down

    API

    Uptime 99.78 %worsening

    Lat 190 ms

    See note 2

  • U2Down

    Dashboard

    Uptime 99.78 %worsening

    Lat 334 ms

    See note 2

  • U3Operational

    Auth

    Uptime 99.84 %

    Lat 157 ms

Subsystem B · Data

  • U4Operational

    Postgres primary

    Uptime 99.84 %

    Lat 41 ms

  • H1Operational

    Backups

    Uptime 96.74 %

    Last ping

Subsystem C · Edge

  • U5Operational

    CDN

    Uptime 99.84 %

    Lat 60 ms

  • U6Operational

    Webhooks

    Uptime 99.84 %

    Lat 273 ms

History

RevDateDescription
2
API and dashboard unavailable · Major impact · open
2 automatic events
  • Dashboard went down
  • API went down
1
Elevated error rates · Major impact · 50 min
2 automatic events
  • API recovered
  • API went down
Details

A bad deploy raised error rates for about 45 minutes. We rolled it back and added a canary stage.

  1. Error rates are back to normal. This incident is resolved.

  2. We found a faulty deploy and are rolling it back.

Powered by Crontap