DemoAcme Cloud · Terminal template

Use this template

Acme Cloud

Live status for the Acme Cloud API, dashboard and data pipeline. Updates land here first, or read the incident handbook.

uptime

7 units, load critical

status --all

*** SYSTEM FAULT: 2 UNITS FAILED ***

7 units loaded, 2 failed

fault: API and dashboard unavailable

Major outage

Incidents

Incident: API and dashboard unavailable

title:
API and dashboard unavailable
impact:
major impact
status:
investigating
started:
(15 min)
units:
api.service, dashboard.service

Requests to the API and dashboard are failing. We are investigating and will post an update within 30 minutes.

  1. []

    We are rolling back a load balancer change made at the start of the outage.

  2. []

    Every region is affected. Background jobs are paused to protect data.

Core platform

  • api.service - API

    Active: failed (Result: unreachable) (Down) since

    Uptime: 99.78% (90d) Latency: 190ms

    History: ███▓██████████████████████████████████▓███████████████████████████░██████████████████████░█▓█████████████████▓█████████████░██████████░

    Incident: 797fb81 (major)

  • dashboard.service - Dashboard

    Active: failed (Result: unreachable) (Down) since

    Uptime: 99.78% (90d) Latency: 334ms

    History: ███████████████████████████████▓█████████████▓██░████████████████████████████████████████░███████████████▓██████▓█░███████████████████░

    Incident: 797fb81 (major)

  • auth.service - Auth

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 157ms

    History: ███░█████████▓███████████████▓█████████████████████████████████████████████████████████████░████▓███████▓██████████████████████████████

Data

  • postgres-primary.service - Postgres primary

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 41ms

    History: █░█████████████████████████▓████████████████████▓█████████████████████████████████████████░████████████▓██████████▓████████████████████

  • backups.timer - Backups

    Active: active (running) (Operational)

    Reliability: 96.73% (90d)

    History: ████████▓█████████████░███████▓███████████████████████████████████████████████████████████████▓██████░███▓█████████████████████████████

Edge

  • cdn.service - CDN

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 60ms

    History: ██████████████████████████████████████▓███▓███░██████████████████████████████████████████████████████████████▓█▓█░█████████████████████

  • webhooks.service - Webhooks

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 273ms

    History: █████████████████████████████▓████░██████▓██████████████████████████████████████████████████████████████▓██░██▓████████████████████████

log --tail 20 # History

  1. [] incident 797fb81: API and dashboard unavailable (investigating)

    -- 2 automatic events --
    • [] dashboard.service: failed
    • [] api.service: failed

  2. [] incident 200cd2f: Elevated error rates (resolved, 50 min)

    -- 2 automatic events --
    • [] api.service: active (running)
    • [] api.service: failed
    -- details --

    A bad deploy raised error rates for about 45 minutes. We rolled it back and added a canary stage.

    1. []

      Error rates are back to normal. This incident is resolved.

    2. []

      We found a faulty deploy and are rolling it back.