DemoAcme Cloud · Terminal template

Use this template

Acme Cloud

Live status for the Acme Cloud API, dashboard and data pipeline. Updates land here first, or read the incident handbook.

uptime

7 units, load elevated

status --all

7 units loaded, 1 degraded

warn: Nightly backups running late

Partial outage

Incidents

Incident: Nightly backups running late

title:
Nightly backups running late
impact:
minor impact
status:
identified
started:
(35 min)
units:
backups.timer

Last night's backup has not checked in yet. Data is safe; we are looking into the storage queue.

  1. []

    The backup is running again and should finish within the hour.

  2. []

    A stuck storage queue worker held the job. We restarted it.

Core platform

  • api.service - API

    Active: active (running) (Operational) since

    Uptime: 99.84% (90d) Latency: 190ms

    History: ███▓██████████████████████████████████▓███████████████████████████░████████████████████████▓█████████████████▓█████████████░███████████

  • dashboard.service - Dashboard

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 334ms

    History: ███████████████████████████████▓█████████████▓██░████████████████████████████████████████████████████████▓██████▓█░████████████████████

  • auth.service - Auth

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 157ms

    History: ███░█████████▓███████████████▓█████████████████████████████████████████████████████████████░████▓███████▓██████████████████████████████

Data

  • postgres-primary.service - Postgres primary

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 41ms

    History: █░█████████████████████████▓████████████████████▓█████████████████████████████████████████░████████████▓██████████▓████████████████████

  • backups.timer - Backups

    Active: active (degraded) (Degraded)

    Warn: 3 of 90 days with failures

    Reliability: 96.70% (90d)

    History: ████████▓█████████████░███████▓██████████████████████████████████████████████████████████·████▓██████░███▓█████████████████████████████

    Incident: 6efffa4 (minor)

Edge

  • cdn.service - CDN

    Active: active (running) (Operational)

    Uptime: 99.84% (90d) Latency: 60ms

    History: ██████████████████████████████████████▓███▓███░██████████████████████████████████████████████████████████████▓█▓█░█████████████████████

  • webhooks.service - Webhooks

    Active: active (running) (Operational)

    Uptime: 99.80% (90d) Latency: 273ms

    History: █████████████████████████████▓████░██████▓███████████████████████████████████████████████▓██████████████▓██░██▓███████████████████████▓

log --tail 20 # History

  1. [] incident 6efffa4: Nightly backups running late (identified)

  2. [] incident 200cd2f: Elevated error rates (resolved, 50 min)

    -- 2 automatic events --
    • [] api.service: active (running)
    • [] api.service: failed
    -- details --

    A bad deploy raised error rates for about 45 minutes. We rolled it back and added a canary stage.

    1. []

      Error rates are back to normal. This incident is resolved.

    2. []

      We found a faulty deploy and are rolling it back.