WhatBrokeToday
Today's Trends
minorArchivedFly.io

Fly.io: Upstash unavailability in FRA region

  1. Investigating(done)
  2. Trending(skipped)
  3. Workaround available(skipped)
  4. Resolved(done)
  5. Archived

Executive summary

Some clients experienced connection timeouts to their Redis databases hosted in the fra or gig regions due to a planned rolling upgrade where replicas were left out of service and returned to Fly routing before they were ready. The issue was resolved by 13:05 UTC.

No verified workaround yet

This incident was resolved upstream without a reported workaround.

Official fix
Technical root cause

During a planned rolling upgrade, replicas in FRA and GIG did not finish draining and were left out of service, leading to unavailability.

Evidence

1 report · 1 StatusShow
  1. [1]
    Status First report
    Upstash unavailability in FRA region(opens official status page in a new tab)

    Status: Resolved Impact From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region. What happened During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections…

    by flyio status

Timeline

  1. First report on status page

    Upstash unavailability in FRA region

    Source
  2. Official statement

    Status: Resolved Impact From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases.

    Source
  3. Status changed

    Status changed: new → archived (resolved more than 14 days ago).