WhatBrokeToday
Today's Trends
highResolvedGoogle Cloud

Google Cloud: RESOLVED: Multiple services were experiencing issues in the europe-west2 region.

  1. Investigating(done)
  2. Trending(skipped)
  3. Workaround available(skipped)
  4. Resolved(current)

Executive summary

A power disruption in the europe-west2 region caused Apigee X runtime and Cloud Bigtable to experience elevated HTTP 5xx errors for over 7 hours. The incident has since been resolved.

No verified workaround yet

This incident was resolved upstream without a reported workaround.

Official fix
Technical root cause

A scheduling misconfiguration placed two out of three datastore replicas in the single zone that lost power, causing the datastore to lose quorum and fail requests instead of failing over.

Evidence

1 report · 1 StatusShow
  1. [1]
    Status First report
    RESOLVED: Multiple services were experiencing issues in the europe-west2 region.(opens official status page in a new tab)

    Cloud Bigtable Impact: Next Steps: Apigee Edge Impact: Next Steps: During the power disruption, the Apigee X runtime in the region returned elevated HTTP 5xx errors for 7 hours and 19 minutes. Apigee X is designed to withstand a single-zone loss as its datastore keeps replicas across the region's three zones, but a scheduling misconfiguration had placed two of three replicas in the zone that lost power, so the datastore lost quorum and the runtime failed requests instead of failing over.…

    by gcp status

Timeline

  1. First report on status page

    RESOLVED: Multiple services were experiencing issues in the europe-west2 region.

    Source
  2. Official statement

    Cloud Bigtable Impact: Next Steps: Apigee Edge Impact: Next Steps: During the power disruption, the Apigee X runtime in the region returned elevated HTTP 5xx errors for 7 hours and 19 minutes.

    Source
  3. Status changed

    Status changed: new → resolved (official status page reports resolution).