Google Cloud Services Hit by Outage in Paris

The incident was later described as "a multicluster failure and has led to an emergency shutdown of multiple zones."

Apr 27th, 2023 7:40am by Chris J. Preimesberger

Featued image for: Google Cloud Services Hit by Outage in Paris

Featured image via Unsplash.

Google Cloud was hit by an outage of a high number of services yesterday when a data center in Paris caught fire around midnight Tuesday. The fire caused the fire department to soak the building with water, precipitating a multi-cluster failure which shut down more than 90 cloud services.

Google Cloud’s status page, calling the cause of the failure “water intrusion,” indicated that multiple cloud services in the “europe-west9-a” zone were affected starting at 1900 PDT on April 25.

The company’s status page explained that “water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone,” and that there was no current ETA for recovery of operations. Customers were advised to fail over to other zones in europe-west9 if they were impacted.

The incident was later described as “a multicluster failure and has led to an emergency shutdown of multiple zones.” The outage impacted more than 90 Google Cloud services for europe-west9-a customers and was ongoing at the time this article was filed.

Google declined to comment beyond its status page statements and stated that details would be provided in its incident report next week. The company apologized to all who were affected by the disruption.

The Disruption

Service disruption details are as follows:

Services that were curtailed temporarily included:

Vertex AI AutoML Image, Cloud Debugger, Text-to-Speech, Vertex AI Matching Engine, AI Platform Training, Cloud Monitoring, Vertex AI AutoML Tabular, Pub/Sub Lite, Speech-to-Text, Hybrid Connectivity, Cloud Key Management Service, Cloud Natural Language API, Cloud Run, VMWare engine, Vertex AI TensorBoard, Apigee, Cloud Developer Tools, Virtual Private Cloud (VPC), reCAPTCHA Enterprise, Cloud Workflows, Cloud Firestore, Anthos Service Mesh, Operations, Cloud Spanner, Vertex AI Explainable AI, Cloud Profiler, Cloud External Key Manager, Vertex AI Workbench User Managed Notebooks, VPC Service Controls, Cloud Armor, Recommender, Google Compute Engine, Google Kubernetes Engine, Identity Platform, Cloud Memorystore, Google Cloud Bigtable, Resource Manager API, Google Cloud Datastore, Traffic Director, Cloud Logging, Web Risk, Artifact Registry, Cloud HSM, Retail API, Vertex AI Vizier, Persistent Disk, Vertex AI Data Labeling, Google Cloud Dataflow, Data Catalog, Google Cloud DNS, Vertex AI Model Registry, BigQuery Data Transfer Service, Google Cloud Storage, Google Cloud Networking, API Gateway, Google Cloud Console, Dataplex, Google Cloud Scheduler, Eventarc, Google Cloud Composer, Identity and Access Management, Vertex AI Training, Cloud CDN, Google Cloud Pub/Sub, Access Approval, AI Platform Prediction, AlloyDB for PostgreSQL, Cloud Build, Vertex AI AutoML Video, Vertex AI AutoML Text, Cloud NAT, Google Cloud SQL, Assured Workloads, Cloud Load Balancing, Recommendation AI, Vertex AI Pipelines, Cloud Filestore, Google App Engine, Secret Manager, Managed Service for Microsoft Active Directory (AD), Cloud IDS, Cloud Domains, Access Transparency, Cloud Billing, Google Cloud Functions, Access Context Manager, Vertex AI Feature Store, Cloud Asset Inventory, Cloud Data Fusion, Storage Transfer Service, Vertex AI ML Metadata, Vertex AI Online Prediction, Vertex AI Model Monitoring, Google Cloud Identity-Aware Proxy, Video Intelligence API, Database Migration Service, Service Directory, Transcoder API, Cloud Endpoints, Vertex AI Batch Prediction, Google Cloud Dataproc, Cloud Machine Learning.

Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT.

GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see the migration guide for gDNS to Zonal DNS for more information.

Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here.

BigQuery: For information related to ongoing BigQuery impact, please see the latest status here.

Workaround: Customers can failover to zones in other regions.