“Unprecedented” Google Cloud event wipes out customer account and its backups

Buried under the news from Google I/O this week is one of Google Cloud’s biggest blunders ever: Google’s Amazon Web Services competitor accidentally deleted a giant customer account for no reason. UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper’s incident log, downtime started May 2, and a full restoration of services didn’t happen until May 15.

UniSuper’s website is now full of must-read admin nightmare fuel about how this all happened. First is a wild page posted on May 8 titled “A joint statement from UniSuper CEO Peter Chun, and Google Cloud CEO, Thomas Kurian.” This statement reads, “Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription. This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.”

In the next section, titled “Why did the outage last so long?” the joint statement says, “UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.” Every cloud service keeps full backups, which you would presume are meant for worst-case scenarios. Imagine some hacker takes over your server or the building your data is inside of collapses, or something like that. But no, the actual worst-case scenario is “Google deletes your account,” which means all those backups are gone, too. Google Cloud is supposed to have safeguards that don’t allow account deletion, but none of them worked apparently, and the only option was a restore from a separate cloud provider (shoutout to the hero at UniSuper who chose a multi-cloud solution).

UniSuper is an Australian “superannuation fund“—the US equivalent would be a 401(k). It’s a retirement fund that employers pay into as part of an employee paycheck; in Australia, some amount of superfund payment is required by law for all employed people. Managing $135 billion worth of funds makes UniSuper a big enough company that, if something goes wrong, it gets the Google Cloud CEO on the phone instead of customer service.

A June 2023 press release touted UniSuper’s big cloud migration to Google, with Sam Cooper, UniSuper’s Head of Architecture, saying, “With Google Cloud VMware Engine, migrating to the cloud is streamlined and extremely easy. It’s all about efficiencies that help us deliver highly competitive fees for our members.”

The many stakeholders in the service meant service restoration wasn’t just about restoring backups but also processing all the requests and payments that still needed to happen during the two weeks of downtime.

https://arstechnica.com/?p=2025177