Earlier this month, Google Cloud experienced one of its biggest blunders ever when UniSuper, a $135 billion Australian pension fund, had its Google Cloud account wiped out due to some kind of mistake on Google's end. At the time, UniSuper indicated it had lost everything it had stored with Google, even its backups, and that caused two weeks of downtime for its 647,000 members. There were joint statements from the Google Cloud CEO and UniSuper CEO on the matter, a lot of apologies, and presumably a lot of worried customers who wondered if their retirement fund had disappeared.
In the immediate aftermath, the explanation we got was that "the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription." Two weeks later, Google Cloud's internal review of the problem is finished, and the company has a blog post up detailing what happened.
Google has a "TL;DR" at the top of the post, and it sounds like a Google employee got an input wrong.
During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.
The most shocking thing about Google's blunder was the sudden and irreversible deletion of a customer account. Shouldn't there be protections, notifications, and confirmations in place to never accidentally delete something? Google says there are, but those warnings are for a "customer-initiated deletion" and didn't work when using the admin tool. Google says, "No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool, and not due to a customer deletion request. Any customer-initiated deletion would have been preceded by a notification to the customer."