After breach, senators ask why AT&T stores call records on “AI Data Cloud”

US senators want AT&T to explain why it stores massive amounts of call and text message records on a third-party analytics platform that bills itself as an "AI Data Cloud."

AT&T revealed last week that "customer data was illegally downloaded from our workspace on a third-party cloud platform," and that the breach "includes files containing AT&T records of calls and texts of nearly all of AT&T's cellular customers." The third-party platform is Snowflake, and AT&T is one of many Snowflake corporate customers that had data stolen. Ticketmaster is another notable company affected by the breach.

AT&T and Snowflake each got letters yesterday from US Sens. Richard Blumenthal (D-Conn.) and Josh Hawley (R-Mo.), the chair and ranking member of the Senate Judiciary Subcommittee on Privacy, Technology, and the Law. The senators asked AT&T CEO John Stankey to answer a series of questions, including this one:

Why had AT&T retained months of detailed records of customer communication for an extended amount of time and why had AT&T uploaded that sensitive information onto a third party analytics platform? What is AT&T policy, including timelines, concerning retaining and using such information?

AT&T's disclosures to customers and to the Securities and Exchange Commission didn't explain how Snowflake is used by AT&T. Snowflake's website says the company's cloud platform provides opportunities for collaborating and sharing data:

Powering the AI Data Cloud is Snowflake's single platform. Its unique architecture connects businesses globally, at practically any scale to bring data and workloads together. Together with the Snowflake Marketplace which simplifies the sharing, collaborating, and monetizing of thousands of datasets, services, and entire data applications—this creates the active and growing AI Data Cloud.

AT&T a featured customer

There was already a public explanation for why AT&T uses Snowflake, but it's written in marketing speak and isn't likely to directly answer the senators' questions. Sometime before the hacks, Snowflake posted a glowing case study on how AT&T lowered costs and gained "faster insights" by switching from internal systems to Snowflake.

Snowflake says it provides a telecom-focused AI Data Cloud service that helps firms like AT&T "improve customer experiences, maximize operational efficiency and increase profitability by reducing costs and monetizing new data products." AT&T's decision to move data to Snowflake apparently allowed it to abandon "complex on-premises systems, including Hadoop" that "were slowing down business."

"The Snowflake Data Cloud has given us the power to harness and integrate data to create insights," AT&T Chief Data Officer Andy Markus is quoted as saying in the promotional material. "With data at our fingertips, we are growing revenue, becoming more cost effective and, most importantly, improving the customer experience."

Markus said the previous internal system made it hard to collaborate with other companies. "Prior to Snowflake, we had a very complex data environment on-premises," Markus said. "That led to a more ineffective operating environment for our business partners, both from a speed and cost perspective."

With Snowflake, AT&T is said to have "a powerful, easy-to-use data management system that efficiently processes hundreds of petabytes of data every day." This makes it easier to share data.

"Using Hadoop for storage and processing, AT&T's monolithic on-premises data warehouse hampered the team from collecting, storing, sharing and processing its vast stores of data," the customer case study said. "By moving to the Snowflake Telecom Data Cloud, Markus and his team achieved their goal of democratizing data across the business."

Snowflake boasted that because of its cloud platform, "this leading telecom provider uses data to advance innovation, create new revenue streams, optimize operations and, most importantly, better connect people to their world."

AT&T said it uses “trusted” cloud providers

When contacted by Ars today, AT&T provided a statement in response to the senators' questions about its use of Snowflake. "Like most companies that deal with large amounts of data, AT&T often uses specialized and trusted cloud services platforms for various functions. These platforms enable companies to work with large amounts of data in a centralized place. In this case, AT&T had put a copy of the data on the third-party platform for analysis related to our business," AT&T told us.

AT&T added that it "analyzes historical customer data for uses that include network planning, capacity utilization, and developing new services and offers."

AT&T did not provide specifics on how long it retains data. "We set our data retention periods depending on the type of personal information, how long it is needed to operate the business or provide our products and services, and whether it is subject to contractual or legal obligations. These obligations might be ongoing litigation, mandatory data retention laws, or government orders to preserve data for an investigation," the company said today.

We also asked Snowflake for details on exactly how phone companies use its platform. A Snowflake spokesperson did not answer our question but told us that the company will respond directly to the senators.

Senators: AT&T user data could be auctioned

AT&T said the stolen call and text records identify the phone numbers that AT&T numbers interacted with, but that no customer names or Social Security numbers were taken. AT&T also said the contents of communications were not included in the breach. But AT&T acknowledged that it would be possible for criminals to find the names associated with specific phone numbers.

AT&T said it does not believe the stolen call data has been made publicly available, and the firm reportedly paid a hacker $370,000 to delete the records. Blumenthal and Hawley are not convinced by the company's assurances, however.

The cybercrime group ShinyHunters has already leaked records of Ticketmaster customers and offered to sell data stolen from Snowflake customers, the senators' letter said. "There is no reason to believe that AT&T's sensitive data will not also be auctioned and fall into the hands of criminals and foreign intelligence agencies," the senators wrote.

Blumenthal and Hawley are concerned about customers' location data being leaked:

While the records do not directly include names and addresses, as AT&T's Securities and Exchange Commission filing notes, the stolen data includes location information and it is easy to find the name associated with a phone number. Taken together, the stolen information can easily provide cybercriminals, spies, and stalkers a logbook of the communications and activities of AT&T customers over several months, including where those customers live and traveled—a stunning and dangerous breach of its customers' privacy and intrusion into their personal lives.

Although the breach apparently didn't include the location of all customers, AT&T's SEC filing said a subset of records contained "one or more cell site identification number(s)."

Basic security failures

Blumenthal and Hawley's letter to Snowflake said the AT&T and Ticketmaster breaches appear "to have been easily preventable." They wrote:

While Snowflake, AT&T, Ticketmaster, and other clients have avoided taking direct responsibility, according to the cybersecurity firm Mandiant, it appears that the cybercrime group behind the breaches obtained companies' passwords from malware infections, including malware bundled with pirated software. Compounding this basic cybersecurity failure, the hacked accounts had often kept the same passwords for several years, failed to implement firewall access, and failed to turn on multi-factor authentication—additional basic cybersecurity failures that seemingly reflect gross negligence, particularly in light of the sensitivity of the data stolen in many of the breaches.

Among other things, the senators want AT&T to explain how hackers gained access to the company's Snowflake workspace, and "a full accounting of the types of data stolen from AT&T and how that data was linked together in a manner that would impact the privacy of customers." The senators asked AT&T to provide answers by July 29.

AT&T last week said it has "clos[ed] off the point of unlawful access" and is notifying current and former customers of the breach. AT&T coordinated with the Federal Bureau of Investigation before revealing the breach publicly, and the Federal Communications Commission said it is investigating.