Skip to content
CERTIFICATES

Renegade certificate removed from Windows. Then it returns. Microsoft stays silent.

The certificate, originally spawned by Symantec, was scheduled to be banished years ago.

Dan Goodin
Credit: Getty Images
Credit: Getty Images

For three days, system administrators have been troubleshooting errors that have prevented Windows users from running applications such as QuickBooks and Avatax. We now know the cause: an unannounced move or glitch by Microsoft that removed a once-widely used digital certificate in Windows.

The removed credential is known as a root certificate, meaning it anchors the trust of hundreds or thousands of intermediate and individual certificates downstream. The root certificate—with the serial number 18dad19e267de8bb4a2158cdcc6b3b4a and the SHA1 fingerprint 4EB6D578499B1CCF5F581EAD56BE3D9B6744A5E5—was no longer trusted in Windows. Because that root was tied to certificates that certify their authenticity and trust, people trying to use or install the app received the error.

Just minutes before this post was scheduled to go live, researchers learned that the certificate had been restored in Windows. It’s unclear how or why that occurred. The certificate immediately below this paragraph shows the certificate's status on Thursday. The one below that shows the status as of Friday.

That time Symantec certs were banished from the Internet

Microsoft has yet to respond to a request to explain the errors. It may be that a glitch caused Windows to remove the root certificate. It’s also possible the removal was intentional, given that it’s one of several that faced an industry-wide blockade following the discovery in 2015 that its parent issuer at the time, Symantec, had improperly issued certificates for google.com, www.google.com, and one other domain. (Symantec sold its certificate authority (CA) businesses to DigiCert in 2017.)

After Google researchers asserted a few weeks later that the number of mis-issued certificates was much higher, Symantec revised the number to 164 certificates for 76 domains and 2,458 certificates for domains that had never been registered. In light of the new information, Google gave Symantec an ultimatum: give a thorough accounting of its ailing certificate authority process or risk having the world's most popular browser—Chrome—issue scary warnings about Symantec certificates whenever end users visited HTTPS-protected websites that used them.

Some 17 months later, Google made good on the threat after its investigation concluded that for years, Symantec-owned CAs had improperly issued more than 30,000 certificates. The company began preparations to gradually nullify Chrome’s trust in all certificates issued by those CAs, which were sold under brands including Verisign, Thawte, and GeoTrust. Effective immediately at that time, Chrome stopped recognizing any extended validation status of such certificates, and as time went on, the browser revoked more and more of its trust.

Mis-issued certificates represent a critical threat to virtually the entire Internet population; they make it possible for the holders to cryptographically impersonate the affected sites and monitor or tamper with communications sent between visitors and the legitimate servers. In particular, certificates for non-existent domains or domains belonging to parties other than the holder are major violations of the so-called baseline requirements that major browser makers impose on CAs as a condition of being trusted by their software.

Symantec’s transgressions were serious. But given Symantec’s status at the time as one of the biggest issuers of certificates, Google and other stakeholders were in a bind. If Google or other browser makers were to nullify all of the Symantec-issued certificates overnight, it would cause widespread outages. The chaos that would result made the issuer too big to fail. The penalties outlined by Google aimed to minimize such disruptions while exacting a meaningful punishment.

Over the next two years, browser makers and other companies that rely on digital certificates to secure Internet communications gradually phased out trust in the certificates. Most timetables called for a deadline sometime in 2019. For reasons Microsoft has yet to explain, Windows continued to trust the root certificates to sign software.

That trust was finally revoked—or at least suspended—on Tuesday, once again with no explanation or notice. The move sent sys admins scrambling to determine why users were receiving certificate errors when trying to run software such as QuickBooks and AvaTax. Eventually, the CEO of security firm Airlock Digital traced the cause to the unannounced change in Windows.

A Microsoft representative offered to provide comment for this story on the condition the information not be attributed to Microsoft in any way. Ars declined.

It’s likely that Microsoft delayed the revocation of the certificate for app-signing purposes because certificates in apps can’t be updated as easily as they can for websites. With no guidance from the company, people troubleshooting error messages are on their own.

One option for resolving problems is to update affected apps. By now, most apps have likely been updated to use certificates not related to the ones that have been blocked. By default, Windows has a feature known as automatic root updates turned on. Some users have it turned off for various reasons, many of them legitimate. The above-linked Reddit thread also provides several scripts people can run to rotate out the root certificate.

Update: 10 minutes after Ars declined Microsoft's offer for a not-for-attribution comment, a company representative sent the following statement:

The VeriSign Class 3 Public Primary Certification Authority – G5 is distrusted as of 2019 and was set to “NotBefore” in a previous release. This means that certificates issued after the NotBefore date will no longer be trusted; however, certificates issued before the NotBefore date will continue to be trusted. In our August Certificate Trust List update, we changed this setting to Disable as a part of our regular deprecation process which caused some customers with specific configurations to run into issues. On August 24, 2023, we rolled back this change to help remediate these issues.

Listing image: Getty Images

Photo of Dan Goodin
Dan Goodin Senior Security Editor
Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at @dangoodin on Mastodon. Contact him on Signal at DanArs.82.
Staff Picks
Xelas
Revoking those broken certs wasn't the issue. The issue was the total lack of communication around this. If the certs were sunset by everyone else by 2019, Microsoft could have spent the last 4 years loudly telegraphing their plan to sunset them in 2023 (if that's what they were planning) to prepare users and admins for this. That said, the certs were known to not be trusted back in 2015, so Microsoft had 8 years to work on this. That's an eternity.

If this was unplanned, then it's even more damning.
afidel
This update was not only stealth rolled out, but not rolled out very uniformly. Our financial system was impacted for about 20 users out of a few thousand, the vendor resigned the binaries and gave us an updated package which we quickly rolled out to all workstations and remote access servers worldwide to avoid it crippling billing. The fact that it was rolled out by a mechanism other than standard patching or a process under the admins control meant that it couldn't be caught by normal QA testing procedures and change management controls that we put all updates through regardless of vendor or technology. It's yet another example of the XaaS mentality taking over the commercial software world where cowboys are allowed to move fast and break things and then roll back their actions rather than moving through a methodical and deliberate process with useful documentation and notice.
morlamweb
The software that I support was hit with this problem this week. One of the client-side programs was signed with a cert that chains back to one of these mysteriously-revoked root CA certs. We're still scrambling to push out updates.
evan_s
I don't think revoking this was a mistake at all. Clearly, it needs to be done but the silent deactivation with no warning is a horrible way to do it. MS could have announced this ahead of time. Provided all sorts of tools for Admins to look for problematic programs. Could have provided warning prompts for users saying this program will stop working in X days. Could have provided work arounds to trust specific certs so you can keep running specific apps based on the revoked root.
Voo42
Show me someone who says something is “too big to fail” and I’ll show you someone who shouldn’t be in charge.
There is only a single issuer field and the validation defined in RFC5280 requires that single point of failure (at least as far as I'm aware of the whole process, who knows if there's some extension out there).

Which means that yes every single CA trusted by major browsers and operating systems is by definition "too big to fail" without a very long deprecation process. And even then there's no solution that won't leave some people with broken apps.

Would be interesting to allow multiple issuers, but I'm sure there's some interesting problems there to solve.
Old_Fogie_Late_Bloomer
This probably explains my neighbor calling me last night because she couldn’t get into Quicken and her computer wouldn’t start back up. The problem resolved itself while I was on the phone, probably due to Microsoft’s silent update.

Man, whenever Apple pisses me off, Microsoft rolls up to remind me how the alternative is arguably worse.
Prev story
Next story