Applied Tech’s Post

As most are aware by now, CrowdStrike is currently facing a significant challenge due to a recent software update error, resulting in global outages on Microsoft systems. This incident has notably impacted various sectors, including airlines, media, and healthcare. While not a cyberattack, the considerable disruption to business operations serves as a sobering reminder on the importance of robust system availability and security. End users are significantly impacted by this outage. Millions of workstations around the world are crashing, leaving users with blue screens and unusable computers. An incident of this magnitude is rare, as many of the workarounds require manual intervention by IT personnel. This situation underscores the need for resilience, rigorous testing, and having rollback options in place when deploying updates. Mistakes and oversight can happen; strong incident response and up-to-date disaster recovery plans are essential. To those impacted, CrowdStrike is actively addressing the situation. They have released a statement with the latest updates, which can be found here: https://shorturl.at/jbhSa #CrowdStrike #Microsoft

🔔 Important Update: CrowdStrike Outage   If you or any of your co-workers are experiencing the "Blue Screen of Death" or issues stemming from the CrowdStrike update - keep reading! What Happened? On July 19, Microsoft machines running CrowdStrike Falcon encountered a BSOD (Blue Screen of Death) and got stuck in a restarting state. CrowdStrike has isolated the issue and issued a fix. The CrowdStrike CEO confirmed that this is not a security incident. Key points: 🖥️ Affected Systems: Only Windows devices are impacted. 🌐 Broader Impact: This issue may affect companies that do not use CrowdStrike Falcon, as many software and platform vendors are experiencing service degradation, including Microsoft, LogMeIn, Jonas, USPS, and more. 🔄 Patch Update: CrowdStrike temporarily ceased patch deployment and released an updated patch to rectify the issue as of 1:27 AM EDT. ❗️WorkSmart's Recommendation: If you are running CrowdStrike, avoid rebooting your systems for the time being to allow the updated patch to apply. For more info, you can refer to CrowdStrike's event page here: https://lnkd.in/gtZQ5TD4 P.S. If you're looking for a committed IT partner to help when incidents like this arise - contact us today! #CrowdStrike #worksmart #WorkSmartIT #keepingITsecure

97% of CrowdStrike systems are back online; Microsoft suggests Windows changes https://ift.tt/b7qsGxX Enlarge / A bad update to CrowdStrike's Falcon security software crashed millions of Windows PCs last week. (credit: CrowdStrike) CrowdStrike CEO George Kurtz said Thursday that 97 percent of all Windows systems running its Falcon sensor software were back online, a week after an update-related outage to the corporate security software delayed flights and took down emergency response systems, among many other disruptions. The update, which caused Windows PCs to throw the dreaded Blue Screen of Death and reboot, affected about 8.5 million systems by Microsoft's count, leaving roughly 250,000 that still need to be brought back online. Microsoft VP John Cable said in a blog post that the company has "engaged over 5,000 support engineers working 24x7" to help clean up the mess created by CrowdStrike's update and hinted at Windows changes that could help—if they don't run afoul of regulators, anyway. "This incident shows clearly that Windows must prioritize change and innovation in the area of end-to-end resilience," wrote Cable. "These improvements must go hand in hand with ongoing improvements in security and be in close cooperation with our many partners, who also care deeply about the security of the Windows ecosystem." Read 7 remaining paragraphs | Comments via Biz & IT – Ars Technica https://arstechnica.com July 26, 2024 at 01:46PM

Crowdstrike says global Microsoft outage caused by ‘defect’ in ‘content update’ Full Article Link >>> https://lnkd.in/gdy26ARN Welcome To Latest IND >> Fastest World News Jul 19, 2024 07:01 PM IST Crowdstrike CEO George Kurtz said, “Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.” Crowdstrike CEO George Kurtz said that the company “is actively working with customers impacted by a defect found in a single […] . . Latest IND . . . . #trendingnews #newstrending #trendingtopicnews #lifestyle #business #news #healthylifestyle #smallbusiness #supportsmallbusiness #lifestyleblogger #luxurylifestyle #businessowner #businesswoman #smallbusinessowner #businessnews

Wow that was a bit of a crazy weekend for some, but not us: luckily we’re fully Linux across our hundreds and hundreds of workloads at GlobalizeMe. But now is not the time, whatsoever, for schadenfreude; with bad planning and shoddy release management this could happen to anyone. My heart does go out to all those consumers affected, not only with leisure activities but in healthcare, transport and finance - millions upon millions of people hit globally. Not to mention all the Windows server engineers slogging their guts out over the weekend to get machines up and running again following the aftermath of this devastating driver patch (and I heard some desktops are affected too in some corporates who were using CrowdStrike to “secure” PCs and laptops). We don’t know the full story yet (some good discussion here https://lnkd.in/eDCfTkgt) but initially a few thoughts spring to mind whatever the technicalities of this are: 1) Releasing critical software patches directly to production servers without testing? Really? In what world does this happen? Perhaps this was another victim of the auto update malaise that we see in the Windows world (I have a couple of windows machines and there’s always a suspicious trust to let them happen but not always comfort I’d say…) but it feels so dangerous to me when dealing with OS-affecting changes. It’s sooooo easy now to spin up a little instance in the cloud (AWS or Azure) and deploy patches there to see what breaks - yes this would require a licence and so forth but wow, think of the cost saved. The mind boggles. 1a) Apparently CrowdStrike did have this fixed by early US time and deployed a new version of the patch. So I’d assume countless servers who didn’t have auto update enabled and who deployed the latter version were okay. This begs the question whether this was ever tested extensively at CrowdStrike first. 2) Critical software releases on a Friday? Really? Most companies I know (including us) have release blocks on a Friday. At GlobalizeMe we may release early on a Friday morning BUT only when it’s super super mission critical and only when we have full staff support over the weekend. I’m not sure this was well planned… 3) Windows Server? Really? I guess I’ve had my head in the Linux / Unix rabbit hole for way too long. Since my days at the the Telegraph from 1998-2004 then over the past 20 years working with companies like Esure.com and working for Pharmaceutical and Law Enforcement clients plus other Fintech and manufacturing clients, it’s always been Unix or Linux for us, for secure, complex and heavy public-facing workloads. I’m surprised at how many huge names are using Windows for this purpose. I’m genuinely ready and willing to be educated as I really didn’t know. Interesting times and again I guess this could happen to anyone. Once again, thoughts and wishes to all affected 🙏

Crowdstrike says global Microsoft outage caused by ‘defect’ in ‘content update’ Full Article Link >>> https://lnkd.in/gQnqFSzh Welcome To Latest IND >> Fastest World News Jul 19, 2024 07:01 PM IST Crowdstrike CEO George Kurtz said, “Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.” Crowdstrike CEO George Kurtz said that the company “is actively working with customers impacted by a defect found in a single […] . . Latest IND . . . . #trendingnews #newstrending #trendingtopicnews #lifestyle #business #news #healthylifestyle #smallbusiness #supportsmallbusiness #lifestyleblogger #luxurylifestyle #businessowner #businesswoman #smallbusinessowner #businessnews

The recent CrowdStrike outage stemmed from a problematic driver file in the Windows/system32/drivers directory, specifically the C-00000291*.sys file. This issue highlights potential failures in the development and deployment process, such as insufficient testing, rushed deployment, or human error. Despite the severity, CrowdStrike clarified that it was not a result of a cyberattack or security breach but rather a coding error in the update. I was trying to list the issues that could introduce the “coding error in the update.” Here are they: Human Error and Incompetence 1. Insufficient Testing: The problematic update may not have undergone thorough testing. Ideally, updates should be tested in a staging environment that closely mirrors the production environment. Even testing on a small number of servers (e.g., five) could have revealed the issue. 2. Quality Assurance Gaps: Gaps in the quality assurance process might lead to overlooked critical issues. This can be due to insufficient resources, pressure to meet deadlines, or miscommunication within the team. 3. Mistakes in Development: Developers might have made coding or integration errors that were not caught for the aforementioned reasons. 4. Lack of Competence: While harsh, it's possible that the entire team responsible for this deployment lacked the necessary expertise or experience. It is important to list the Impact and Repercussions: 1. Customer Trust: Such incidents can severely damage customer trust and the company's reputation, especially for a cybersecurity firm like CrowdStrike. 2. Operational Disruptions: The outage significantly disrupted clients' operations, leading to potential financial and legal consequences. Who was affected and who didn’t: Machines running Windows 7 and above were affected, but the issue did not affect Mac or Linux systems. Remediation involves booting affected systems into Safe Mode, deleting the problematic file, and rebooting. While this process sounds straightforward, it requires manual intervention on every affected machine, which is time-consuming and labor-intensive. #crowdstrike #IToutage #softwareupdate #MicrosoftWindows #MajorGlobalIToutage

Check out this article about the recent Microsoft Windows outage and the impact on global IT systems: [link here]. Now is the perfect moment to reassess your Backup/DR solution. Would it effectively support you in a similar situation? We offer resources to assist in determining the right solution for your environment. Let's connect and discuss further!

Last week’s Crowdstrike incident was a significant event for many, but we are pleased to report that none of our clients were affected. After confirming our stability, my team and I extended our assistance to others in our network, despite some team members battling COVID. This weekend has been extremely hard for those in IT and Security. Approximately 8.5 million Windows devices were impacted. While Crowdstrike’s sensor configuration update triggered a logic error resulting in system crashes, it’s worth noting that Microsoft acknowledged that operating systems like macOS wouldn’t face this problem. Unfortunately, due to "legal agreements", Microsoft cannot implement the same protections, making it impossible to prevent such issues on Windows in the future. If you are using Windows, we strongly recommend considering macOS or ChromeOS for better reliability and security. This incident underscores the critical role of IT in safeguarding business operations. Here are some key lessons we can all take away: 1. Assess Tool Access: Always evaluate the level of access that tools have to your system, whether they are security tools requiring kernel access or SaaS tools needing email permissions. 2. Backup & Recovery: Often an afterthought, backup and recovery plans are essential. Set up annual checkpoints to ensure a comprehensive plan covering not only data, but endpoints, identities, and systems. 3. IT Strategy Integration: Incorporate these considerations into your overall IT strategy. Opt for solutions that reduce risk, ensure reliability and security, and offer resilience. As seen, macOS and ChromeOS are examples of more secure alternatives to Windows. 4. Prioritize IT: IT might be in the background, but it’s vital for modern businesses. Companies that prioritize IT and give it a strategic role benefit significantly, as evidenced by the events of last week. Don’t wait until it’s too late again. Take some time to talk with your trusted IT team or partner and review this situation to ensure you are ready for the next one.

CrowdStrike recently published a Preliminary Post Incident Review analyzing their outage. In their blog post, CrowdStrike describes the root cause as a memory safety issue—specifically a read out-of-bounds access violation in the CSagent driver. We leverage the Microsoft WinDBG Kernel Debugger and several extensions that are available free to anyone to perform this analysis. Customers with crash dumps can reproduce our steps with these tools. Based on Microsoft’s analysis of the Windows Error Reporting (WER) kernel crash dumps related to the incident, we observe global crash patterns that reflect this: