Chenxi Wang, Ph.D.’s Post

Many people and businesses are mistakenly blaming Microsoft for the recent IT global outages. First, I want to acknowledge how frustrating and disruptive this has been, especially for those traveling. I've only seen a glimpse of the chaos, but many of my friends have been caught up in this debacle. While it might appear Microsoft was the cause, I assure you this global outage was not due to Microsoft. The root of the issue was a combination of factors, but not Microsoft's fault. The primary source was a software company called CrowdStrike, which provides computer security for much of the world. CrowdStrike released a version of their software that affected business computers running Microsoft Windows. Additionally, many IT departments, including those in the airline industry, failed to properly vet updates before allowing CrowdStrike to automatically update their systems. Talk about a case where automation backfired. It's truly unfortunate that so many people were impacted by this mass outage! #crowdstrike #outage #Microsoft #security

Many people and businesses are mistakenly blaming Microsoft for the recent IT global outages. First, I want to acknowledge how frustrating and disruptive this has been, especially for those traveling. I've only seen a glimpse of the chaos, but many of my friends have been caught up in this debacle. While it might appear Microsoft was the cause, I assure you this global outage was not due to Microsoft. The root of the issue was a combination of factors, but not Microsoft's fault. The primary source was a software company called CrowdStrike, which provides computer security for much of the world. CrowdStrike released a version of their software that affected business computers running Microsoft Windows. Additionally, many IT departments, including those in the airline industry, failed to properly vet updates before allowing CrowdStrike to automatically update their systems. Talk about a case where automation backfired. It's truly unfortunate that so many people were impacted by this mass outage! #crowdstrike #outage #Microsoft #security

What's the buzz about a global shutdown of Windows and Microsoft? Alright, let me break this down for you in simple terms. So, there's this company called CrowdStrike that makes security software for computers. A lot of big companies use their stuff. Today, they messed up big time. They sent out an update to their software that had a nasty bug in it. This buggy update caused Windows computers to freak out and show the dreaded "blue screen of death." It's like when your computer throws a tantrum and refuses to work. But here's the kicker - it didn't just happen to a few computers. It affected businesses, banks, airlines, and even hospitals all over the world! Flights got grounded, banks couldn't do transactions, and some TV stations couldn't even broadcast. It was chaos! CrowdStrike's CEO had to come out and apologize. They're working on fixing it, but it's not easy. They have to go through each affected computer one by one to sort it out. The whole thing is a big reminder of how connected everything is these days. One little software hiccup can cause a global headache! #techoutage #crowdstrike #microsoft #bluescreenofdeath #azure #softwareengineering #corporatelife #bluescreenday

Y2K Bug Strikes 24 Years Later! I was into another kind of troubleshooting, working at the office this evening, and faced no system or network collapse. I thus had no reason to suspect that the world was in the midst of what I am tempted to call Microsoft Mayhem! CrowdStrike! Small mercies! Lucky me! When I did get the leeway to check BBC about two hours ago, I felt like a Rip Van Winkle. In a web-led world (www) news travels fast and furious and a few hours of detachment from it are enough to make one feel disconnected from what is raging. The world was already abuzz with news about the outage. The question that came to me instantly was: “Is the current outage a reminder of what was anticipated during the Y2K paranoia? I ran the question in Copilot, a Microsoft coinage and innovation, and it came up with an immediate and appropriate response: “Yes, the outage has stirred memories of the Y2K bug.” Screenshot under “Comments”. Overblown fears preceded the bug that did not bite. Now, in an AI-powered world, it is naive to expect a warning of any impending attack. Just like CrowdStrike struck from out of the blue, leaving the U.S. in chaos and creating ripples worldwide - Singapore, India and a few other countries have reported impacts too! #outage #CrowdStrike #microsoft #y2k https://lnkd.in/gpT6en3s

Hello to everybody! A bit late to this, but no one in the IT field can forget last Friday's 07/19 technical glitch. As a developer, before moving to production, I followed the following process: Debug and Run the Code: Ensure the code runs without errors. Functional Test: Confirm that all parts of the code work as intended. Unit Test: Use sample data to verify that everything is functioning perfectly. Impact Assessment: Ensure that the changes won’t affect downstream processes or other systems. Once all these steps are completed, the changes are moved to the UAT (User Acceptance Testing) environment. Testers then validate the latest changes, conduct a couple of test runs, and sign off on the release👨🏽💻. After these validations, we move the changes to production, with backup plans, if its fails we need to revert back to the earlier version💁. These processes are followed even for simple code changes💆🏾♂️! Considering a company like CrowdStrike , which updates patches in Microsoft OS, it’s critical. The incident caused issues in the kernel memory, impacting Windows OS for billions of users. They can’t simply run updates without thorough testing. I believe they should have alpha and beta testing phases to validate such significant changes. Everyone from Microsoft claimed that it wasn’t a cyber attack or anything malicious. However, the issue seemed more than a cyber glitch because it impacted systems worldwide🤷🏾♂️. #crowdstrike #microsoft #technology #glitch #system #malicious #testing

We've witnessed many events like last week's CloudStrike snafu before, but this one was particularly disruptive, especially in the post-COVID era where many organizations still have numerous employees working remotely. Kudos to all my colleagues and IT professionals who have spent countless work and travel hours jumping from computer to computer to fix the problems this update has caused. Regrettably, the responsibility here clearly lies with CrowdStrike, and it is likely that we will see numerous lawsuits in the coming months and years. Their software is designed to embed itself into the operating system's kernel to prevent interference during normal operation, such as malware or viruses attempting to disable it. So having Windows allow a replacement for a driver during a CloudStrike update makes sense. Having it cause a major crash - not so much. As IT leaders, we increasingly rely on software vendors to handle their own patching and testing before pushing updates to production. This is because it has become too cumbersome to test every patch ourselves, especially in large enterprise organizations. The critical question now is what software engineers can do to develop effective roll-back procedures during "official" patching in case of catastrophic issues. I anticipate that Microsoft is working on better compartmentalization to prevent the OS from failing to boot in similar future scenarios. However, Microsoft faces the challenge of dealing with the OS's legacy code, including policies and registry entries dating back to the Windows 3.51 era. I am eager to see what the future looks like, especially as we collectively pressure giants like Microsoft and CloudStrike Ventures to make significant changes to their platforms.  #Cloudstrike #Microsoft #CIOGenius CIO Genius

Incident response planning doesn't just apply to your 50 person team. The majors do it to. Microsoft met this week with government officials and security vendors this week to discuss how to ensure there is not another systems-stopping incident like this summer's Crowdstrike outage. They discussed plans "to design new tools that would help [partners] operate away from the Windows kernel." While there is currently not a timeline in place to take action on this meeting and these ideas, that is not the important part. Microsoft and partners are iterating in light of past experiences, an integral part to (1) learning from past incidents, and (2) preparing to circumvent future issues. Or at least be prepared with how to deal with them. Three take homes for you and your business: - When you last had a tech hiccup, what did you learn and what action did you take? - Do you have an incident response plan in place to protect your business, data, and reputation? - How are you working with your partners and clients to ensure they feel safe work with you as well? #IncidentResponse #TechStrategy #CybersecurityPlanning #BusinessContinuity #MicrosoftSecurity #Crowdstrike #DataProtection #SecurityVendors #LearningFromExperience #ITResilience #PartnerCollaboration #ClientSafety #RiskManagement #SecurityLeadership More on this story: https://lnkd.in/gnGmj85G

Not the Friday anyone in tech (or many other departments) wanted... https://lnkd.in/enmBRgsW We are constantly reminded by incidents such as this that the fundamentals are important and should never be ignored. Code reviews, regression testing, sandboxing, pre-production gates, etc exist for a reason. It will be interesting to see exactly what took place once the dust settles, and the litigation begins. Modern IT is complex, and the shift to cloud-based infrastructure only compounds this. That's not to say that it shouldn't be embraced, merely a reminder that you need to understand your fundamentals and have a robust BCP in place should the worst happen. Good luck to everyone battling for uptime today and over the weekend. #microsoft #crowdstrike #outage

Testing Disasters #28 Crowdstrike... oops. I think there's already quite a lot of information/posts out about this already. But obviously the main points are... Test Test Test Considering the number of systems worldwide that were affected by this release, it seems pretty obvious that they Crowdstrike did no testing at all on the deployment.. or the machines they did test on were so modified from a semi-standard Windows installation that they somehow did not get the problem to occur Never do a production deploy on a Friday (even if it is a "minor" change). Because if something goes wrong, everyone is going to be doing overtime to fix it. The problem was the classic NULL pointer error.. oops. Developer created a pointer that should have been to an object but was to empty space, so when anything went back to reference this object it couldn't find it and crashed.  Since as an anti virus it had low level access, Windows thought this was a problem and therefore just crashed the system. Marketing points as well from this problem.. When it came out, the CEO originally did not apologise for all the impacts and knock on effects this had to many people all around the world.. and the poor IT support staff who would have to go around manually fixing all the machines to correct this problem. Bad CEO. Your software screwing up might not seem that bad.. until you realise that airports, doctor offices and many other places might run it and then all the people that are affected in many different ways. Poor Microsoft.. it was not actually a Microsoft problem at all.. but because it happened only on Windows machines (the same kind of error could happen on any OS), they have been tagged with a lot of bad publicity from it. So my TL:DR - Test(!) on a vanilla/standard machine before you deploy anything to live (especially when it's to 100,000s of machines) - Don't do Production deployments before the weekend (and remember about Worldwide Timezones) - If your company screws up in a massive way, then at least come out with an apology! #microsoft #outage #crowdstrike #windows #testing  

Friday witnessed a major tech outage that left many people scrambling. The culprit? A seemingly "harmless" update that ended up causing blue screens and malfunctions for millions of devices. Is it just an 'unfortunate' incident or does it highlight the wider issues of interconnected systems?   We need to revisit our practices to ensure that what was just an accident doesn’t become a blueprint for malicious actors to exploit in the future:   🔏 Secure Software Supply Chains: Ensure your codebase remains free from vulnerabilities at every stage of development. ☁ Cloud Native Adoption: Scale and adapt quickly, minimizing downtime caused by unforeseen issues. Adopt shift-left practices to identify and mitigate vulnerabilities early in the development process. 🐧Linux Systems: Move to Linux for its stability and security features, reducing the risk of such outages. 💻 Modern Development Practices: Implement continuous integration and continuous deployment (CI/CD) to ensure updates are tested and rolled out safely. 🧪Automated Testing: Proactively identify and fix problems before they escalate into major outages. P.S. CrowdStrike Flow Source: JoElla Carman and Nigel Chiwaya / NBC News Quote: Chandrakanth {Chandu} P. #CrowdStrike #BSOD #Microsoft #TechOutage #SoftwareDevelopment #SoftwareSupplyChain #SoftwareSupplyChainSecurity