The Ripple Effect

The Far-Reaching Consequences (and Lessons) of CrowdStrike’s Error

Our lives are dependent on technology and the ripple effects from one company’s error will remain for weeks, months, and years to come.

Consider the impact in people’s daily life, Small-Midsize business (SMB) where they have limited support in recovery, to CrowdStrike legal implications and financial impact to their investors. 

We cannot stop wondering...

1. Could this have been avoided?

2. What in the world was CrowdStrike doing?

The comments and post incident info continue to pour in, but to avoid finger pointing or misinformation, we must focus on the first question above to achieve the goal of cyber resilience.

The ripple effects are just beginning for SMBs, whether as end-users or solution providers, that were affected, and we are (or will be) all victims. It’s crucial to raise awareness among your employees.  Phishing attempts, appearing as authentic Microsoft alerts, have notably increased (see the screen grab of a phishing attempt that I received).

Be vigilant of such deceptive emails with subtle discrepancies, like an extra “dot” in the address.

Knowledge and Awareness is the best and most cost-effective solution to any business.

I recently had an email conversation with Sheryl Root, Director and Associate Professor for Carnegie Mellon University, (and Netswitch Advisory Board) and the purpose of our conversation was to understand the incident, how such an event impacts otherwise unrelated businesses, and how do we explain all of it while keeping the topic verbiage simple, not technical, for CMU’s Executive MBA students. 

The context in this dialogue is to encourage companies to shift from a Proactive Strategy to an Adaptive Strategy integrated with Governance, Risk, & Compliance (GRC) to align the existing I.T. & Cybersecurity solutions and define KPI and business priorities.  How to collaborate with staff and supply chain partners to assess cyber risks. Consider the contents of risk “baskets” and the potential operational and financial impacts.

You are best to do this with a Business Impact Analysis (BIA) of your business. (If you’re a DIY’er and would like a free BIA Template, join our Group to download a Business Impact Analysis template).

The BIA Template will help you answer, “What is in our Business Continuity Plan to immediately minimize the impact?”  Maybe your personal iPad from home can become your business production tool(??).

Conclusions…

• Keep It Simple Stupid (We all know K.I.S.S.)

• Cyber Resilience becomes the priority.

• Trust But Verify 

Here is the email conversation…

Sheryl: How would Netswitch and your SARA (Security And Risk Assessment) help alleviate some of this?  Through better info analysis on the existing system and processes?

Stanley: Hottest topic in the world of IT. Our perspective is about cyber resilience with business continuity plan starts with “What do we have in that ONE basket?” And how do we recover it?

 Avoiding having all eggs in one basket can be expensive, but there are cost effective ways to mitigate that risk, such as layers of redundancy with multi-cloud or hybrid (with an external storage), or multi-vendor, multi-OS in the CrowdStrike case.

 Sheryl: Why did this happen?

Stanley: CrowdStrike, security software used by businesses to safeguard Windows machines from cyberattacks, pushed out a faulty update.

At this point, there are assumptions that this may fall into the category of “Human Error” - someone didn’t follow the update release policy and procedure, they pushed the “Enter” button too fast.  

With that as an assumption, GitHub or other development tools have workflow safeguards with layers of protection to make the procedures visible with management approval checks and balances, so it shouldn’t affect production environments with new patch releases.   

In the end-user’s side, especially on the critical production networks, why can’t they test out any new patches before release?  Are they concerned with the Zero-Day Exploitation, so they update the end devices as soon as the patch is released? This is what we will discuss with the Business Impact Analysis, and the solution will depend on the company resources and their priorities. 

As this was unfolding last week, our engineers in Asia were getting the calls first, and thinking it was a Windows desktop issue. After we could not recover the local desktops (Windows OS), we connected them to Remote Desktop in the Cloud with the Mac and Linux OS where we could and with as many as we could.

Sheryl: Why did the update fail?

Stanley: Similar to above – we think Human Error and All Eggs in One Basket.

We recommend that companies shift from a reactive approach to a proactive and preventive one. The next question then becomes, “In the event of a solution failure, what is our backup plan?”.

We all know there are no guarantees in life. So, we need to articulate one or two more steps for businesses to take so they can set up continuity solutions.

For instance, Netswitch Technology Management we designed a network for a client with their head office in San Francisco and several branches across the US and Asia. If the San Francisco office is incapacitated and can’t recover within two hours (e.g., due to an earthquake), operations would switch to a cloud infrastructure in the Central Time Zone, enabling users to work remotely or from home. If both the Central and West cloud locations were compromised, operations would then switch to the Microsoft Cloud in Asia, with the Asian office handling operations.

We consistently advocate for this three-layer tactical process.

Sheryl: How did that mismatch happen? Is there too much trust in cybersecurity and where’s the transparency and accountability.” 

Stanley: LOL! Goes back to Data Visualization with Visible Resilience. To Netswitch, where is the “Sweet Spot” for the company’s resilience maturity? Will your company survive if a third-party supplier fails?

IT & Cybersecurity are not much different to the Logistic Industry during pandemic, and how much it affected and cost business with the port congestion? Which leads to the companies’ insurance policy and coverage. Our discussion involves CrowdStrike’s legal liability, and last I heard it’s $1.3Billion financial impact and counting. 

 As you know, I can go on, but I’ll conclude with my personal observation.

Business executives and mid-level management need a proactive approach, starting with going through a Business Impact Analysis to identify: 1) potential production impact, 2) potential financial repercussions, 3) operational gaps or deficiencies, 4) resources and skillsets to execute unbiasedly, 5) know how to define the KPI’s in measuring and monitoring. 

My question to your class would be…"Can a company afford to conduct business as usual after seeing this cyber meltdown, do they understand that this was yet another warning sign?”

If they cannot, what are they doing about it?

I am very curious to learn from what your CMU Exec students have to share.

See you next week in class.

Stanley