CrowdStrike: Lessons Learned from ProCircular

Published in Member Communities on August 21, 2024

By Aaron Warner, CEO, ProCircular

On July 19, 2024, CrowdStrike released a routine software update for its Falcon endpoint detection system that caused a widespread IT outage affecting millions of Windows devices globally. The update was pushed directly to all systems running CrowdStrike, laptops, and servers and caused the machine to reboot into a loop. Machines that had taken the extra step of encrypting the hard drive included an additional step that required entering a 40-character "BitLocker Key" manually. It saves you from someone stealing a drive and mounting it on another computer - with BitLocker, it's not accessible without the key.

If that sounds horrible, that’s because it is. This often requires an IT professional to use a USB key to manually fix each machine. In a company with many distributed laptops, that's about as bad as it gets. Just imagine reading a 40-character alphanumeric key to someone whose native language is French.

If you know an IT professional, particularly an IT sysadmin or helpdesk tech, you should hug them and buy them a coffee.

As always, there were plenty of lessons learned and opportunities to improve.

The good news:

• People are taking steps to protect themselves, LOTS of them.
• CrowdStrike quickly identified and isolated the problem, deploying a fix within hours.
• No one lost anything other than time. It wasn't a cyberattack or business email compromise; no dollars or intellectual property were stolen.

It’s worth noting that CrowdStrike is a solid member of the cybersecurity community. They’ve been entirely transparent, pushed a fix quickly, and their response turned what could have been a week-long disaster into a weekend. CrowdStrike has contributed significantly to the world’s knowledge of hackers and their methods, and their CEO has made no bones about their mistakes.  ProCircular offers several good alternatives to CrowdStrike, but having any form of monitoring is better than nothing at all. 

Perhaps most importantly, this was a solid practice run for "Game Day" in cybersecurity. There's a saying that only two kinds of companies exist - those who've had a cyber breach and those who don't know it yet. This outage forced many mission-critical companies to practice the incident response plans they built and evaluate the successes and failures incorporated into those updated plans. The world is an increasingly volatile place, and these plans will be put to the test again in the coming years.

The bad news:

• The outage affected approximately 8.5 million Windows devices, causing significant disruptions across various sectors.
• Many organizations, particularly airlines, continued to experience delays and technical issues days after the initial incident.
• The financial impact of the outage is estimated to exceed $1 billion.

This update crashed critical infrastructure across the board, and companies in Iowa were just as affected as multi-national airlines. ProCircular fielded numerous calls from clients in healthcare, state and federal law enforcement, and several insurance industry members. All are back up and running; many are looking at alternatives to CrowdStrike. All of them asked us questions about how to avoid these issues in the future. 

Takeaways:

There are a few overarching themes to what we've learned:

• Critical infrastructure should not be so vulnerable to a single software update from one provider.
• Cybersecurity application providers must employ safeguards and implement rollback mechanisms for updates affecting boot drivers and core system components.
• Communication and coordination between cybersecurity providers and their clients during crises have room for improvement.

If you're still looking for how this applies to you or your organization, here are some considerations:

• Protect your endpoints, and don't consider EDR an unnecessary risk because of this outage. Ransomware is far worse than this sort of downtime. Imagine the same outage, but someone demands you pay a million dollars to regain access to your systems. There are alternatives to CrowdStrike, and EDR is an essential layer of your overall security program.
• Perform a Business Impact Analysis to identify critical systems and determine acceptable downtime for your unique organization. Whether a cyber-attack or a vendor-caused outage is the risk, these efforts will help you prioritize recovery efforts during an outage.
• Develop an emergency plan that outlines clear responsibilities, communication protocols, and recovery procedures. This plan should include:
  • A defined crisis team with primary and backup contacts for each service area.
  • Pre-defined conference lines for technical and business teams.
  • Communication templates for various channels (email, text, voice).
  • A crisis checklist to guide response efforts and document the resources you need. This will help your organization establish your cyber provider, legal counsel, law enforcement, and the right insurance contacts ahead of time so you’re ready on game day.

We're not entirely out of the woods, but by taking notes from the CrowdStrike debacle, we can improve our resiliency and better prepare for what comes next. While this one layer failed us and affected millions of people, the opportunities for improvement are worth at least that much in the long run. That which does not kill us makes us stronger. 

TAGS

1. business solution
2. cybersecurity
3. data
4. technology