CrowdStrike: Lessons Learned from ProCircular

Published in Member Communities on August 21, 2024

ProCircular Blog

By Aaron Warner, CEO, ProCircular

On July 19, 2024, CrowdStrike released a routine software update for its Falcon endpoint detection system that caused a widespread IT outage affecting millions of Windows devices globally. The update was pushed directly to all systems running CrowdStrike, laptops, and servers and caused the machine to reboot into a loop. Machines that had taken the extra step of encrypting the hard drive included an additional step that required entering a 40-character "BitLocker Key" manually. It saves you from someone stealing a drive and mounting it on another computer - with BitLocker, it's not accessible without the key.

If that sounds horrible, that’s because it is. This often requires an IT professional to use a USB key to manually fix each machine. In a company with many distributed laptops, that's about as bad as it gets. Just imagine reading a 40-character alphanumeric key to someone whose native language is French.

If you know an IT professional, particularly an IT sysadmin or helpdesk tech, you should hug them and buy them a coffee.

As always, there were plenty of lessons learned and opportunities to improve.

The good news:

  • People are taking steps to protect themselves, LOTS of them.
  • CrowdStrike quickly identified and isolated the problem, deploying a fix within hours.
  • No one lost anything other than time. It wasn't a cyberattack or business email compromise; no dollars or intellectual property were stolen.

It’s worth noting that CrowdStrike is a solid member of the cybersecurity community. They’ve been entirely transparent, pushed a fix quickly, and their response turned what could have been a week-long disaster into a weekend. CrowdStrike has contributed significantly to the world’s knowledge of hackers and their methods, and their CEO has made no bones about their mistakes.  ProCircular offers several good alternatives to CrowdStrike, but having any form of monitoring is better than nothing at all. 

Perhaps most importantly, this was a solid practice run for "Game Day" in cybersecurity. There's a saying that only two kinds of companies exist - those who've had a cyber breach and those who don't know it yet. This outage forced many mission-critical companies to practice the incident response plans they built and evaluate the successes and failures incorporated into those updated plans. The world is an increasingly volatile place, and these plans will be put to the test again in the coming years.

The bad news:

  • The outage affected approximately 8.5 million Windows devices, causing significant disruptions across various sectors.
  • Many organizations, particularly airlines, continued to experience delays and technical issues days after the initial incident.
  • The financial impact of the outage is estimated to exceed $1 billion.

This update crashed critical infrastructure across the board, and companies in Iowa were just as affected as multi-national airlines. ProCircular fielded numerous calls from clients in healthcare, state and federal law enforcement, and several insurance industry members. All are back up and running; many are looking at alternatives to CrowdStrike. All of them asked us questions about how to avoid these issues in the future. 

Takeaways:

There are a few overarching themes to what we've learned:

  • Critical infrastructure should not be so vulnerable to a single software update from one provider.
  • Cybersecurity application providers must employ safeguards and implement rollback mechanisms for updates affecting boot drivers and core system components.
  • Communication and coordination between cybersecurity providers and their clients during crises have room for improvement.

If you're still looking for how this applies to you or your organization, here are some considerations:

  • Protect your endpoints, and don't consider EDR an unnecessary risk because of this outage. Ransomware is far worse than this sort of downtime. Imagine the same outage, but someone demands you pay a million dollars to regain access to your systems. There are alternatives to CrowdStrike, and EDR is an essential layer of your overall security program.
  • Perform a Business Impact Analysis to identify critical systems and determine acceptable downtime for your unique organization. Whether a cyber-attack or a vendor-caused outage is the risk, these efforts will help you prioritize recovery efforts during an outage.
  • Develop an emergency plan that outlines clear responsibilities, communication protocols, and recovery procedures. This plan should include:
    • A defined crisis team with primary and backup contacts for each service area.
    • Pre-defined conference lines for technical and business teams.
    • Communication templates for various channels (email, text, voice).
    • A crisis checklist to guide response efforts and document the resources you need. This will help your organization establish your cyber provider, legal counsel, law enforcement, and the right insurance contacts ahead of time so you’re ready on game day.

We're not entirely out of the woods, but by taking notes from the CrowdStrike debacle, we can improve our resiliency and better prepare for what comes next. While this one layer failed us and affected millions of people, the opportunities for improvement are worth at least that much in the long run. That which does not kill us makes us stronger. 


TAGS

  1. business solution
  2. cybersecurity
  3. data
  4. technology

From Our Experts

Enhance HME Training with Microlearning Solutions thumbnail Enhance HME Training with Microlearning Solutions Discover how microlearning can transform training in the home medical equipment industry, improving compliance, engagement, and performance with VGM Education. CMS Proposed Rule: What It Means To O&P Providers thumbnail CMS Proposed Rule: What It Means To O&P Providers Watch this update from VGM's Melanie Ewald, VP of Payer Relations and Reimbursement, on the potential impact of the CMS proposed rule on O&P providers. Ronda's Reflections: Four Hot Topics Affecting the DME Industry thumbnail Ronda's Reflections: Four Hot Topics Affecting the DME Industry Read about upcoming NCD implementations, oxygen equipment audits, lymphedema compression garment compliance, and PTAN billing tips. A Letter from the President: Planning with Purpose thumbnail A Letter from the President: Planning with Purpose As we turn the calendar to October, I imagine many of you are taking a deep breath, looking at what's left of 2025, and asking, “How do we finish strong?” Boosting ROI with Effective Home Medical Equipment (HME) Training thumbnail Boosting ROI with Effective Home Medical Equipment (HME) Training Maximize ROI in home medical equipment (HME) with strategic training programs that enhance compliance, customer satisfaction, and streamline operations. Steve Cranna of VGM Canada Announces Retirement thumbnail Steve Cranna of VGM Canada Announces Retirement Steve Cranna of VGM Canada has announced his retirement as Director of Government Relations and New Business Development at the end of September 2025. Jeff Kessler Named New Director of OPGA thumbnail Jeff Kessler Named New Director of OPGA OPGA is proud to announce that Jeff Kessler will be the director of OPGA effective October 1, 2025. The Impact of Continuous Glucose Monitoring in Managing Diabetes thumbnail The Impact of Continuous Glucose Monitoring in Managing Diabetes Discover how continuous glucose monitoring (CGM) is revolutionizing diabetes care & its benefits of real-time data, trend tracking, & improved health outcomes.