by Richard Sonnen, Sentinel Strategic Consultant

On July 19, 2024, a significant global IT outage occurred due to a CrowdStrike Falcon agent update that impacted Windows clients and servers. This event caused widespread disruptions, leaving many organizations scrambling to restore their systems and resume normal business operations.

As IT teams worldwide continue to work tirelessly to recover affected computers, the experts from Sentinel Technologies want to share some practical tips to expedite the recovery process and minimize downtime.

Tips for Speeding Up Recovery

Here are some of our favorite tips and tricks from Sentinel technicians and clients as they fine-tune their recovery efforts:

Utilize QR or Barcodes for Guided Recovery

By creating QR codes or barcodes that employees can scan with their smartphones, you can provide step-by-step instructions for the recovery process. This approach offers several advantages:

• Reduces the need for IT staff to personally attend to each device
• Ensures consistent application of recovery steps
• Allows employees to contribute to the recovery effort, potentially speeding up the process
• Provides a user-friendly interface for non-technical staff

Leverage Microsoft's Recovery Tool

Microsoft has released an updated recovery tool with two repair options:

• Recover from WinPE (recommended for most scenarios)
• Recover from safe mode (useful for certain BitLocker configurations)

Prepare for BitLocker Challenges

If your devices use BitLocker encryption, ensure you have recovery keys readily available. For devices using TPM-only protectors, the safe mode recovery option may be preferable.

Note that any BitLocker key that is recovered should be treated as compromised and rotated once the device is back in service.

Consider PXE Boot for Network-Wide Recovery

For organizations with devices across multiple subnets, setting up a PXE server can facilitate remote recovery without physical access to each device.

Prioritize Critical Systems

Identify and prioritize recovery of mission-critical systems to minimize business impact.

Communicate Clearly

Keep your team and stakeholders informed about the recovery process, expected timelines, and any actions they need to take.

Looking Ahead: Preparing for Future Outages

While the immediate focus is on recovery, this event serves as a stark reminder of the importance of robust Business Continuity and Disaster Recovery (BCDR) planning.

Sentinel Technologies is here to assist you not only in recovering from this current outage but also in developing a strategic BCDR/BCOR (Business Continuity and Organizational Resilience) plan. Our expertise can help you implement solutions such as:

• High Availability in the Hybrid Cloud
• Desktop as a Service for business continuity
• Robust change control and update management processes

Don't let this outage be just a painful memory – let it be the catalyst for building a more resilient IT infrastructure. Contact us or your Sentinel Account Manager today to discuss how we can help safeguard your organization against future disruptions and ensure your business stays operational, no matter what challenges arise.

Remember, in today's interconnected digital landscape, it's not a question of if another outage will occur, but when. Let's work together to ensure you're prepared.

Learn more:

• Navigating Global IT Outages: How Sentinel Can Safeguard Your Business Continuity
• Business Continuity Solutions After CrowdStrike
• Lessons Learned from the CrowdStrike Outage