The Pulse
The official blog of Sentinel Technologies
Lessons Learned from the CrowdStrike Outage
by Mike Easter, Sentinel Senior Manager of Enterprise Architecture and Innovation
IT teams all over the globe continue to execute a tedious manual process to recover each computer impacted by the CrowdStrike outage. As they sacrifice critical time, people, and resources while business processes remain impaired, it is incumbent on business and technology leaders to immediately begin the capture, analysis, and integration of the two primary lessons learned in the hope this might not happen again.
Lesson 1
The IT Supply Chain will continue to be a significant risk for organizations of all types. Governance processes such as Software Quality Control, Change Management, CI/CD, and Release Management must become a major point of accountability for producers.
Likewise, consumers of these products must remain vigilant as they wrap their own quality control, change management, vendor contract requirements, and vendor management policies, procedures, and risk mitigations into business operations. It has now become painfully evident that additional care and management is required with regard to the products, platforms, and vendors everyone leverages to do business with. One “perfect storm event” has the potential to devastate organizations at such a level it becomes impossible to recover.
Lesson 2
Business Continuity Planning and Preparation just jumped even higher on the priority scale for organization and technology leaders. We have now seen a relatively innocuous “error” disable a sizable part of the global business economy. What if this had been a malicious action instead?
Organizations must now plan for mass outage events, from any cause, so they can rapidly resume operations when something unexpectedly strikes at the ability to function and generate revenue. The good news is that this does not have to be difficult or expensive to accomplish!
If the companies impacted had made a few low-cost, strategic investments prior to the CrowdStrike outage of July 2024, this incident could have been reduced to little more than a hiccup.
High Availability in the hybrid cloud, with change control and update management.
By maintaining a failover instance of critical systems in the hybrid cloud, an organization may rapidly spin-up these “standby” systems and resume business when something disrupts normal functionality of the data center. Data can be synchronized into the environment and kept in a constant ready state for the organization to use. When a problem occurs in the data center, IT simply spins-up the standby systems, redirects the connections, and business resumes.
Desktop as a Service, as part of the Business Continuity and Disaster Recovery Plan.
Endpoints can also be rendered unavailable for use. By maintaining ready-to-go desktop capabilities in the cloud (Desktop as a Service), organizations enable their employees to access corporate workloads and tools from any device, with the added benefit of no organizational data touching the device. This rapid restoration of your end users' ability to function as normal during the recovery process minimizes any potential financial, productivity, and reputational damage.
Sentinel has the experience and skills to help your organization advance its Business Continuity and Disaster Recovery Plan, so you can bounce back quickly and easily when the next unexpected outage or incident occurs. Contact us or reach out to your Sentinel Account Manager today to start the conversation!