If you know me, you'll see that I'm a high-energy person who loves to write about fun, upbeat, future-thinking topics around technology. While I'll try to bring that energy into this post, this is probably one of the more serious conversations we'll have.
I was hoping to save this topic for a bit later, but having just opened the 2023 AFCOM webinar on security, it felt like it was an appropriate time to stay on this subject. If you want to catch the recap of the AFCOM New Year, New Security webinar, you can find it here. However, almost as soon as I got off, I received several LinkedIn and personal messages asking me to sit down for a security chat. I realized something. Much of this industry is in the same boat regarding physical infrastructure security.
Look, we know that outages are expensive. According to the Uptime Institute's 2022 Outage Analysis, the consequences and cost of downtime are worsening, with 60% of failures now resulting in at least $100,000 in total losses. They also found that when significant outages happen, over 85% of the incidents stem from staff failing to follow procedures or flaws in the processes themselves.
But this is where we need to pause for a second. Yes, a cybersecurity incident is something scary. However, with physical security, we go beyond data loss; at that point, we're talking about impacts on people.
A broader approach to physical data center security
I had the chance to spend some time at a data center company (Switch) that takes security very seriously. If you know, you know.
But as we look at the broader data center landscape, physical infrastructure security is not all built the same. It became clear that physical data center security can become a blind spot for overall management and operations. That scope and visibility must evolve to keep up with the real physical threats against our infrastructure.
First, there's the cybersecurity element of physical infrastructure. "This overall data center inventory needs to be all-inclusive," Nasser Fattah, North America steering committee chair at Shared Assessments, told Data Center Knowledge, "including power, HVAC, fire suppression system, UPS, CCTV, et cetera, because these solutions may be connected to the IT and data network, in one form or another, which can become an unauthorized access point. To exacerbate matters, often IoT devices are not included in the patch cycle, leaving them vulnerable to exploits."
But what about physical security specifically?
OK, as I promised, I didn't want this to become a conversation around traditional cybersecurity and ransomware. Don't get me wrong. It's still imperative. But we're shifting gears for a second.
As I'm sure you're all aware, you've seen recent mainstream reports covering the attacks on physical infrastructure—specifically, the electrical grid. And you won't be surprised to know that federal government regulators have also been keeping a close eye on this. Just earlier this year, the Federal Energy Regulatory Commission (FERC) and the Department of Energy's Office of Cybersecurity, Energy Security, and Emergency Response (CESER) held a joint technical conference discussing supply chain risk management in light of increasing threats to the Bulk Power System.
The result of this meeting was felt quickly. A few days after the meeting, FERC directed the North American Electric Reliability Corporation (NERC) to re-examine its Physical Security Reliability Standard, CIP-014-1.
Congress responded as well to these emerging physical infrastructure threats. They increased CESER's budget by almost 7.5% and appropriated $20 million for the Cyber Testing for Resilient Industrial Control Systems program.
The conversation didn't stop at Bulk Power Systems in the joint meeting. It was also established that attacks against distributed energy resources (DERs), like edge power systems and electric vehicle infrastructure, are growing. In a new report titled Cybersecurity Considerations for Distributed Energy Resources on the U.S. Electric Grid, CESER identified the cybersecurity threat to DER operators, vendors, developers, owners, and aggregators as posing a significant and growing risk. According to the Department of Energy, you will see yet another new report identifying policies and procedures for enhancing the physical and cybersecurity of distributed resources and the electric distribution system.
From what we've seen in the attacks against forms of critical infrastructure, it has become evident that there are significant vulnerabilities. As a result, customers and the federal government are pushing the private sector to mitigate those threats as a condition for doing business.
"Data center operators have a unique opportunity to redefine Aristotle's famous "the whole is greater than the sum of its parts" concept. For example, building an on-premises microgrid (through investment or as-a-service) can help improve sustainability credentials while tackling the abovementioned vulnerability and enabling cost predictability. It could even help tackle not-in-my-back-yard sentiment by providing clean, low-cost energy to its neighbors, " said Vlad Galabov, director of cloud and data center research at Omdia."
But what about the deadbolts on our data center doors? Aren't those enough?
While we haven't explicitly mentioned data centers, these threats, and vulnerabilities still apply. Data centers are critical infrastructures. And data centers are a target. “If a lone attacker can take out a utility substation and disrupt power to a community, data centers have the same vulnerability,” states Alan Howard, Principal Analyst at Omdia. “A well-designed data center would ideally have a redundant dual-feed power and fiber architecture that would largely mitigate a full failure, notwithstanding an attacker sophisticated enough to understand electrical engineering and single points of failure.”
I want to reference some interesting, first-of-its-kind findings from the new AFCOM State of the Data Center report. For the first time, we saw physical security rank as the number one priority for DCIM and data center management implementation planning. Moreover, 80% of respondents either have or will be integrating security into DCIM.
Bill Kleyman, Data Center Knowledge contributing editor, and Data Center World program chair
Similarly, human threats (outside and inside) made the top 5 list of primary security and infrastructure Threats. This is the first time outside and inside human threat vectors have entered the top five. Security conversations don't need to happen in the shadows. "What we're now seeing is a shedding of the old ways and an adoption of the new," says David Ellis, Enterprise Solutions at Genetec, referring mainly to the growing pains that many data centers have experienced as they attempt to integrate physical security systems with their cybersecurity counterparts seamlessly.
Like the trial-and-error process that most companies experience as they attempt to merge physical data center processes with the cloud, seamlessly "hybridizing" cyber and physical security systems can be tough to accomplish within a tight timeframe.
"Modernizing data center infrastructure systems (hardware and software) is likely to enable more seamless connectivity and operations, but also the quicker rollout and administration of security policies. As newer data center infrastructure has better automation and alerting systems, it can help operators cope with the chronic staff shortages of the data center industry," adds Galabov. "This also addresses some of the security vulnerabilities caused by human factors. As vendors continue to strive to improve their product's efficiency, the modernization effort will likely result in significant cost reduction through energy efficiency gains."
"Better security, higher resiliency, lower costs, better sustainability credentials, automation, staff morale, and business resilience are all factors that boost customer satisfaction and acquisition, and thus revenue, proving that Aristotle was right," said Galabov.
This is an excellent time to take a quick pause. Data center pros that have read this far into the article are probably thirsty for some action items and steps, and I don't want them to be disappointed.
A new kind of data center deadbolt
As we discussed during our webinar, the time for mere "security awareness" is long past. While educating employees on proper security protocols is essential, it isn't enough to thwart all potential security breaches. "I cannot emphasize enough: do your homework," Ellis explained. "But make sure you follow through with actual actions."
I spoke with a few security leaders in the physical data center space. Before I go into what they talked about, use this time as a reflective moment to look at your physical security. Find one item to improve, and you'll already be ahead. With that in mind, here are four key points to consider when looking at physical security:
- You are a target—more than ever before. There's no sugarcoating this statement. We know critical infrastructure is under attack, and we've seen data centers and telecommunications infrastructure threatened. This won't change, unfortunately. Think very critically about your physical infrastructure security and be sure to adapt to changing times.
- Move away from real-time and reactive to proactive and predictive. The traditional Standard of Operation (SOP) has been very linear. An incident happens, you find out about it, you check the tape, review the data, and take some action. A lot can happen between the time something happens and your action steps. Modern security systems now include deep levels of integration with various security data points. This includes biometrics, opt-in facial scanning, and Vision AI integration for license plate validation. Systems can spot a threat or intruder frequently faster than a human can. Use these solutions to create a proactive architecture.
- Use data intelligence. Infrastructure organizations are adopting AI-driven, human-in-the-loop security systems to use data for physical security. The evolution of how we spot and respond to threats must change, and data is a powerful ally for physical security. It might be time to evaluate new solutions if you still use reactive-response camera systems. Finally, data intelligence and security integration have seen massive investments from DCIM leaders. Deeper security visibility is now a specific asking point when deploying DCIM into the data center. I do not doubt that DCIM will become a big part of the proactive nature of physical security.
- Secure your supply chains. I spoke with a government security expert that wished to remain anonymous. This person made it clear that a concentration of critical infrastructure will quickly become a target. And it might not be the building itself but the vital resources supporting it. This expert said you need a good supply chain capable of responding to emergencies.
I'm sure there are a bunch of other great tips out there. And if this is your area of expertise, please leave a comment. This security tide can help raise all data center boats. Most importantly, never stay complacent regarding physical data center security. Start conversations, reach out and ask questions, and evolve your security strategies to do your best to stay ahead of the risks.
