GridEx VII Reveals Continuing Challenges of Post-Covid Hybrid Work, Communications and Technology

In November 2023, the North American Electric Reliability Corp. (NERC) and its Electricity Information Sharing and Analysis Center (E-ISAC) conducted the seventh biennial GridEx.

“GridEx is one of the largest and most comprehensive security drawings that are done in any critical infrastructure sector,” said Manny Cancel, senior vice president and CEO of E-ISAC. “It has been going on for more than 14 years, and the purpose of the drill is to really exercise and stress our incident response plans in the face of serious cyber and physical security scenarios.”

GridEx consists of two components. The first, called Distributed Play, is a two-day span of simulated incidents ranging from cyber and physical attacks on substations to disinformation on social media—all designed to test utilities’ security and resiliency systems as well as their plans and response. While most participants were electric utilities, there was also participation from natural gas and water/wastewater entities.

“With Distributed Play, we really do try to break the grid and then discuss how we would put it back together,” Cancel said. “And whether it is improving ways to communicate, improving restoration priorities, correspondence and collaboration with other critical infrastructure sectors, and so on, it’s really important because we all rely so much on the energy sector.”

The second component of GridEx consists of an executive tabletop that gathers industry executives and government leaders from around the United States and Canada to explore the challenges presented by cyber and physical attacks against the electric grid and electric market system.

“With the Executive Tabletop—which takes place after the Distributed Play—we bring together the CEOs of the sector, some other key CEOs, as well as the federal government and agencies from the United States and Canada to have some real deliberate conversations,” Cancel said. “They talk about what is needed to handle very serious scenarios and use the information from the drills to flush out those strategies.”

When the simulated dust from Distributed Play settles and the Executive Tabletop conversations end, NERC issues a report that provides security and resiliency improvement recommendations to the industry. These recommendations come from post-event survey data that is distributed to participants.

Distributed Play Recommendations
The recommendations that came from the GridEx VII Distributed Play exercise revealed a continuing emphasis on cyber threats, including those that have come from the post-COVID shift to hybrid work.

“We are a couple of years removed from the COVID-19 pandemic, and the impacts of that have introduced some challenges in terms of how we staff things,” Cancel said. “In some respects, it’s helped us because people working virtually across a large geographic area in some ways replicates what would happen in a real-world attack. So, it helps us learn how to coordinate across the entire continent.”

Other recommendations included focusing on improving communications through better technology and better systems that reduce confusion between various levels of the organization. To view the full report, visit https://bit.ly/gridexVII.

Key Recommendations from Distributed Play and the Executive Tabletop

Distributed Play Recommendation:

Non-federal government partners and electric utilities should advance coordination efforts.
While the National Incident Management System advocates that emergency management coordination with governmental partners go from the “bottom up,” starting at local, up to state and so on, GridEx VII revealed that these collaborative efforts really need to be further analyzed and tested to ensure that the response communications are truly effective. The report noted that one planner realized that they struggled to effectively communicate individually with each of the many counties they partnered with and that a state-level coordination would likely be more effective.

Executive Tabletop Recommendation: 

The industry should evaluate opportunities to employ alternate technologies for operator voice (i.e., interpersonal) communications essential to operating the grid.
Participants agreed that, in the absence of all other automated or manual means to retrieve and share telemetry information needed to operate the grid, effective and efficient operator-to-operator voice communication is essential. For example, RC and BA operators need to communicate with transmission operators (TOPs) to coordinate switching and generator operators to direct dispatch. TOPs need to communicate with local utility operators who perform switching at the distribution level.

Distributed Play Recommendation: 

Communications and response in a hybrid work environment should be further refined.
The post-COVID impacts were still felt in the second GridEx that has happened since the pandemic as participant results revealed there are still more efforts needed to thoroughly define best practices for remote work. Participants revealed communications issues, lack of backup location plans and issues with software access for those who need it. One participant revealed that 911 calls made could be routed to the wrong law enforcement territories.

Distributed Play Recommendation: 

Response planning should be augmented to ensure comprehension of technical information across functional teams and external response partners.
As the saying goes, “Communication is the key to success.” This proved to be true during the Distributed Play exercise where some reported that there were comprehension issues in trying to communicate technical information to both non-technical and internal technical groups. One participant noted that they realized the terminology used would vary across different utilities, which created confusion.