It’s just aware of everything; it sees it all. The RAS is in charge now.
Fast forward a couple years. It’s 2013; things have improved for Georgia. There were no more country-wide blackouts, but there were still outages. Outages that, in some cases, were larger than necessary simply because each controller only had regional visibility. Now that the power system was more stable, GSE could start reducing these outages even further.
Time for Phase Two.
International Projects Technical Manager,
Enter: the SEL Special Protection Systems team.
Krishnanjan “Krishna” Ravikumar, Ashish Upreti, Tyler McCoy, Roberto Costa, and Brian Clark specialize in designing and extensively testing solutions for large- and small-scale protection systems. Using proactive controls and contingency-based actions, their solutions integrate into a customer’s existing system to help eliminate the core causes of blackouts.
In this phase, the challenge became how to minimize a power outage during a fault; how to add more control; how to add more visibility, more intelligence, more functionality. For GSE, this meant a remedial action scheme (RAS).
“The great thing about a RAS system is that it sees more information,” said Clark. “All of the information from every device around the country, all of that information is shared and brought up to the centralized controller.”
A RAS system is defined by centralized management and control. It shows the bigger picture of how electric power migrates and behaves from generation plants to load centers throughout the country, and it uses this expanded visibility to make more informed, intelligent, and quantitative decisions about the best way to keep the power system stable.
All devices are unified under the RAS controller, which sits in the National Control Center, collecting information from every device throughout the country. It interfaces with the existing SCADA system to monitor voltage, power flow, generation, and load levels. It automatically modifies power thresholds based on operator settings. It sees every action from primary and secondary equipment. And it sees it all in real time.
Where an emergency control system has only regional visibility, a RAS system has total visibility.
“This is how we move a system from emergency control to true management,” said Dolezilek. “Before, each relay was only able to take care of one line. It’s a great system for emergencies, but a RAS system is more capable and sophisticated. It makes finite decisions about which loads to shed and which to keep to best match a particular unbalance of power.”
The ability to make these precise decisions comes from adding more points of control. SEL and GSE now had more area to consider than just The Backbone, which meant they needed to expand the number of tripping thresholds and transmission lines taken into account. So, from the existing emergency control system, the new RAS system grew.
“The second phase didn’t have a lot of hardware updates in the substations because that groundwork was already laid out by John, Diego, and Fernando,” said Costa. “We added a few more SEL-451 and SEL‑2440 devices to enhance data gathering and control options—just small expansions where we needed to take more load into account for the RAS controller to make the best decisions."
But there was one significant update. Significant because of the functionality and speed it provided for GSE and their system. Significant because it’s one of a kind. The multiplexer that changed everything.
The SEL ICON.
“It’s different than anything else on the market.”
“No other device comes close.”
“It gives GSE the speed they need.”
“An ICON—it’s everywhere, right now.”
“It guarantees protection-class message delivery; no other device can do that.”
Dolezilek, Upreti, Clark, Calero, and more—they all agree on the pivotal nature of the ICON in a RAS system.
The ICON Integrated Communications Optical Network is a wide-area network (WAN) communications solution that combines multiple Ethernet, serial, telephone, video, current differential, and hard-wired local-area network (LAN) signals into one time-division multiplexing (TDM) fiber connection. It links all the devices together with each other and the RAS Controller.
During Phase Two, all existing Ethernet switches were disconnected from the WAN. In their place, an ICON network.
“With the ICON, we could add all these other capabilities to their fiber-optic cable in addition to Ethernet without jeopardizing the quality or performance of the protection and control messages,” said Dolezilek.
The TDM technology in the ICON works like a data pipeline system with several parallel pipes. Different message types are assigned to their own, dedicated “pipes” on the network—one pipe for serial protocols, one for Ethernet, one for protection messages, etc. Within each pipe, all communications are time-deterministic, meaning they are guaranteed to arrive within a specific time. So, no matter how much network traffic there is, those messages are protected. With traditional Ethernet, some messages can be assigned a higher priority over others, but there’s no guarantee they’ll arrive when they need to because all the communications share one pipe.
The new ICON network covers almost all the substations in Georgia.
And it’s fast.
“The message transit time between the RAS controller and any other relay in that country is probably less than a millisecond,” said Clark. “And that’s because of the ICON.”
The communications infrastructure and speed in a power system are critical because that’s how all of the protection and control messages are delivered. They need to arrive at the specified locations within certain time limits or the system will black out.
“When you lose a generator or transmission line, you can’t be waiting 10 seconds or even 300 milliseconds to start shedding load because then it’s over,” said Upreti. “Everything’s blacked out.”
The RAS system designed by the Special Protection Systems team runs every 2 ms, every hour, every day. No exceptions. It’s constantly checking for faults, monitoring and analyzing power flow, and checking for messages from relays. With a protection system that fast, the communications network needs to be able to keep up. The ICON network’s communications signals travel in approximately 1 ms, so it does the job. But, that’s when everything is working correctly. What happens if there’s a fault in the communications system?
That’s when something called the failover speed becomes paramount. Failover speed is the time it takes a communications device to detect a problem, like a fault in the network, and find an alternate path to continue delivering those critical protection and control messages.
Most modern multiplexer technologies are not designed with failover speeds that meet the requirements of mission-critical protection systems. Their failover times are 50 ms or slower, which is fine for business applications, streaming movies, and sending emails. However, for GSE and other critical infrastructure facilities, 50 ms is too slow.
The ICON fails over in less than 5 ms. This is what truly sets it apart from the rest. The ICON detects, isolates, and recovers from a communications system fault so fast that it gives the RAS system enough time to still detect, isolate, and recover from a power system fault in the usual 24 ms.
“Typical multiplexer technologies, which have 50-millisecond failover speeds at best, wouldn’t be able to meet that 100-millisecond time requirement from GSE,” said Dolezilek. “And certainly not the 24 milliseconds we routinely accomplish. It’s simple math. Their recovery times are longer than our entire process cycle.”
Without a device as fast as the ICON, the RAS system would try, but fail, to maintain stability in the presence of a communications failure.
“That is why the ICON is unique, that is why the ICON was created, and that is why the ICON is necessary for RAS systems,” said Dolezilek.
Just like every power system, every RAS design is unique. There is no “set process” to design a RAS system because the requirements and conditions of operation are different for each power system.
RAS designs are made up of logic equations, also called algorithms. These logic equations represent the customer’s specific requirements and conditions that the RAS system must fulfill in their power system.
The RAS logic equations must work for all specified conditions or scenarios, meaning they should keep the power system stable year-round, no matter what.
One triggering scenario might be a fault on a 220 kV transmission line. If that happens, how should the RAS system respond? Another condition could be a fault on a 500 kV transmission line in April with three generators offline for maintenance. How should the RAS system respond then? Or, how about a fault on a hot summer day in August with all generators running at top capacity? There are also contractual requirements to consider, which define places that should never lose power, like hospitals and military bases.
“We ended up with 68 triggering scenarios based on GSE’s requirements that would result in the RAS system taking an action to keep the power system stable,” said Costa. “Each triggering scenario is a contingency, and each contingency requires different control actions from the RAS system to keep the power system stable. Even further, each unique combination of simultaneous contingencies requires different control actions.”
Once the Special Protection Systems team converts all of the customer requirements into logic equations, they program the equations into the relays, RAS controller, and other devices to prepare for the simulation testing.
This is the fun part.
“Simulations, I mean, for every field—weather, aeronautics, power systems—that’s the coolest part,” said Ravikumar. “This is the part where you get to push the boundaries, the limits, really verify what the system can do; see how it responds, gather the data, do the analysis. This is when you really come to understand your system on a whole other level.”
SEL uses a Real Time Digital Simulator (RTDS®), the largest in the United States, to build a model of the customer’s power system to test the logic for all given scenarios.
Building the simulation for GSE was different from any other project the Special Protection Systems team had done before.
“It was their entire country,” said Costa. “We had to become very familiar with their country, the challenges that they had, how the power flows, the most critical scenarios, and then we had to come up with workable solutions for the power system reaction for each of those scenarios. So, it was a lot of meetings and emails and cooperation between our team and GSE. We had to gain their perspective in order to come up with logic to keep the system stable while respecting all of their contractual requirements.”
Normally, the Special Protection Systems team designs RAS systems for industrial systems or segments of a larger system. They might model an oil refinery or just one utility’s coverage area.
“Modeling an entire country made us rethink our process because it was not that simple,” said Ravikumar. “We had to come up with some new, solid fundamentals and philosophies.”
The group used the opportunity to expand their internal processes for larger systems. They documented every step, every roadblock, every lesson learned. Since Georgia, they have modeled Bahrain, UAE, Turkey, Uruguay, Vietnam, and Peru. With each country-wide scheme, they continued refining their approach. What began as a four-month process dropped to three months, then two months, then just six weeks to model an entire new country.
Once the model is built, the Special Protection Systems team connects the actual protection, monitoring, and control devices to the simulator in the test lab and runs through every case.
“This is where we identified some ‘boundary’ areas in the logic,” said Ravikumar. “These were areas of operation that hadn’t yet provoked a power system failure, but were coming dangerously close.”
When the RAS system takes an action in response to a fault, it has to make sure that a load-shedding decision in one area doesn’t overload lines in another area as a result.
“It’s our job to look at the power system from a stability standpoint, from a dynamic standpoint, and make recommendations to improve the logic,” said Ravikumar. “Without running simulations, you don’t always know, exactly, how the control actions will affect the rest of the power system. But we can see that. We look at the power system as a whole and figure out what to do differently to save it from a blackout.”
This might all sound so simple, so easy. Just plug in some logic, hook the relays up to the simulator, and see all the answers displayed nice and clean. But it’s not that simple and it’s not so obvious.
Those making the recommendations to change the logic need to do so with a thorough understanding of the intricacies of power system operation and the total effect any change will have.
“If you advise a customer that they need to change their logic, you better have strong, fundamental reasoning behind that,” said Ravikumar. “They come to us because we have the experience with different power systems and configurations. They are trusting us and our expertise to make the right call. So we’re really careful before we recommend changing the logic. We do our homework.”
‘Doing their homework’ often meant spending 18 hours a day for three months in the RTDS lab, testing and revising the GSE power system model.
“It’s actually a really good team building experience,” said Upreti. “Yeah, we’d work long nights, but it was fun; a lot of people get to know each other well.”
“We ended up eating a lot of pizza,” said Clark.
“It was always the pizza, Chinese takeout—really anywhere that would deliver in town at all these crazy hours,” said Costa.
“By the end of the project, the delivery drivers all knew exactly what we wanted and where to find us,” said Ravikumar.
All the months of late nights, testing, delivery food, retesting—all of it was in preparation for the Factory Acceptance Test (FAT).
In January 2014, representatives from GSE arrived at SEL Headquarters and stayed for two weeks to see the system simulations for themselves in the RTDS lab.
This was a chance for the customer to see exactly how their new protection system would respond to their predefined conditions and scenarios before being installed. The Special Protection Systems team set up the room, complete with whiteboards, desks, and three large screens displaying the RTDS power system model, the interface to the RAS controller, and the new GSE synchrophasor system.
“Together with GSE, we selected about 80 of the most critical scenarios to test,” said Costa. “After each test, we documented the system behavior and response, and we discussed any concerns or questions from GSE. We talked them through every decision and reasons why.”
One of the unique things SEL does for their customers is “free-form” testing. Once a customer has seen their system simulations in action, it often leads them to think of new “what if” scenarios. These might include testing how their system would behave if they added a new generation source or rearranged the load distribution in the future. It’s an opportunity for the customer to run the show and experiment with these future possibilities.
“Ashish, once he’s all set up, he can take the entire power system and give it a new configuration and run a new simulation event is less than an hour,” said Clark. “He can do that all day long for any conceivable scenario. And GSE kept pushing that, which was great because you can’t always do that in real life. So there were a lot of really good findings that came from that.”
During free-form testing, GSE was able to see how far they could push their system and still operate safely. Any time their simulated power system blacked out from these new scenarios, SEL and GSE went to the whiteboard and figured out how to fix it.
“Free-form tests are on the spot, so you don’t know how the system will react; it’s an experiment,” said Ravikumar. “All the times the system blacked out, we’d fix the logic right there with GSE and retest the system until it worked. No one leaves until it’s all figured out. That’s the good part.”
The other good part for GSE was learning how to mitigate these events with SEL in a safe environment. They could retest their power system over and over, and the people of Georgia and their power system would never be affected.
At a surface level, the two weeks of simulations gave GSE the assurance that their new system would work after it was commissioned. But on a deeper level, the simulations also gave them a better understanding of their power system as a whole and the intricacies of how every device relates to the others. It made them even better experts.