Dealing with Hurricane Harvey as a Local Houston Telecommunications Provider

USA - English

• USA - English

(PRWEB) September 23, 2017 -- Hurricanes, like any natural disaster, are hard to predict. One thing that you can predict is that the more you plan and prepare, the better equipped you are to deal with the challenges. By the time news of Harvey’s advance into the Gulf of Mexico was announced, Houston businesses needed to have a Disaster Preparedness and Recovery Plan at the ready, because there wasn’t time to come up with one before it would affect the area. PS LIGHTWAVE takes its responsibilities of providing critical communications connectivity to area governmental agencies and first responders very seriously, especially during a crisis. PS LIGHTWAVE is a local Facilities-Based Telecommunications Provider, which owns and operates a fiber optic network encompassing over seven counties with over 5,200 route miles. Managing a vast metropolitan fiber network that is the equivalent of the distance between Miami, Florida and Budapest, Hungary, PS LIGHTWAVE’s performance during the storm was impressive with 98% of their network devices staying up 100% of the time and 100% network core uptime.

Rhonda Cook, President/CEO of PS LIGHTWAVE attributes the success to, “Having a very committed, experienced, and passionate team in place with good organization and armed with a well thought out disaster preparedness and recovery plan.” Hand-in-hand with a highly respected expert in the field of safety and disaster recovery, Leo Wrobel, the company developed a solid plan that it continues to review and enhance. PS LIGHTWAVE provides Ethernet data circuits, private fiber optic networks, Internet, and VoIP telephony to commercial enterprises, governmental clients (municipalities, counties, state and federal agencies, and school districts), and other telecom carriers, including cellular back-haul.

Pulling the Trigger

When Hurricane Season launched, PS LIGHTWAVE began hosting multiple in-house Disaster Preparedness and Recovery Plan meetings that included representation by all relevant staff. Days prior to Harvey, they ramped the frequency of the meetings up and started reviewing and discussing the Disaster Preparedness and Recovery Plan, as well as checking off the immediate tasks that needed to be in place. Implementation of the plan included putting procedures into place, securing additional materials, adjusting employee shifts, identifying internal and external back-ups for all critical business functions, and testing and activating the emergency communication plan. This included everything from updating employee, contractor, and vendor contacts, securing vehicles on high ground, making sure sufficient cash was on site at each business location in the event power outages hampered credit cash purchases for gas and other necessities, setting up a job cost number to track restoration costs, establishing advance network priority restoration plan, stocking food and toiletry provisions for employees, preparing an alternate command center site, assigning key lead positions and back-ups for each, establishing schedules and call in procedures for frequent conference calls with leadership team and all key operations staff, and releasing employees early, prior to the storm, to allow them to complete tasks for their own families.

During the Harvey event, updates were distributed every two hours internally by the Network Operations Center (NOC) to keep key personnel and leadership team informed and on the same page. Regular conference calls with key personnel were held throughout the storm event to strategize and prioritize efforts. Communications were sent out daily via email, website, and social media to keep network status updates readily available to clients.

Setting the Wheels in Motion

Once the DP&R Plan was activated, many activities were set into motion simultaneously. Some of the actions initiated and continued through recovery included, but were not limited to:

• Established Tree of Responsibility for Employee Contact – Each executive, manager, and supervisor was responsible for making contact with his or her direct reports each day of the storm to verify their personal safety/home status.
• NOC Staffing 24x7x365 – The local NOC shifts were adjusted to ensure continuous monitoring, troubleshooting, and dispatching capabilities throughout the duration of the storm and flooding aftermath. Backups were in place in the event some personnel were unable to make it in to always maintain phone connectivity and to be accessible to support customers.
• Daily eBlast, Website Alerts, and Social Media Posts – Daily communications through multiple platforms were sent to keep customers informed on the company’s actions and reassure them that crews were on top of restoring any damage to the network as swiftly and safely as possible.
• Proactively Called and Emailed Customers Individually – Personal check-ins with the clients were done during and after the storm to check on their status, personally and professionally, and let them know PS LIGHTWAVE was there for them and able to assist.

The PS LIGHTWAVE Director of People and Culture, Kurt Rushe, said, “Cross-training is a good practice, but in disaster planning it is imperative. We made sure that each role had a back-up ready to take over their responsibilities, when needed. You need to think beyond flashlights and bottled water. For example, we made sure we had a contingency plan to ensure that any of the potential pitfalls that could interfere with payroll processing were covered. Our staff depends on us to take care of their needs, just as they are taking care of our customer needs.”

Restoration After the Storm

Once Harvey had passed, dropping 51.88” of rain, the city also had to deal with man-made flooding with the releases of two of the major dams. Even with flooding and impacted routes, PS LIGHTWAVE immediately had network supervisors and route designers driving the fiber routes and visually inspecting the network and route accessibility.

Wali Muhammad, PS LIGHTWAVE Capital Construction Manager said, “There was a lot of water! It was a bad situation and hard to get around the city, so we had to find alternates ways around to get our customers up and running. There were some extremely bad areas, like Hwy 59 & San Jacinto River, Baytown, Interstate 45 & Cypresswood Dr, Memorial Dr & Eldridge Pkwy, and even some places where the roads had just collapsed. It was hard to see people trying to get their lives back together, so we tried to stay out of their way as much as possible while we did our work.”

The NOC also began to assess customer impact from the storm, flooding, and power outages while continually monitoring and tracking all power outages in the network. They responded to all customer calls and support tickets initiated by customers, plus opened tickets for all issues identified internally. They also began dispatching PS LIGHTWAVE crews to repair fiber and supply nodes with temporary generators when utility power was interrupted and service had switched over to UPS and/or back-up power systems

One of their nodes was located in a building right in the path of the convergence of the Addicks and Barker Cypress reservoir releases, and the building site had lost power. When they began the releases at 1am, a PS LIGHTWAVE Network Technician, Jayson Coones, was one of the first responders. Jayson recalled, “Even though we were in a large utility bucket truck, we had to closely monitor the rising water because we weren’t sure how high it would get. We knew we had to make it happen. We arrived at the building, and we set up portable generator in place to get the node back up and restore service. One employee stayed until the water level was rising to a level that would make the road no longer safe to travel by vehicle. We later worked out a plan to utilize a boat to deploy a portable generator. It was sad to see people displaced from their homes or trying to figure out how to get to their homes again over the next few days, but it was good to see so many people helping others, firefighters, police, and just people from all over that came to help.”

The PS LIGHTWAVE Fiber Splicing Manager, Jim Parker, used a jet ski to get to the node to refuel the portable generator for four days that the building was completely surrounded by water. Parker recalled, “It was kind of surreal driving a jet ski on a street you normally drive on. We had to go through police checkpoints set up to keep the curious out and they were trying to avoid looters. We wore waders to avoid making contact with the water and a life jacket, just in case we needed it. The waters finally receded enough that we were able to resume driving through the water with a bigger splicer truck to keep re-fueling until the power was finally restored.”

Post Mortem - Assessing and Preparing for Future Events

After the flurry died down and the network was close to 100%, with a few exceptions being handled with manually fueled generators, all key team members and the leadership team met to conduct a Post Mortem Assessment. In the meeting, the DP&R Plan implementation was reviewed to assess what worked well and according to plan, new methods and ideas that were implemented that needed to be added to the plan, near misses and what could be done to mitigate them in the future, and what didn’t work according to the plan and how to alter the plan to better address for future events. PS LIGHTWAVE advises that a post mortem assessment be held as soon as practical following every crisis or near miss with the intent of acknowledging what went really well, and more importantly, learning from what could be done better and incorporating ideas on how to step up to an even higher level of preparedness performance. Vice President of Operations, John Lambert, “Part of that process, should include recognizing and rewarding the performance of specific stand out team members to celebrate their achievements. We have planned a company event to include a presentation and acknowledgement of the team’s performance and highlight those individuals who shined during the event.”

For more information or interviews, contact:
media(at)pslightwave.com

About PS LIGHTWAVE

PS LIGHTWAVE, a leading telecommunications service provider headquartered in Houston, Texas, owns one of the nation’s largest private fiber optic Metropolitan Area Networks (MANs). The facilities-based fiber optic network spans approximately 5,000 route miles, features 1,500 on-net locations, and delivers managed Gigabit Ethernet services, Internet, Dark Fiber, Voice over IP (VoIP), and services to carriers, enterprise, cities and municipalities and educational institutions throughout Texas. PS LIGHTWAVE’s switched Layer 2 infrastructure is backed by 24/7/365 Network Operations Center (NOC) support and connects 50+ fault-tolerant multi-gigabit Ethernet rings for added redundancy, security, high-availability, and low latency. At PS LIGHTWAVE, Great Connections Happen Here. For more information about PS LIGHTWAVE, please explore our website or call 832-615-8000.

Robbie Adair, PS LIGHTWAVE, http://www.pslightwave.com, +1 (713) 581-4489, [email protected]