Volunteers Mikala Smith, left, and Hussan Abdulmagid remove debris from a home that was destroyed by an EF-3 tornado on Friday, May 23, 2014, in Duanesburg, N.Y.
The National Weather Service (NWS) issued a memo on Friday afternoon detailing the likely causes and lessons learned from the major computer disruption on Thursday afternoon, which knocked out the central automated system through which the agency issues life-saving severe weather warnings.
The memo says the data disruption was triggered by an upgrade to the weather data dissemination system itself, specifically a modification to a network firewall. The firewall was supposed to continue to allow data to pass through it, but "within minutes, engineers noticed that data were not traversing the firewall." The memo claims all warnings did reach the public despite the loss of the main automated delivery system, which is a questionable claim considering that many storm chasers, television meteorologists, online news sites and others in the weather community reported disruptions in crucial radar data and severe weather warnings on Thursday.
The outage, which the NWS now says lasted from 3:49 to 4:25 p.m. ET on Thursday (although the system was not declared fully operational until 4:37 p.m. ET), occurred as severe thunderstorms erupted from the Denver metro area eastward to the Mid-Atlantic. A tornado warning was issued at one point for downtown Denver, and one strong tornado touched down near Albany, New York. Another weaker twister struck in Delaware around the time of the outage.
"As you're well aware, we experienced an outage to a key part of our weather data dissemination system yesterday. The outage occurred on current operational NWS dissemination systems, despite the fact that proper procedures were followed," the memo read. It was sent to all NWS employees by Laura Furgione, the deputy director of the agency, and provided to Mashable by a NWS spokesman.
In other words, the agency is going after anyone who is outside the agency who may have released information about the outage that they learned via the NWS chat system.In other words, the agency is going after anyone who is outside the agency who may have released information about the outage that they learned via the NWS chat system.
On Thursday, screenshots of the nwschat system, showing forecasters coping with the outage and clearly demonstrating its severity, circulated among the press. (Mashable showed one of the screenshots, but is not a subscriber to nwschat.)
"Although processes and procedures were followed properly, we will use this event to improve the change process and procedures associated with operational NWS dissemination systems," the memo states.
Meteorologist Jason Samenow of the Washington Post's Capital Weather Gang blog reported that he did not receive some severe thunderstorm warnings that were issued for the Washington area, and therefore was delayed in covering them on the website. One of the warnings was never transmitted, Samenow said.
In the memo, Furgione praised the NWS staff for finding alternative ways to issue warnings.
"It was through your diligent work that any affected products, which were delayed or may not have been fully disseminated through normal, automated channels, did reach the public and emergency management officials through other means such as social media, email distribution lists and direct phone calls," she wrote.
The outage affected 41 watches, warnings and advisories, according to the memo. "30 products (including a tornado warning) were not delivered through the primary dissemination system, but were disseminated by other systems," Furgione said. "11 products were delayed approximately 5 minutes."
Furgione said that watches, warnings and advisories distributed through other means, including the NOAA Weather Wire that goes to emergency managers, and NOAA Weather Radio, were distributed properly.
"It was not a total loss of data flow out of the forecasts offices and NCEP, because there are several other dissemination channels that continued to operate normally," the memo stated, using the acronym for the National Centers for Environmental Prediction.
"The firewall systems do include backup and failover capabilities. However, the nature of the firewall outage prevented the failover capabilities from executing and the failover had to be forced manually to resume normal operations."
One remaining question is why, after such a high-profile technical failure that came in the wake of several other technical glitches, the NWS decided to issue an internal memo that was distributed to reporters, rather than hold a press conference or issue a more digestible public statement from the NWS director.