Resolved - 10/27 Investigating Network Instability

[Quags]

We are aware of Network Instability on different routes. This is currently being investigated. More information will be provided as it is known.

Update 1:56 PM - Network admins believe they have isolated the cause instability and are applying fixes. Some core routers will be restarted.

Update: 2:24 PM - We believe this is fully resolved. A full RFO will be given once all the information is available.

 

[Quags]

At 12:00 PM EST the network located in Secaucus, NJ experienced unexpected and unplanned disturbances. Multiple routes were lost and BGP sessions with providers and clients dropped. Ten minutes after the drop all sessions restored automatically. Thirty minutes later routes dropped again and later restored. During the second drop, we logged an unknown Cisco bug that resulted in the line cards running out of memory. After further investigation more memory was allocated and the affected routers were rebooted one by one. All service was restored at 2:00 PM EST and has been stable so far. We will continue to monitor the stability of the network over the next 24 hours in order to ensure the problem has been resolved.