Current local time is 08 May 2021 @ 03:45:33 EDT

Outages ticket CT-347893

Resolved
CT-347893
Emergency
6 June Outage

2020-06-07 15:26:34
At approximately 06:32 on 6 June, a significant outage has been detected on our network.

Incident Discussion:

Kradorex Xeron@2020-06-07 15:26:36:

At 06:32, an initial outage condition occurred with a recovery at 06:45. At 07:08 a second outage condition had occurred with subsequent notification from emergency services and the power company that there had been an explosive incident in a high-voltage power vault situated nearby our facility. An 8 by 12 city block grid had lost power at this point.

At approximately 07:45 I arrived on scene, smoke was emitting from the power vault. All power had been cut to the building complex. I gained access to our suite and all systems were de-energized due to safety cut-offs.

At approximately 12:00, the power company was able to access the vault and conduct repairs and by 13:30, power had been restored to our facility. I performed a controlled re-energization of our systems and conducted a damage inventory. All external communications were cut and one piece of communications equipment beyond our network border was destroyed as a result of a beyond tollarance surge condition. No damage was found within our network.

Through some on-site contacts, it was understood that multiple street level communications cabinets had experienced surge-related damage, many in the immediate vicinity of our facility were destroyed. Our upstream provider arrived at 15:30 and begun replacement work on that equipment. At 17:30, the equipment had been replaced and communications was restored.

At 17:45 I contacted our upstream provider and requested the in-suite communications equipment that was destroyed to be replaced. They informed due to workload, that wouldn't be possible until Monday-Tuesday. I manually transitioned our network over to a secondary backup connection and shifted network services over on a temporary basis. This was underway between 18:00 and 20:30. By 21:00, most services were online.

The replacement of the destroyed in-suite equipment remains outstanding at this time.

Kradorex Xeron@2020-06-08 11:54:02:

Upstream provider attended our facility and replaced the relevant equipment. We have transitioned our network from the secondary connection back to our primary. DNS updates are in progress.

*** Please note any mentioned ticket numbers when contacting Digibase regarding an above incident. ***