2020-06-07 15:26:34
At approximately 06:32 on 6 June, a significant outage has been detected on our network.

Incident Discussion:

Kradorex Xeron@2020-06-07 15:26:36:

At 06:32, an initial outage condition occurred with a recovery at 06:45. At 07:08 a second outage condition had occurred with subsequent notification from emergency services and the power company that there had been an explosive incident in a high-voltage power vault situated nearby our facility. An 8 by 12 city block grid had lost power at this point.

At approximately 07:45 I arrived on scene, smoke was emitting from the power vault. All power had been cut to the building complex. I gained access to our suite and all systems were de-energized due to safety cut-offs.

At approximately 12:00, the power company was able to access the vault and conduct repairs and by 13:30, power had been restored to our facility. I performed a controlled re-energization of our systems and conducted a damage inventory. All external communications were cut and one piece of communications equipment beyond our network border was destroyed as a result of a beyond tollarance surge condition. No damage was found within our network.

Through some on-site contacts, it was understood that multiple street level communications cabinets had experienced surge-related damage, many in the immediate vicinity of our facility were destroyed. Our upstream provider arrived at 15:30 and begun replacement work on that equipment. At 17:30, the equipment had been replaced and communications was restored.

At 17:45 I contacted our upstream provider and requested the in-suite communications equipment that was destroyed to be replaced. They informed due to workload, that wouldn't be possible until Monday-Tuesday. I manually transitioned our network over to a secondary backup connection and shifted network services over on a temporary basis. This was underway between 18:00 and 20:30. By 21:00, most services were online.

The replacement of the destroyed in-suite equipment remains outstanding at this time.

Kradorex Xeron@2020-06-08 11:54:02:

Upstream provider attended our facility and replaced the relevant equipment. We have transitioned our network from the secondary connection back to our primary. DNS updates are in progress.

2020-04-15 16:15:05
There is at present a packet loss condition on the upstream connectivity outside of our network.

Incident Discussion:

Kradorex Xeron@2020-04-15 16:15:06:

Confirmed. Appears to be a noise condition on the physical line. Upstream provider has scheduled a field technician call for tomorrow, 16 April.

2019-07-24 18:28:29
Beginning at approximately 01:00 (1 AM) our network has encountered routing issues with the upstream provider.

Incident Discussion:

Kradorex Xeron@2019-07-24 18:42:39:

At 01:30 arrived on-site. Determined to be an upstream issue. Upstream provider was also contacted to jointly conduct diagnostics.

At 01:45 it was determined to be a configuration issue between our primary edge router and our upstream provider. Diagnostics indicate that network internally is not experiencing any issues.

At 02:15, a configuration change was completed that provided an alternate means to get our network online. This change was successful, but required temporarily renumbering our network. This subsequently required various DNS changes and a modification to our DNS infrastructure.

At 03:00, DNS changes were underway and propagating.

At 04:00, Services begun returning to the Internet as accessible.

Kradorex Xeron@2019-07-24 18:46:31:

Provider will be arranging to complete a reconfiguration of our routing session so we may reinstall the edge router that had to be removed to complete connectivity restoration.

2019-07-19 01:17:42
Beginning approximately 16 July, an ongoing repeated disruption in our connectivity has been detected.

Incident Discussion:

Kradorex Xeron@2019-07-19 01:19:15:

Issue has been diagnosed and exists outside of our network. At around 17:00, we have been in contact with our upstream provider to provide correction. Provider truck roll is scheduled for 19 July during business.

Kradorex Xeron@2019-07-19 13:03:21:

Upstream provider has attended and has indicated that over the past two days, an alternate provider in the area running cabling cut the facility's main distribution line. Repair had been completed and provider has conducted diagnostics to ensure connectivity is stable.

A 24 hour monitoring period has been comenced.