2020-11-15 03:49:15
Environment Canada has published a weather warning for the area of the Digibase Network Operations Center (DB-NOC) facility in regards to a wind storm for today, November 15.

We are advised that wind speeds may gust to 90+ KM/h or ~60+ MPH during this storm. We do not anticipate an outage condition, but impacts to operations are possible during this dangerous storm. Efforts will be made as soon as it is safe to ensure the restoration of services ASAP if this is the case.

2020-06-07 15:26:34
At approximately 06:32 on 6 June, a significant outage has been detected on our network.

Incident Discussion:

Kradorex Xeron@2020-06-07 15:26:36:

At 06:32, an initial outage condition occurred with a recovery at 06:45. At 07:08 a second outage condition had occurred with subsequent notification from emergency services and the power company that there had been an explosive incident in a high-voltage power vault situated nearby our facility. An 8 by 12 city block grid had lost power at this point.

At approximately 07:45 I arrived on scene, smoke was emitting from the power vault. All power had been cut to the building complex. I gained access to our suite and all systems were de-energized due to safety cut-offs.

At approximately 12:00, the power company was able to access the vault and conduct repairs and by 13:30, power had been restored to our facility. I performed a controlled re-energization of our systems and conducted a damage inventory. All external communications were cut and one piece of communications equipment beyond our network border was destroyed as a result of a beyond tollarance surge condition. No damage was found within our network.

Through some on-site contacts, it was understood that multiple street level communications cabinets had experienced surge-related damage, many in the immediate vicinity of our facility were destroyed. Our upstream provider arrived at 15:30 and begun replacement work on that equipment. At 17:30, the equipment had been replaced and communications was restored.

At 17:45 I contacted our upstream provider and requested the in-suite communications equipment that was destroyed to be replaced. They informed due to workload, that wouldn't be possible until Monday-Tuesday. I manually transitioned our network over to a secondary backup connection and shifted network services over on a temporary basis. This was underway between 18:00 and 20:30. By 21:00, most services were online.

The replacement of the destroyed in-suite equipment remains outstanding at this time.

Kradorex Xeron@2020-06-08 11:54:02:

Upstream provider attended our facility and replaced the relevant equipment. We have transitioned our network from the secondary connection back to our primary. DNS updates are in progress.

2020-04-15 16:15:05
There is at present a packet loss condition on the upstream connectivity outside of our network.

Incident Discussion:

Kradorex Xeron@2020-04-15 16:15:06:

Confirmed. Appears to be a noise condition on the physical line. Upstream provider has scheduled a field technician call for tomorrow, 16 April.

2019-07-24 18:28:29
Beginning at approximately 01:00 (1 AM) our network has encountered routing issues with the upstream provider.

Incident Discussion:

Kradorex Xeron@2019-07-24 18:42:39:

At 01:30 arrived on-site. Determined to be an upstream issue. Upstream provider was also contacted to jointly conduct diagnostics.

At 01:45 it was determined to be a configuration issue between our primary edge router and our upstream provider. Diagnostics indicate that network internally is not experiencing any issues.

At 02:15, a configuration change was completed that provided an alternate means to get our network online. This change was successful, but required temporarily renumbering our network. This subsequently required various DNS changes and a modification to our DNS infrastructure.

At 03:00, DNS changes were underway and propagating.

At 04:00, Services begun returning to the Internet as accessible.

Kradorex Xeron@2019-07-24 18:46:31:

Provider will be arranging to complete a reconfiguration of our routing session so we may reinstall the edge router that had to be removed to complete connectivity restoration.