2019-07-24 18:28:29
Beginning at approximately 01:00 (1 AM) our network has encountered routing issues with the upstream provider.

Incident Discussion:

Kradorex Xeron@2019-07-24 18:42:39:

At 01:30 arrived on-site. Determined to be an upstream issue. Upstream provider was also contacted to jointly conduct diagnostics.

At 01:45 it was determined to be a configuration issue between our primary edge router and our upstream provider. Diagnostics indicate that network internally is not experiencing any issues.

At 02:15, a configuration change was completed that provided an alternate means to get our network online. This change was successful, but required temporarily renumbering our network. This subsequently required various DNS changes and a modification to our DNS infrastructure.

At 03:00, DNS changes were underway and propagating.

At 04:00, Services begun returning to the Internet as accessible.

Kradorex Xeron@2019-07-24 18:46:31:

Provider will be arranging to complete a reconfiguration of our routing session so we may reinstall the edge router that had to be removed to complete connectivity restoration.

2019-07-19 01:17:42
Beginning approximately 16 July, an ongoing repeated disruption in our connectivity has been detected.

Incident Discussion:

Kradorex Xeron@2019-07-19 01:19:15:

Issue has been diagnosed and exists outside of our network. At around 17:00, we have been in contact with our upstream provider to provide correction. Provider truck roll is scheduled for 19 July during business.

Kradorex Xeron@2019-07-19 13:03:21:

Upstream provider has attended and has indicated that over the past two days, an alternate provider in the area running cabling cut the facility's main distribution line. Repair had been completed and provider has conducted diagnostics to ensure connectivity is stable.

A 24 hour monitoring period has been comenced.

2019-07-05 15:17:03
A limited outage condition has been caused by the failure of a 120VAC breaker. Breaker was unable to be reset.

Incident Discussion:

Kradorex Xeron@2019-07-05 15:19:23:

For liability reasons, an external electrician had to be dispatched for the breaker replacement.

2019-02-01 03:42:21
At approximately 11:27 EST 31 January, external connectivity has been lost to our network.

Incident Discussion:

Kradorex Xeron@2019-02-01 03:55:28:

~14:00 EST - Initial analysis indicated possible power incident reported by the power company. This was disproven as our facility still had full power and aside from a surge event no further issues. Communications issue was suspect and confirmed with communication with our upstream provider who advised us of a street level incident.

Kradorex Xeron@2019-02-01 03:57:42:

18:20 EST - Connectivity restored. Network connectivity self-healed and all services resumed public online operation.

Initiating 24 hour monitoring to ensure stability.

Kradorex Xeron@2019-02-01 04:00:13:

19:52 EST - Connectivity lost again. Received report that street level incident had not been fully and correctly resolved. Upstream provider continued work on their end.

Kradorex Xeron@2019-02-01 04:02:24:

23:20 EST - Connectivity restored. Network connectivity self-healed and all services resumed public online operation.

Resetting 24 hour active monitoring to this point and resuming monitoring to ensure stability and correct resolution.

Kradorex Xeron@2019-02-11 23:03:39:

Connection is stable.