by forhire » Thu Sep 07, 2006 4:28 pm
As promised the full travel log:
At approximately 12:30 PM PST on Wednesday Sept 6th we briefly lost our private line between our Morton and Mossyrock locations. After resetting the interface service was restored fully by 12:50 PM. We thought all was well.
At 5:00 PM later in the day we lost connectivity again. This time the loop didn't recover. We contacted the Telco for assistance. At about 8:00 PM the telco informed us that the circuit would not be repaired until morning. It was suspected that a guard rail installation crew had crushed or severed their cable on a narrow stretch of Highway 12. It was going to be a long night.
During this time I was working to work around the issue without much luck. It turned out that our upstream had some static routing that wouldn't allow me to bypass our private line. By 10:00 PM it was decided to simply allow the access servers to take calls without authenicating. Partial internet was better than nothing.
During this outage e-mail and our website was not reachable from Mossyrock.
Early this morning I arrived to check the status. Poltelco was coming down and they arrived ans started work around 10:00 AM. It appears that a guardrail pilon had crushed the cable. Because this segment isn't fiber it took a while for the cable to heatup and burn thru. I suspect the they hit the cable at 12:30 and it finally burned at 5:00.
Poltelco finished their work and service was restored around 2:00 PM.
This outage took 22 hours off my life.
All service has been restored.
Thank you for your patience,
Randy