xenogere

Bounced like a basketball

Posted on Jan 3, 2007 by jason

I apologize for the brief site outage this morning.  A combination of events caused it.

I began several updates to Apache and other server components around 11 a.m. CST.  My TTY session was buzzing along nicely when suddenly it froze—right in the middle of the Apache updates.  After giving it a minute to catch its breath as I assumed it was a server issue, I finally tried to connect via other methods and realized I was unable to reach the data center network where my server lives.

Realizing the console session had died in the middle of the server updates and that the network was unreachable, I immediately began the process of contacting technical support.  Before I was able to complete that, however, the network came back online and the server came back on the air.

When I connected to it and checked the update process, I found it had died midway through rebuilding Apache, so that had rendered the httpd services unstable and/or down (depending on which site I tried).  I killed all the processes and ran another Apache update.  I was thankful to see that one complete without interruption.

Due to the hiccups and instability of the services after the crashed update session, I went ahead and bounced the whole box to get clean sessions across the board.

Needless to say, a bit of network instability in the middle of a secure TTY session can cause problems if a forced logout occurs . . . and that’s precisely what happened.  I don’t know what caused the initial network outage but suspect it was something minor, ill-timed though it might have been.  Had that not happened, there would have been no outage at all since I can perform all of the major server updates without significant downtime (the only connectivity problems occur when services are restarting, but those moments last mere seconds only and generally pass unnoticed).

All of this did remind me of something John Gillmore once said: “The Internet interprets censorship as system damage and routes around it.”  Not that this had anything to do with censorship, but the brief issue with the network connection did rise and fall rapidly and without warning, so the whole thing reminded me of this quote and how it seems the internet gets rather displeased with the loss of access across routes and immediately shuffles things around to reinstate lost connections.

By the way, as I was drafting this post, the site became unavailable again for the briefest of moments.  I suspect there are network issues at play either in the data center or leading to the data center, but I will contact my tech support peeps and find out what’s going on.  If something major is happening, I’ll update this post with the details.

Permalink Trackback