Thursday, April 30, 2015

Averting a Titanic IT Outage from Underground Cable Destruction

Vulnerable Underground Fiber Calls for Wireless Mixed Media Backup

You’re the captain of your corporate IT ship. You’ve thought it all out - your routes, requirements and resources, and you prepared for the worse by spreading your network out to different carriers. You even have different fiber routes so that if one fails you have the other. But are you really protected?
When your carrier tells you that they’ve got back ups to back ups and they’re (statistically) never going down, can you believe it? From what I’ve seen, I wouldn’t. Carrier outages happen more than we think, and while they’re hardly publicized, call some of your IT colleagues and ask if they ever lost a connection to their carrier or Internet provider.
I figure that Bloomberg must have the best IT safeguards anywhere, but their network crashed anyway. We don’t know what caused it, but no doubt those folks stayed up endless nights working to prevent such a disaster. But it still happened, and I wonder what stone was left unturned that might have mitigated the damage.
In a similar vein, I wonder how “mixed media backup” dropped from the telecom vernacular. It was a popular phrase in the 90's (pre-Google), and what it meant was that - as a policy decision, whatever terrestrial lines you had were backed up over the air (using point-to-point microwave). What makes more sense than that? Yet it’s far from common practice these days, even as IT managers lose sleep thinking of ways to plug network vulnerabilities.