Comments on: Virginia IT Systems Lack Network Redundancy

By: Blogrotate #8: The Weekly Roundup of News for System Administrators | The Pythian Blog

Fri, 27 Nov 2009 20:58:21 +0000

[...] Virginia IT Systems Lack Network Redundancy is a scary little number. According to the article, “…in just five weeks this fall, the Virginia DMV suffered 12 computer system outages, putting individual offices out of business for a total of more than 100 hours”. [...]

By: PGT

PGT — Tue, 24 Nov 2009 16:20:24 +0000

I should mention that I used to work for the carrier that had a VITA contract for network connectivity in Richmond. I believe they lost it a few years back, possibly with Northrop coming into the picture. I believe much of the IT staff we worked with at VITA got let go (due to the Northrop contract??), though my memory might be foggy on that as it wasn’t my account and it’s been several years now.

By: PGT

PGT — Tue, 24 Nov 2009 16:16:59 +0000

the state has access to route maps? So what? Unless they have a map of the ‘as built’ for the solution they rely on, that means nothing. One has to investigate the CLR (circuit layout record) from A to Z points to verify the path a circuit takes. This is necessary even if you’re using two carriers – you want to make sure there’s no single path of failure. At the end of the day, you’re dealing with the company that owns/maintains the cable plant (Verizon most likely in VITA’s case) so any carrier providing a circuit relies on them for a least the ‘last mile’ unless they do their own build into the facility (like Cox or Cogent or similar business model).

By: William Russell

William Russell — Tue, 24 Nov 2009 14:58:00 +0000

The problem is not the lack of available redundancy, the problem is who is paying for the redundancy. Apparently the state never bothered to demand a readily available feature – there are plenty of telecom access points near this site and it is only a question of connecting to the points. That still costs money – maybe running new lines – and involves somebody thinking about it to begin with. Both parties could be at fault, but Northrup isn’t that dumb.. Northrup had an obligation to raise the issue at the design stage and most likely did. Virginia had an obligation to raise it at the proposal stage for all parties consideration. However, I’m inclined to agree with G-Man above, probably both parties knew, winked, and moved on.

As to carriers “open disclosue” of routes, this is the state not a commercial client. Somebody with the right hand should ask the left hand what is going on. The state has access to route maps that commercial clients don’t. I’ve seen them.

By: Mark MacAuley

Mark MacAuley — Tue, 24 Nov 2009 14:07:59 +0000

My experience is everyone wants Tier IV/uber redundant until they see the price tag. Listen, this could be a number of issues and as for no redundancy in Northrup’s network, there has to be redundancy. Wherther or not configurations were correct, etc. – there are too many possible points of failure as others have pointed out. Bottom line is they will get it fixed and this will be a non-issue shortly…

By: PGT

PGT — Tue, 24 Nov 2009 13:48:26 +0000

I think the author pulled out his ‘Jump to Conclusions’ mat to link VITA’s woes to Cloud Computing. Seriously?

As for redundancy, most carriers won’t go ‘open kimono’ on disclosure of actual routes for a circuit until after an outage. They see it as ‘proprietary’ info and only the most determined customer will get their due diligence honored. I’ve been on the carrier side of the table – a hosting customer wanted a redundant circuit and spent money for a second demarc in his building. He wanted a concatenated circuit from us as a backup to his primary bandwidth. I tried to get him the route maps but Legal refused. Fast forward a year and one of our techs was doing maintenance on a switch and unplugged the live side of his circuit (thinking he was working on the secondary path), taking the guy down.

He demanded (and got) his route map when threatening to sue for breach – turns out, the circuit was diverse except for one 1/4 mile section where both strands rode the same conduit to go under a bridge (the company was too cheap to install a new path). The shocker was that this collapsed section was from the Lucent 5ESS switch site and POP….every circuit coming out of that site was exposed should there be a failure on that short stretch of road.

By: G-Man

G-Man — Tue, 24 Nov 2009 01:57:04 +0000

These seems to happen most often when value engineering comes into play and the cost of the redundant circuits on a ring topology, turns into a lateral or linear topology. I would bet the state or NG struck the redundancy and felt good to save the money when someone with the State did not complain since the apps may have been resilient without them. Obviously it was not stress tested in 05…

By: nate

nate — Tue, 24 Nov 2009 00:23:13 +0000

Multiple carriers of course often doesn’t give you complete redundancy. AT&T has a funny slide showing a cut trunk that carried something like 4 of the biggest carriers(including them), all in the same trunk, so all 4 were lost when the cable was cut.

And in my experience at least it can be pretty much impossible to determine how diverse the path is from the carriers. There are so many shared exchanges.

I bet most commercial data centers are the same, they may advertise multiple carriers and stuff, but I bet in most cases it’s not too far upstream from them that a single cable/structure can take out all of those carriers simultaneously to that facility.

But I suppose for a shop the size of the one mentioned in the article it could be one of those CYA situations.

I wonder if the Northrop people actually did the work and determined that multiple carriers wouldn’t buy them much so they didn’t implement them.

By: Austin

Austin — Mon, 23 Nov 2009 20:56:20 +0000

With the buying power in that contract, it should be easy to get two carriers and use lots of VPNs.