More Servers! Verizon Adds Capacity for HealthCare.gov
October 30th, 2013 By: Rich Miller
Can more hardware solve the problems plaguing HealthCare.gov? The troubled web site was taken offline Tuesday night after the Obama administration asked service provider Verizon Terremark to beef up the infrastructure supporting the site. Verizon said it was adding server and storage hardware to try and stabilize the online health insurance marketplace, which has been experiencing downtime and performance problems since it launched on Oct. 1.
“Since HHS asked us to provide additional compute and storage capacity, our engineers have worked 24/7 to trouble-shoot issues with the site,” said Jeff Nelson, vice president of global corporate communicationsfor Verizon Enterprise Solutions. “At the request of HHS’s deputy CIO, we are now undertaking infrastructure maintenance, which should be complete overnight. We anticipate the strengthened infrastructure will help eliminate application downtimes.
“Verizon is committed to supporting our HHS client and stabilizing their www.healthcare.gov website,” Nelson said.
The downtime follows an outage that began Sunday when Verizon Terremark experienced networking problems, which impaired communications between parts of the Healthcare.gov application. Verizon Terremark emphasized that the outage was caused by a network problem within the infrastructure supporting the site, rather than an outage for an entire physical data center.
The problems for HealthCare.gov are expected to be discussed on Capitol Hill today when U.S. Health and Human Services Secretary Kathleen Sebelius testifies before Congress.
It remains to be seen whether adding server and storage capacity can bring meaningful performance improvements to HealthCare.gov. Amid finger-pointing among contractors and the administration, outside analysts have suggested the site has major weaknesses in its architecture, poor coding and that the site was inadequately tested prior to launch. Web site performance experts say the site was poorly optimized to handle heavy loads of users.
“The trouble the government is facing when taking itself online is that people have come to expect the high level of availability and performance they get from Amazon and Google with every web site they use,” said Kent Alstad, Vice President of Acceleration at Radware, which specializes in application delivery. “In our experience supporting high volume, high performance web sites, this is a challenge for vendors of all sizes. This is a solvable problem that will likely take a significant effort to bring up to standards we expect.”
While writing this comment I just heard Sec. Kathleen Sebelius point strictly to Verizon. No mention of any other cause – that is so incredibly horrible. I myself, as a 1099′er went through the registration process. Clearly the network, storage or capacity can present issues, but it’s also VERY clear to me that code is the main problem (based on the behavior I’m seeing). I do not believe Sec. Sebelius is unaware of this reality and I find her testimony disturbing. Despite the political upheaval it is frustrating that a United States Secretary would knowingly leave out complete and relevant information and not take FULL responsibility for the problems. A rushed project is always going to end up in a train wreck – all of us here understand that. However you come out on ObamaCare we must agree that it was the political rush to implement that landed caused all this.
No private company would spend over $150 million dollars and not end up bankrupt after a failure like that. But, if you are in an organization that actually can print your money and have no real financial risk in failure, then you get failure. And, this anecdotal evidence is exactly and precisely why Government should NOT run ANYTHING that can be run in the private sector. Ok, so worse case scenario, if we have to have ObamaCare they should have at LEAST made the management and assets of the program competitive and in the private sector, and the contract terms renewable based on performance.
So, we should not be surprised. To be clear, I doubt the contractors hold main culpability, it’s no doubt rushed and shoddy Government Project Management.
PGTPosted November 1st, 2013
The networking failure appears to be bad maintenance/config on a core routing component that should be part of an HA pair (one side went down, the other failed due to config issues).
The subject of this article is the addition of new server and storage capacity; that’s clearly a Gov issue. 100% chance the vendor had everything ordered ready to go…a lack of capacity is directly traceable back to demand and how the gov predicted and procured to meet it. Anybody in IT knows it takes more than a few days to procure, deliver, configure and deploy on the raised floor.