Authorize.net has since been able to restore its web site and most of its processing services through a backup data center. But with key electrical systems out of action for an indefinite period, building staff are turning to portable generators to provide an interim solution for tenants, who are being told that power may be restored sometime Friday evening.  

“Fisher officials, CP Management and Egis Engineering have been coordinating with Seattle City Light to restore power to the building,” building owner Fisher Communications aid in a statement. “Fisher is bringing in electrical generators to restore power to the building, at which time it can further assess the situation. The company is working to restore normal service to its customers as soon as possible.”

Fisher said the electrical fire started at 11:10 p.m. Pacific time Thursday in a garage-level electrical room where Fisher Plaza East receives its power from Seattle City Light. “The cause of the electrical fire remains under investigation, and the sompany is working closely with Seattle City Light and outside experts to determine what caused this incident,” said Fisher.

There are first-person reports from the scene from TechFlash and a blog post from Jeremy from GeoCaching about his visit to the building. Blogger Kyle Mulka is compiling a list of affected sites.

An account on the NANOG mailing list reports that the incident triggered the sprinkler system in an electrical equipment room. As noted in the NANOG report, water damage could complicate recovery efforts

This is the second major outage at Fisher Plaza in the past year. Last June many sites at the building were offline after an electrical event at the building. Fisher Plaza previously had a major outage in 2006 (more details here on that incident). The owner of the building, which also houses the operations of Seattle TV and radio stations, has been attempting to sell Fisher Plaza. 

The incident is being discussed at Web Hosting Talk and on Twitter.

RELATED STORIES:

• Seattle Telecom Hub Fisher Plaza is For Sale
• 2008: Outage at Fisher Plaza Impacts Redfin
• 2006: Data Center Outages Bring Headaches, Headlines

An unusual series of equipment failures contributed to Monday’s power outage at a Rackspace data center near Dallas, in which several parts of the facility lost power. The most significant failure involved a bank of generators that malfunctioned, leaving several computing clusters without backup power. Other failures involved equipment that connects the data center to its two utility feeds, a UPS system and a transfer switch. The outage affected two of the four phases of data center space in the facility in Grapevine, Texas.

“We sincerely apologize for this disruption and know that it impacted our customers’ businesses as well as the experience of many who use the web,” Rackspace CEO Lanham Napier said Tuesday in summary on the Rackspace blog. “Although we have had some issues with this data center before, please know that we will do what it takes to improve its reliability and performance. We owe you an action plan to prevent this type of thing in the future, and we’ll get that to you as soon as it is ready.”

The DFW data center is the company’s largest facility, with 144,000 square feet of space. The facilty in Grapevine figured into a 2007 power outage that interrupted service for many prominent web sites.

An analysis of the “root cause” of the outage has not yet been completed. But Rackspace made a preliminary incident report public after it was posted online. In an update Wednesday afternoon, Rackspace said it is “making progress in understanding what caused the interruption. We have our suppliers and external consultants onsite working with us on this process. We will continue to provide status updates as we learn more.”

Data centers are engineered to avoid a “single point of failure.” When a facility loses power, it is usually due to a combination of failures. That was true at Rackspace, which experienced problems at multiple points in its power infrastructure. The incident was triggered when the breaker on the data center’s primary utility feed tripped, and the facility switched to generator power. The generators then failed to hold load properly.

“What we saw yesterday was a situation where the generators started fighting with one another on the bus,” Rackspace said of the generator challenges. “The generators were unable to get properly synchronized. Eventually, they failed in a cascading manner and we lost all of the generators. Each generator failed on a loss of excitation – an inability to maintain the magnetic field. But it was really the inability to get synchronized that created that fault.”

In an update Thursday afternoon, Rackspace said it would conduct maintenance early Friday morning on its main utility breaker and generators, which will mean shifting some customers to generator power from midnight until about 6 a.m.

The Google App Engine status page, showing the higher latency seen today (July 2).

It’s been a rough week for uptime. Google App Engine has been struggling with performance problems for hours, and appears to be down. The problems began at 6:30 a,m, Pacific time, when App Engine began experiencing high latency and error rates. “All applications accessing the Datastore are affected,” Google said in a notice to developers. Shortly afterward the service went into “unplanned maintenance mode” and began operating as read-only, meaning developers couldn’t update their apps. “Our engineering teams are looking into the root cause of the problem,” Google said.

The App Engine Status Page is currently unavailable. The App Engine team is providing offsite updates via Twitter. UPDATE: At 3:15 Eastern, the status page is back: “Datastore read access has been reenabled and the team expects Datastore write access will be reenabled shortly.”

The data provides only a small snapshot of how companies are using Amazon’s cloud, and doesn’t include the use of EC2 for back-end and back-office systems or for research. The use of EC2 for web site hosting became viable for major sites in March 2008, when Amazon introduced static IP addresses and “availability zones” to provide a backup web site should one instance or data center experience trouble.

It should also be noted that a sample of the top 500,000 web sites represents a small slice of the larger web. Netcraft’s Web Server Survey found more than 238 million web sites in June.    

The data offers fuel for cloud computing’s boosters, and its detractors as well. Although the economics of EC2 are perhaps more attractive for large-scale computing jobs than hosting web sites, the data suggests that a number of sites with meaningful traffic are using EC2 as their primary hosting provider.  

At the same time, the 0.3 percent market penetration by the cloud computing standard bearer could be seen as evidence that the adoption of cloud hosting is still in its early stages, and that cloud hype is running well ahead of implementation.

And what of InfiBase and its motivations?  The company describes itself as an “early stage startup working on a revolutionary solution for cloud computing.” While it’s in stealth mode, the company is offering cloud data analysis. “Although we can’t hope to estimate Amazon’s financials, I hope that our numbers can cast some light into the size and growth of Amazon’s elastic compute cloud,” writes InfiBase’s Guy Rosen.

Facebook VP of Technical ops Jonathan Heiliger discussed some of the launch management strategies at last week’s Velocity 2009 and Structure 09 conferences, including the use of a “dark launch” and the decision to go live with the username feature at a period of low traffic.

But Cook digs into some interesting “just in case” scenarios the Facebook team developed to manage a variety of traffic and web site load scenarios. “Facebook comprises hundreds of interlocking systems, although to users it’s presented as a simple web page,” he writes. “Throttling back the behavior of certain facets allows us to lighten the demand on our infrastructure without compromising major site functionality.”

Facebook was prepared to take drastic steps if the username land rush challenged its capacity.  

“In the event that Facebook became overwhelmed with traffic and suffered performance problems as a result, we also prepared for what we called ‘Nuclear Options’ such as cutting off nearly all the functionality on the Profile page, turning off Facebook Chat, and completely disabling the Home page,” Cook writes. “Any of these options were an absolute last resort to keep the site functional as they would have resulted in a severely degraded user experience.”

None were needed, as Facebook smoothly handled more than 200,000 username registrations in the first three minutes, and more than 1 million in the first hour. “Through the entire launch we had no issues handling the additional load,” Cook reports.

The downtime in Sydney occurred at the Equinix SY1 data center, which suffered a power loss during a utility power outage that affected 200,000 customers over five Australian states, including New South Wales, Victoria and Tasmania. The downtime disrupted VoIP phone service for much of the region. Equinix SY1 is a 65,000 square foot facility. The company’s SY2 data center was unaffected.

“The SY1 incident occurred at 10:56 AM local time and resulted in the loss of power to customers in one section of the center for approximately 12 minutes,” Equinix reported. “The outage was caused by the failure of a UPS system to deliver power after the utility provided power was interrupted in a statewide power outage with Energy Australia. This UPS unit serviced approximately 1/3 of the center. 2/3 of the center was not impacted. Within 12 minutes of the initial loss of power, all systems were restored.”

The power outage at the Equinix PA2 Saint-Denis IBX Center caused downtime for  several prominent customers, including the video site DailyMotion and the Paris operations of the European web hosting provider ClaraNet. Early media reports say the facility-level loss of power was brief, but some customers sites are taking longer to recover.

“The PA2 incident occurred at 11:33 AM local time and resulted in the loss of power to customers in one section of the center for approximately one minute,” Equinix said, confirming the brevity of the outage. “The outage was caused by the human error of a vendor that was conducting routine maintenance on a UPS system. The error caused one UPS unit to go offline, impacting the delivery of power to approximately 1/3 of the PA2 center. 2/3 of the center was not impacted. Within one minute of the initial loss of power, all systems were restored.

“Throughout both incidents, Equinix executives were immediately engaged to manage the most expeditious possible resolution,” the company said. “Our top engineers and vendors were immediately on-site to: first, restore power; second, determine the root cause of the outage, and; third, enact appropriate measures to ensure against the recurrence of related issues. Our customer service and sales staff were immediately engaged to notify customers and assist with specific issues caused by the incidents.”

The first phase of the project will feature an $800 million investment in a 35-megawatt data center, with a second $800 million, 35-megawatt phase to follow. The initial phase is currently in the design stage, with construction scheduled to begin in June 2010 and be completed by March 2013, according to documents (link via Mark Fontecchio). Project specs call for a Tier III raised-floor facility.

The Utah project was enabled by a new funding bill signed last week by President Obama, which also provides money for an expansion of the power infrastructure at Fort Meade, Maryland where the NSA’s primary data center is located. The NSA’s operations at Fort Meade have reportedly been power-strapped since 2006, when the agency maxed out the electric capacity of the Baltimore Gas & Electric power grid.

The NSA facility will be the third huge data center to be built in Utah, which has prevailed in several of the most competitive data center projects in the Western U.S. In May 2008 Oracle Corp. (ORCL) said it would build a $285 million data center in West Jordan. In December West Jordan was also selected as the site for a $334 million eBay data center.

Between them, the three projects represent more than $2.2 billion in data center investment in the state of Utah.

That’s not to say that cloud computing won’t impact the data center business. It will drive a gradual shift in who runs the facilities that host some of your apps and data. Data center infrastructure will continue to evolve - a process that predated the emergence of cloud computing - influencing which data centers become obsolete and what kind of new facilities are built.

Having said that, the growing interest in cloud computing has business implications for subsets of data center users and businesses. That includes the web hosting industry in general, and one hosting product in particular, according to Randy Bias at Cloudscaling.

“There is an imminent threat to hosting companies of any kind that provide VPS services today,” Randy writes, referring to virtual private servers. “Larger clouds, including the AWS behemoth and Rackspace, are already launching and preparing to launch price competitive services to capture the VPS market. The threat is serious in that not only do these clouds provide value based on pricing, but they also provide the traditional values of cloud computing around self-service.”

Virtual private servers are hosting accounts that run inside a virtual machine (VM) on a hosting company’s server. VPS accounts offer more resources and security than a shared hosting account, but less than a dedicated server. Hosting companies can run multiple VMs on a server, with each housing a customer account.

Conceptually, it’s a short hop from a VM on a hosting provider’s server to an instance on a cloud platform. As such, the VPS market may be an early indicator of whether customers can get comfortable with cloud computing.

CloudSwitch also said it has hired John McEleney as its Chief Executive Officer and President. The Burlington, Mass. company has 18 employees and a product in alpha testing, with a beta phase scheduled for later this year. Company executives are saying little about its product beyond the fact that it is software-based and will provide an “enterprise gateway to the cloud.”  

McEleney previously was CEO of SolidWorks Corporation, a 3D modeling company with $350 million in revenues. “Cloud computing is a major platform shift and the CloudSwitch approach is unique in addressing the needs of enterprise applications and environments,” said McEleney. “The level of interest we’re seeing from prospects, partners and investors is a reflection of the strength of our value proposition and the exponential growth of the cloud market.”

The CloudSwitch management team also includes executives with experience at BMC, EMC, Netezza, RSA and Sun Microsystems.

“Having worked closely with John McEleney for many years, we are excited to back him and the CloudSwitch team,” said Elliot Katzman, Commonwealth Capital Ventures. “We believe in partnering with entrepreneurs at the early stages of innovation, and saw the opportunity to accelerate the series B financing as a way to accelerate the growth of the company in this highly dynamic market.”

For more news from Big Blue, visit our IBM channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.