UPDATE: As of 6:20 p.m. Eastern time, prominent sites hosted on Wordpress.com were beginning to come back online after about two hours of downtime.”We are back running at full capacity now,” Mullenweg reported at 6:30 pm. “Closely monitoring services for any aftershocks.

UPDATE 2: Matt has provided more info on the Wordpress.com blog. “We are still gathering details, but it appears an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven’t experienced before, and broke the site,” he reports. “It also broke all the mechanisms for failover between our locations in San Antonio and Chicago. All of your data was safe and secure, we just couldn’t serve it. We need to dig deeper and find out exactly what happened, why, and how to recover more gracefully next time and isolate problems like this so they don’t affect our other locations.”

The Wordpress.com service is hosted by Layered Technologies, with much of  its infrastructure housed in the provider’s Chicago data center (see video). Wordpress.com hosts about 10.2 million blogs.

The damage to the relay fuses left the ATS unable to start the facility’s diesel backup generators, as they normally would in a utility outage. With the generators offline, the data center switched over to battery power from the uninterruptible power supply (UPS).

“During this time, the UPS batteries were drained, and once the batteries were drained, power was lost to the data center floor at approximately 4:56 am PST,” NaviSite reported. “Power was restored to the data center by manually starting the generators and transferring load to the generators. Power was restored to the data center at 5:35 am PST.”

In the wake of the incident, NaviSite is taking steps to add resiliency to its power infrastructure. “We have an electrician coming in to re-architect our surge suppression environment,” said Allen Allison, Vice President of Managed Services for NaviSite. “We are looking at the surge suppression and switch infrastructure for all of our facilities.

Allison said the key components of the power infrastructure at the San Jose facility are tested regularly. “We perform complete load testing on a monthly basis,” he said. “We also perform a test for the ability of the transfer switch to throw on the generators.” The power surge, however, is hard to simulate. “There’s no way to inject such a heavy surge (into a test),” Allison said.

During the outage, NaviSite kept customers informed via e-mail, a company blog and Twitter. Chief Marketing officer Claudine Bianchi said this type of multi-facted communications effort is critical during outages.

“We found that being transparent was very beneficial,” said Bianchi. “Usually in circumstances like this, lack of information is the biggest issue for customers.”

While Twitter is an increasingly important venue for customer updates during hosting outages, Bianchi said e-mail remained the primary communications tool for NaviSite’s many enterprise customers, many of whom receive notifications via their Blackberry devices.

The University Data Center experienced “an excessive heat condition” on Tuesday afternoon, according to the Daily Pennsylvanian. The incident occurred when one of the two glycol pumps supporting the Data Center was accidentally switched from automatic to manual during an equipment replacement, resulted in overheating.

“We are experiencing an outage due to an extremely high number of whales,” reports the Twitter status page. “A sudden failure coupled with problems in switching to a backup system produced a high number of errors for around 90 minutes. This made the site largely inaccessible. No data was lost or compromised during this outage.”

The “whales” comment refers to the “Fail Whale” – the downtime mascot that appears whenever Twitter is unavailable. The appearance of the Fail Whale indicates a server error known as a 503, which then triggers a “Whale Watcher” script that prompts a review of the last 100,000 lines of server logs to sort out what has happened.

When at all possible, Twitter tries to adapt by slowing the site performance as an alternative to a 503. In some cases, this means disabling features like custom searches. In recent weeks Twitter.com users have periodically encountered messages that the service was over capacity, but the condition was usually temporary. At times of heavy load for more on how Twitter manages its capacity challenges, see Using Metrics to Vanquish the Fail Whale.

Twitter’s last major downtime event was a 3 hour, 40 minute outage on August 6, when Twitter was among the social networking sites targeted by an electronic attack, which prompted the service to beef up its network defenses.

While some Twitter-watchers continue to debate whether its growth is continuing, co-founder Ev Williams posted Jan. 12 that “across all metrics that matter, yesterday was Twitter’s highest-usage day ever. (And today will be bigger.)”

“Generator power has been restored to the data center in San Jose, but the site was without power for approximately 45 to 60 minutes,” NaviSite reported on the company blog. “The data center has been and continues to run on generator power.  We are still waiting for street power to become available, but will not switch back over until we have an understanding of what caused the original issue.”

NaviSite (NAVI) did not indicate the precise cause of the outage, but one of its customers supplied more information. “Our backup power systems initially functioned correctly shifting to battery power as a bridge to generators which then failed to turn on,” ProStores said on its Twitter feed.

ProStores provides turnkey web stores for retailers who sell their goods on eBay. AuctionByteshas more coverage of the ProStores outage.

The power interruption started at 9:19 a.m. local time when a module failed on an uninterruptible power supply (UPS) and the unit failed to transfer the load properly, Rackspace said in its status update. Power was restored for most customers by 11:30 a.m., but a subset of servers failed to restart properly.

“Although the majority of devices were down for a minimal amount of time we had particular issues with a group of about 220 Whiteboxes which have taken several hours to bring up,” Rackspace reported. “DCOps have had to manually intervene to bring the servers back online. In numerous instances they had to replace power supplies in servers, replace firewalls, reconfigure switches and to log on to servers to get them to boot properly. We can only apologise for this incident. We take this type of incident extremely seriously and will work to fix the root cause.”

Rackspace has experienced a number of outages this year, but all had involved servers housed at its Dallas/Fort Worth data center.

“Starting yesterday we began experiencing very high loads on our storage devices for cluster WC1 in DFW,” Rackspace said on its status page. ”In order to reduce load we have shut down processes like CRON to ensure core site content continue to load.  While load spikes are common in our cloud infrastructure, we have not been able to fully identify the root cause of these unusual issues.

“We are working with engineers from inside and outside the company with the best expertise on these issues to resolve them and develop a plan of action to ensure we do not repeat this state. We have a series of changes that are being implemented in real time.  We are being careful to minimize issues as we proceed.”

UPDATE: At about 6 pm Central time Rackspace provided an update: “We have been seeing improved performance on our Cloud Sites WC1 storage cluster for the last few hours. Assuming stability continues we will resume CRON operations this evening. At this time, we cannot declare victory on this issue, but we have many plans in place to continue to increase headroom and ensure stability under all conditions.”

“The Salesforce.com Technology Team has resolved the service disruption issues on all instances from 12:10PM PST to 1:25PM PST,” the company reported on its status dashboard. “All services are restored at this time. We are performing a review of the incident and will take any corrective action needed. We apologize for any inconvenience this may have caused you and appreciate your patience.”

UPDATE: Users on Twitter report continuing problems trying to log onto their apps. Saleforce.com reported a second, briefer outage affecting its NA2 instance. “The problem began at 1:49PM PST and was resolved as of 1:56 PM PST,” the company said.

The service technician from Eaton Corp. was injured during scheduled maintenance on the uninterruptible power supply (UPS) system in Peer 1’s fourth-floor data center at 151 Front, the largest carrier hotel and data center hub in the Toronto market. The technician was replacing a failed fan in a UPS unit.

“During the maintenance at approximately 8:46PM EST there was a visible arc flash (no confirmed cause yet) from the UPS causing 2nd and 3rd degree burns to the Eaton service technician and witnessed by our data center manager who also went to the hospital to have his eyes checked for retinal burns,” said Ryan Murphey, Vice President of Facilities & Data Center Operations for Peer 1, in an e-mail update. “The Eaton service tech was transferred to another hospital with a burn unit and was in serious/critical condition and our data center manager was treated and released.”

All staff were cleared from the fourth floor suite while the fire and police investigated, but were cleared to return and power was restored about three hours after the incident. Peer 1 said it had additional staff on hand to provide support for customer site restoration.

“Wrap around bypass was inspected for damage by our electrical service company and engaged after given approval by the building, fire department, Toronto Hydro and the MOL (Ministry of Labor) restoring power to the data center suite at approximately 11:31PM EST,” Murphey wrote.

Peer 1 provided customer updates on the incident on its support forum.