AOL Launches New ‘Lights Out’ Data Center
October 11th, 2011 By: Rich Miller
AOL has begun operations at a new data center that will be completely unmanned, with all monitoring and management being handled remotely. The new “lights out” facility is part of a broader updating of AOL’s infrastructure that leverages virtualization and modular design to rapidly deploy and manage server capacity.
The new approach to infrastructure is driven by AOL’s focus on content, according to Mike Manos, AOL Senior Vice President of Technology Operations. As the company builds its portfolio of blogs and web sites – which includes The Huffington Post as well as blogs like Engadget – it is adopting a flexible infrastructure that can deploy server capacity quickly in locations with strong user populations.
The new data center, known as ATC, is a big step in this direction. Manos describes the facility in a blog post, but does not offer details on ATC’s size or location.
AOL is using building its data center using standard blocks of infrastructure that can be deployed in increments of 500 kilowatts to 1 megawatt of power. AOL is not currently using containers, as Microsoft did while Manos was heading the company’s data center effort, but could do so in the future.
Moving Content Closer to Users
“The beauty of the approach is we can get capacity up any where quickly,” said Manos. “Keep in mind we are moving the company to be a content player – you don’t need big facilities, but you do need coverage where the readers are.”
Deployment requirements shift when you don’t have to factor staffing into the equation, which is the approach AOL has taken. “ATC is a 100 percent lights out facility,” Manos writes. “There are absolutely no employees stationed at the facility full time, contract, or otherwise.” AOL isn’t the first data center operator to deploy unmanned facilities, but is one of the largest providers to do so.
“Failures and issues on-site go into a planned maintenance mode,” said Manos. “We simply move the instances (or create new ones) to other data center facilities and the failed equipment is addressed in a scheduled way using outsourced or vendor partners. This is hugely cost beneficial for us as we can now build SLAs around the service without having to worry about the cost of staffing for ‘something to go wrong’.”
Management Software A Key Enabler
Manos said AOL has developed configuration management software that has accelerated the process for deploying new server capacity. “We went from provisioning servers in days, to getting base virtual machines up and running in under 8 seconds,” he writes. “We used Open Source products and added our own development glue into our own systems to make all this happen.
“We have been running this technology and approach in our existing larger facilities for a few months to ensure the model works under failure scenarios. We did trial the approach in staffed facilities just in case. After a quarter of proof we took the leap.”
Manos also said that AOL has been underappreciated as a leader in Internet infrastructure.
“While it may be fun and ‘new’ to look at the tens of thousands of machines at Facebook, Google, or Microsoft – it is often overlooked that AOL had tens of thousands of machines (and still does!) and solved many of the same problems years ago,” Manos writes. “To be honest it was a personal revelation for me when I joined. There are few companies who have had to grow and operate at this kind of scale and every approach is a bit unique and different. It was an interesting lesson, even for one who had a ton of experience doing something similar in ‘Internet Darling’ infrastructures.”
[...] We’ll let you decide: AOL Launches New ‘Lights Out’ Data Center [...]
As IT moves to the cloud and becomes commoditized, every part of the data center is going to need to become extremely dynamic. The facility, the servers (physical or virtual), power, and cooling are going to have to respond to drastic changes as IT demands ebbs and flows. AOL seems to get this and is structuring their IT around this need.
[...] We seem to be stuck in a national productivity delusion, finding ways to do things, increasing our working hours in the name of productivity. Ensuring things fall apart in the name of productivity. China makes everything, so what do we do and what do we have to export? Innovation, the government cries, however innovation brings efficiency. This has clearly been seen by the US State of Georgia, who spent millions tempting high-tech companies to relocate their data-centres to Georgia in the name of job-creation, only to discover huge town-size data centres require only two people to run it, someone to turn it on, the other to ensure the colleague does not fall asleep. This is innovation, efficiency and progress. In fact, now no-one is required… [...]
Why is a sizeable part of my web traffic coming from Manakin Sabot in Virgina — near Richmond? Is that where AOL keeps one of its main servers? Is AOL searching my site from there? Is this the most basic question people ask, and am I just a newbie?
[...] Large organizations that invest heavily in their data center infrastructure are actively exploring robotics solutions to help them better control their data centers. IT shops such as Amazon and Google are looking at ways to create a fully automated, lights out data center. AOL has taken a first in that direction with an unmanned data center facility. [...]