Uber’s driver and rider numbers around the world have been ballooning, and so has its revenue. The mobile ride-hailing company has also been expanding into adjacent markets, such as scooters and food delivery.
Along with all that growth comes demand for data center capacity to handle all the data flowing between driver and rider smartphones and the company’s computing network. To make sure Uber can scale its infrastructure on pace with demand growth, Dean Nelson, its executive in charge of infrastructure, has been making rounds on the data center conference circuit, explaining Uber’s needs to potential data center builders.
Head of Uber Compute did this first in June, at Datacloud Europe in Monaco. His second appearance came this Monday at DCD London, where in a keynote presentation he once again outlined the company’s infrastructure strategy and the basic parameters of a standard Uber data center, according to DCD, the event’s producer.
Like most young companies that have scaled quickly, Uber has been using a mix of cloud services and commercial data center providers to host its data.
Per DCD:
They call this ‘the tripod strategy:’ “It is not on-prem versus cloud, it’s not an or, it’s an and. When you get to scale, you have volumes where you can get to price performance that rivals cloud. But you can’t move like cloud can, we want to be able to say ‘turn it up’ and they have massive scale to do that immediately.”
Nelson hasn’t said which cloud provider or providers Uber has been using. As he explained, the top cloud providers’ global scale enables their customers to expand into new territories (Uber is now in almost 80 countries) or increase capacity in existing ones quickly.
Once a customer reaches certain scale, however, it starts to make financial sense for them to deploy their own data centers. In Uber’s case that means leasing facilities from data center providers, which the company then fills with its own computing equipment:
Said Nelson:
“The thing that has kept me up for the last two years is keeping up with this demand. Every forecast has been wrong, it’s been too small. The ability for us to leverage the best of and the speed of the cloud, but the price of on-prem is a really big strategic advantage.”
Because of that cost advantage, most Uber data center workloads are hosted in the company’s own data centers, he said. Today, those data centers are in the US, according to Data Center Frontier, but Nelson wants to expand dedicated Uber infrastructure to Latin America, Europe, the Middle East, North Africa, and Asia-Pacific.
His goal is to get to tens of regions – three data centers per region – and 100s of availability zones (single sites). Availability zones can be edge sites far away from core regions, deployed to improve application performance for local users.
Nelson outlined some very specific basic requirements for a standard non-edge Uber data center he’d like developers to build for him. Each site should provide 5MW of power across 10,000 to 15,000 square feet of space to accommodate 576 IT racks.
Per common practice, Uber has defined a standard “pod” as an increment of compute capacity it expands in (a single Uber pod is 16 racks), and a single data center fits 30 such pods. That’s 480 racks. There are also 32 networking racks and 64 to be ready for the unexpected.
Uber has separate racks for compute, database, various storage tiers, and GPUs (used for machine learning).
According to a photo of one of the slides Nelson displayed at DCD, the company uses Apache Mesos, the open source compute cluster management system, in its compute racks; the Apache Cassandra open source database; and the Apache Hadoop open source distributed storage framework for “warm storage” (it also has a cold storage tier).
