eBay is “replatforming.”
Taking a few pages out of the hyperscale infrastructure playbooks of Google, Facebook, and Microsoft, the engineering team behind the ecommerce giant shared this week that it’s been redesigning its entire platform.
The three-year program involves changing everything about its infrastructure, including the way it decides where to build data centers, designing its own servers, and completely rebuilding the software stack that runs on top.
The team is now halfway through the process, Mazen Rawashdeh, VP of platform engineering at eBay, wrote in a blog post, and has already seen “meaningful” improvements in predictability, control, and flexibility.
Rawashdeh didn’t share much detail about eBay’s new custom servers, saying only that they cater to eBay’s individual needs. The details should become available sometime in the future, however; like Facebook and Microsoft have done, eBay plans to open-source the designs for its servers, he said.
It took nine months to create the server prototype, according to Rawashdeh.
Companies operating hyperscale cloud platforms have been designing their own servers for many years. There are several big benefits to doing that: optimizing hardware performance for your specific application needs, not paying for extra features you don’t use, and relying less on third-party vendors, whose business needs inevitably influence the design and cost of their products.
The big benefit of open-sourcing hardware designs is lowering the barrier to entry into the market of huge server deals for more suppliers. More competition usually translates to lower costs for the customer and more assurance that they can get the hardware they need when they need it.
The backs of eBay's custom servers installed in a rack
Switching to custom servers is just one part of the three-year exercise. “The replatforming of our backend infrastructure included designing our own hardware and AI engine, decentralizing our cluster of data centers, moving to an edge computing architecture, and leveraging the latest in open source technologies,” Rawashdeh wrote.
Lower in the piece, he said the company was “decentralizing the cluster of data centers we have in the United States,” shifting to a “Point of Presence (PoP) strategy.”
What this means is that eBay is deploying its servers in more data centers around the country than just the big core facilities it’s been hosting its platform in over the years. The core ones include eBay-built and operated data centers in Phoenix; Sandy and South Jordan, Utah (both just outside of Salt Lake City); as well as facilities leased from Switch in Las Vegas and outside of Reno, Nevada.
It’s possible that eBay has more data centers whose locations haven’t been disclosed publicly.
There’s recently been a push to deploy computing infrastructure in more locations, physically closer to end users, to improve application performance and reduce network costs associated with transporting data over long distances. In addition to their big core data centers, companies have been deploying servers in fewer numbers in or near big population centers, so that their application data doesn’t have to travel far, as a result reducing latency and improving user experience.
eBay’s new PoP strategy has similar goals. Per Rawashdeh:
Through this edge computing approach, we can create a faster, more consistent user experience, saving 600-800 milliseconds of load time. We are deploying online services and data closer to our users, enabling dynamic and static caching capabilities, decreasing latency and improving the experience.
eBay’s new PoPs are most likely going into colocation data centers in or near major population centers that host infrastructure for big carriers as well as local internet service providers. Those would be the players necessary to get your data from a remote core data center to the PoP and then deliver it to local users.
The software stack eBay engineers have built to run on top of this new physical infrastructure consists to a great extent of various open source technologies adopted for the company’s needs. Those include Kubernetes, Envoy Proxy, MongoDB, and Apache Kafka, Rawashdeh said.
He also told SDxCentral this week that part of the replatforming effort is doing away with OpenStack in favor of Kubernetes and Docker. That’s a big hit to OpenStack, the popular set of open source cloud infrastructure software of which eBay has for years been one of the biggest successful use cases.
As the SDxCentral article pointed out, 95 percent of eBay’s traffic ran on its OpenStack cloud just as of May 2017.
eBay also used open source software to build its new AI engine, which all its teams can now share and use to increase productivity, according to Rawashdeh. The engine has accelerated production of new features, he wrote, such as computer vision, image search, and social sharing.
