Database Sharding Helps High-Traffic Sites

Several presentations this week have focused attenton on database sharding – breaking a large database into smaller pieces to provide faster access to the data. ZDNet reports how Google used database partitioning to manage a 12 gigabyte database ^[1] for its financial planning unit. Google’s Chris Schulze discussed the technology during a presentation at Hyperion’s Solutions 2007 conference in Orlando.

In addition to using sharding to streamline its huge in-house databases, Google last month contributed its data partitioning solution ^[2] to the open source Hibernate ^[3] project. A description:

Hibernate Shards offers critical data clustering and support for horizontal partitioning (also called sharding) to Hibernate. Now, customers can keep their data in more than one relational database for whatever reason-too much data or to isolate certain datasets, for instance-without added complexity when building and managing applications. Hibernate Shards is designed to encapsulate and reduce the complexity of building applications that work with sharded datasets.

Database sharding is also used by Digg, which outlined the details of its database management techniques in a presentation at the MySQL conference ^[4] in Santa Clara this week.

Digg engineer Tim Ellis said sharding has helped Digg.com achieve faster web site performance, but breaking a database into several smaller ones increases complexity. Ellis explains that a database can be sharded by table, date or range.

Sharding is similar to partitioning, says Ellis, but with several key differences. Sharding usually involves divvying up data onto different physical machines. Partitioning, in contrast, typically occurs on the same piece of hardware. And while MySQL does not natively allow sharding, it does support partitioned tables, federated tables and clusters.

Sharding is a familiar term to those familiar with the infrastructure of online games ^[5], where popular MMORPGs like World of Warcraft segment their user base across multiple physical machines running instances of the game.

---

Article printed from Data Center Knowledge: http://www.datacenterknowledge.com

URL to article: http://www.datacenterknowledge.com/archives/2007/04/27/database-sharding-helps-high-traffic-sites/

URLs in this post:

[1] manage a 12 gigabyte database: http://www.zdnet.com.au/insight/software/soa/How-Google-fixed-up-its-financial-management/0,139023769,339275077,00.htm

[2] data partitioning solution: http://www.infoq.com/news/2007/03/hibernate-shards

[3] Hibernate: http://www.hibernate.org/

[4] MySQL conference: http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9017778&source=rss_news50

[5] infrastructure of online games: http://www.datacenterknowledge.com/archives/2006/Sep/13/eve_online_30000_users_on_one_server_shard.html

[6] Rich Miller: http://www.datacenterknowledge.com/archives/author/richm/

Click here to print.