Google Turns to Tape to Rescue Lost Gmail
March 1st, 2011 By: Rich Miller
Last month we had an article on whether tape backup was obsolete, which discussed some of the reasons why many organizations continue to use tapes. One of the outfits is Google, and this week those tape backups proved invaluable to a group of Gmail users whose email boxes disappeared during a software update. When a bug wiped out Gmail boxes, Google turned to the tape to restore the data.
“I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers?” wrote Ben Treynor, VP Engineering and Site Reliability Czar. “Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue.
“To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they’re protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it’s taken us hours to get the email back instead of milliseconds.”
Google says it will release a full incident report once its analysis is completed. The company said the bug affected 0.02 percent of Gmail users.
It should be noted that Google offers these services at little to no charge. Clients who demand their mission critical data to be redundantly stored have the peace of mind to know that reputable data centers are in lock step with their needs. Sure there is a cost to a higher level of service, but a data loss can result in a company going out of business so it is referred to as a value. How palatable is a free/low cost service if one event can be the end of your business?
[...] #2: The world hasn’t “moved on.” Tape remains a critical part of a well-planned storage solution for many organizations. In fact, when 40,000 Gmail users logged in recently to find their e-mail accounts empty thanks to a software bug that affected disk-stored data in multiple data centers, it was tape that saved the day at Google. [...]