Rains Flood Seattle T-Mobile Data Center
December 4th, 2007 By: Rich Miller
Torrential rains in the Pacific Northwest have flooded a T-Mobile data center in Bothell, Wash., affecting customer activations and the performance of the Tmobile.com web site. The outage reportedly took down servers supporting T-Mobile service activation portals, company web sites and myFaves updates. UPDATE: See the comments section for reader updates with additional details and information on customer impact.
More than 4 inches of rain and gale-force winds battered the Seattle area yesterday, leading Gov. Christine Gregoire to declare a state of emergency. In Bothell, an eastern suburb of Seattle, flooding has closed major roadways and led to the evacuation of many local businesses.
One poster on a blog discussing the outage identified himself as a T-Mobile employee and provided some detail on the situation at the company’s Bothell facility:
The report is correct – The main data center is flooded right now, so we only have email and our coverage/outage map. No accounts can be accessed due to the systems being out, so this can and will affect short codes such as 646, 225, the website and just about every system. It started around 7:30pm EST and was still happening when I left around 2:00 am EST. They were hoping to at least get read only systems online in the next few hours, so that we can at least do a little for the customers. Hopefully it will be back soon – Any techies have a clue how long it takes to dry out servers and get them back online – That should give a better idea on timelines.
For additional coverage of the storm’s impact, see coverage in the Seattle Times. The area is home to many major data centers, but at present this is the only report we’ve seen of storm-related downtime.
“The area is home to many major data centers, but at present this is the only report we’ve seen of storm-related downtime.”
The vast majority of Seattle area datacenters are either Downtown (Westin, Fisher, 1000 Denny, etc) or in Tukwila (on the hill at the south end of the industrial area just south of Boeing Field). Both of those locations are in hilly areas with near zero risk of flooding. Bothell has a small concentration of datacenters and callcenters: T-Mobile, AT&T Wireless, OnFiber, Time-Warner Telecom, Worldlink, to name a few. North Creek runs right through Bothell and is prone to minor flooding every decade or so, and a major flood risk is always looming.
We moved our datacenter to Tukwila 3 years ago, ironically from the North Creek area in Bothell that was affected by this storm. We were at North Creek for 7 years and one of our perennial worries was an autumn storm causing North Creek to overflow its levees. Our datacenter was located on the second floor of a building there, so the facility itself was somewhat safe, but getting people in and out, phones, power, etc. added up to too much risk.
North Creek is an idyllic locations with a park-like setting. I miss the path along the creek with the salmon returning every October. But honestly I’m happy to have traded the wooded park for the industrial environs of Tukwila when the storms arrive to bring in the winter.
MatthewPosted December 4th, 2007
Yes this is aggrivting as a T-Mobile employee but who can stop acts of God. I guess that won’t be covered in insurance lol. I wish these guys the best luck because I need to actvate some phones if they expect me to hit my quota for the month and be the T as they say here at T-Mo.
T-Mobile is the only one down. The data center in Bothel is in a flood path. To have all of your customers down and all because you had your servers in a flood path is proof t-mobile is not worried about us customers. What about redundant servers in multiple locations? Nice to see they can afford ad banners everywhere but won’t multi-home systems or geographically distribute them. No ETA when they will return. Servers are flooded, they can not even look up ballances or credit accounts for us UMA users that are getting raked over the UMA being down and having to use the buggy T-Mobile GSM towers (formerly cingular’s old towers that were given up by att).
This post is directed mostly at Arthur, but may be used by all as a resource for information current to the T-Mobile server downtime. I felt compelled to write this as once again, the misinformed rise to the top and feel obligated to speak negatively about that which they do not understand.
My credentials: wireless guru who works directly with AT&T, T-Mobile, and Sprint / Nextel. I was on the morning conference calls hosted by T-Mobile’s president and staff for the American side discussing the issues directly and the solutions to be provided.
What happened: Floods affected the main servers which supported all activation portals, company websites (public and internal), as well as star services (#646#, #225#, etc.) and myFaves updates. What Arthur doesn’t realize is an attempt was made to migrate the data over to the back-up servers, but that was unable to be completed due to environmental issues.
What T-Mobile is doing for our new customers: Customers who wish to activate through T-Mobile are being given waived activations (may take up to 45 days to apply) in addition to the $50 bill credit for myFaves rate plan sign-ups that some retailers are able to offer. FlexPay new customers do not pay activation, so this is a mute point, however it is expected that approximately 100 bonus minutes will be added to their account considering some of the provisions the retailers must take to activate them at this time (see an an authorized T-Mobile representative for details)
Existing customers can continue to use their phones as normal, however things such as updating the myFaves settings or any other billing related updates can not be made at this time. Payment can NOT be accepted at T-Mobile corporate stores, however independent vendors (PreCash, etc.) are unaffected by this flood and money will credit to account as normal. FlexPay customers who’s bill would be due WHILE the flood situation is being resolved will NOT be disconnected for failure to pay, however the bill will be due immediately after. FlexPay customers who’s bill was due BEFORE this incident may be disconnected for failure to pay and must week an alternate independent vendor to pay the bill.
In my experience with these companies, there have been time when whole systems go down. T-Mobile has set up several internal conference numbers that we (employees, dealers, independent retailers, etc) may call in directly to have our questions answered. I have not seen this from any other company (so yes, I got annoyed at Arthur for thinking T-Mobile is not worried about their customers). For both our needs and those of the customers, additional operators have been tasked to all working call centers to answer questions and concerns you may have.
The progress: as of 6:00 am this morning, power was restored and the long process of rebooting the systems has began again. Proof of the progress can be seen in the main corporate website, t-mobile.com, being live once more. As more applications come online, retailers can expect to see their Samson, Watson, and iCam systems functioning normally. The earliest estimates put these all being corrected around 15:00 PST, however it may be up to 48 hours. Thankfully, though some IT staff did have to relocate during the flood, no one was injured from within the company, and additional staff have already been sent in to facilitate with bringing it all back online.
Please note that what I have mentioned above is NOT to be considered official until T-Mobile directly acknowledges this through a press-release of some sorts.
mikePosted December 4th, 2007
Understandably an attempt to get backup servers online can take time…but migration of data? What kind of backup servers need data migrated to them? That defeat the purpose of a backup if you ask me….
UPDATE: For those of you who may or may not know, all systems are live and active. This occurred at approximately 18:00 EST today.
Despite the fact that there were a day’s worth of activations to be done in addition to those for the evening, T-Mobile did an amazing job staffing their activations call center so that after the voice prompt asked for English or Spanish, a representative picked up the line immediately (I tested this personally, at three different times).
In addition, customer care did a great job staying on the line doing something for me that I too could have done, so from my standpoint, I just want to congratulate the team on continuing to do an amazing job and to let the readers of this blog know the status.
I was hoping to receive an official confirmation about the promotions and incentives to thank customers for their patience in this process, however I still recommend all incentives be considered official ONLY when T-Mobile directly acknowledges them.
Internal Notice: T-Mobile employees (both direct and indirect sales) should check back with their NARs, ADRs, and DMs for updates. The war room number may or may not be closed as of now (and obviously number and credentials will not be posted here), however Care is able to fully resolve all issues and your other questions should be directed there.
It has been a pleasure, DCK. Hope this post helps.
shakPosted December 13th, 2007
but, H@H (Hotspot @ Home) SErvice is STILL down to this day. I have been unable to use my H@H service with Tmobile for the past week.
wow, i’m so glad i switched a few months ago….
I work in IT, and tmobile dropped the ball big-time with this.
They really need to go over their disaster recovery plans, and business plans to keep things operating when disasters strike.
aka…dont’ put everything in one location.