Amazon Web Services (AWS) says the price spikes are caused by periods of “constrained capacity” in particular availability zones and customer bidding strategies that employ above-market bids to prevent interruptions of their computing jobs. Amazon says these strategies result from confusion about how spot pricing works, and has posted a video outlining effective approaches to spot bidding.

“These price spikes are new, and they call into question assumptions that many users have made about how the auctioning of computing resources works,” wrote Jonathan Boutelle, co-founder and chief technology officer of Slideshare, in a guest post Tuesday at GigaOm, which shared strategies for EC2 users to manage the impact of these price spikes on cost and availability. “Almost every class of servers has hit spikes of more than 10 times their retail price in last few months.”

The spike to $999.99 per hour was noted back in September by SEO Moz and a discussion on the AWS Developer Forum.

“We are seeing a number of customers using bidding as their sole source of managing interruption, rather than building automation into their applications,” wrote Amazon’s Dave Ward in the forum thread. “We advise against this behavior … it is important to be able to handle interruption in your application, as prices will change and could potentially be higher than your maximum bid price.”

That’s what happened in the case of the extreme price spike, which was related to a sudden increase in demand for a particular type of instance which depleted the unused capacity. Since there was a customer with a number of bids at the price of $999.99 to prevent interruption, the price spiked up to that level.

A key task for users is structuring their use of EC2 between spot instances and reserved instances, so that a sudden surge in both demand and price won’t interrupt an important service or result in excess fees, according to Boutelle.

Meanwhile, Amazon is adding additional availability zones as it deploys new data center capacity, and says it hopes to include more zones in its spot market in the future.

The new data center capacity allows Amazon users to house and serve data from Sao Paulo. The new region will offer the full range AWS services, including its Elastic Compute Cloud (EC2) and Simple Storage Service (S3), as well as Amazon’s Route 53 DNS service and CloudFront content delivery service..

“With the opening of this new Region, AWS customers in South and Central America can now enjoy fast, low-latency access to the suite of AWS infrastructure services,” wrote Amazon Web Services’ Jeff Barr in announcing the new region. As part of the expansion, portions of the AWS web site will now be available in Portuguese.

Brazil is a hot emerging market for IT infrastructure, which will be critical to supporting communications services for two global events being hosted by Brazil: the 2014 World Cup and 2016 Summer Olympics in Rio de Janeiro. Equinix and Terremark have expanded their infrastructure in Sao Paulo, while private equity firm Silver Lake Partners has acquired a stake in Locaweb, Brazil’s largest provider of hosting and cloud services.

A look at the Amazon Perdix container, which is reportedly being used to deploy new data center capacity in Oregon.

Amazon Web Services announced today that it is now serving its data from servers in Oregon, where the company has been building several data centers. The new data center capacity has allowed Amazon to create a second availability zone in the US-West region, offering additional options for users to house and serve data across several geographic areas to improve redundancy and reliability. The new region will offer the full range AWS services, including its Elastic Compute Cloud (EC2) and Simple Storage Service (S3).

Amazon has deployed servers in remote areas of Oregon using its Perdix modular data center. Local media say AWS has deployed modules in Umatilla, Oregon. The company has  submitted plans to build a similar facility with six modules in the Port of Morrow, and also has also resumed work at a site in Boardman, Oregon where the company bought land in 2008.

Amazon operates data centers in multiple regions, allowing users to add redundancy to their applications by hosting them in several regions. In a multi-region setup, when one region experiences performance problems, customers can shift workloads to an unaffected region. This allows customers to avoid having its cloud assets constitute a “single point of failure.”

Industry analysts say recent outages at Amazon have prompted users to adjust how they approach cloud architectures. “End users now want to mandate that they have multi-cloud strategies,” said  William Fellows, co-founder and Principal Analyst at The 451 Group.

For more on Amazon, see our Amazon Channel. For additional video, check out our DCK video archive and the Data Center Videos channel on YouTube.

Amazon outlined steps it would take to address the issues, and announced a 10-day service credit for all customers that were affected.Here are some of the notable issues addressed in the incident report.

• Why The Generators Didn’t Start: “Normally, when utility power fails, electrical load is seamlessly picked up by backup generators. Programmable Logic Controllers (PLCs) assure that the electrical phase is synchronized between generators before their power is brought online. In this case, one of the PLCs did not complete the connection of a portion of the generators to bring them online. We currently believe (supported by all observations of the state and behavior of this PLC) that a large ground fault detected by the PLC caused it to fail to complete its task. We are working with our supplier and performing further analysis of the device involved to confirm.” Amazon said it will add more redundancy and isolation for its PLCs, and is working with vendors to add a backup PLC.
• Problems With Management Software: In several cases, software programs that manage tasks complicated the recovery process. The first case came shortly after the outage. “The management servers which receive requests continued to route requests to management servers in the affected Availability Zone. Because the management servers in the affected Availability Zone were inaccessible, requests routed to those servers failed. Second, the EC2 management servers receiving requests were continuing to accept RunInstances requests targeted at the impacted Availability Zone. Rather than failing these requests immediately, they were queued and our management servers attempted to process them. Fairly quickly, a large number of these requests began to queue up and we overloaded the management servers receiving requests, which were waiting for these queued requests to complete. The combination of these two factors caused long delays in launching instances and higher error rates for the EU West EC2 APIs. “
• Problems With EBS Software: The most serious downtime affected customers using Amazon’s Elastic Block Storage (EBS). A software bug detected prior to the power outage created complications during the recovery process. There is a lengthy description of the EBS issues and how they were addressed.

The recap closed with an apology. “We will do everything we can to learn from this event and use it to drive improvement across our services. As with any significant operational issue, we will spend many hours over the coming days and weeks improving our understanding of the details of the various parts of this event and determining how to make changes to improve our services and processes,” Amazon wrote.

AWS Direct Connect is currently available in Equinix IBX data centers in the company’s Ashburn, Va campus, and will be available in additional facilities later this year. The service will make it easier for hundreds of enterprise customers to access Amazon’s cloud services.

“We’re very pleased to work with Amazon Web Services to provide customers in our facilities with an easy onramp to the cloud through AWS Direct Connect,” said Steve Smith, CEO of Equinix. “AWS Direct Connect opens up a wide range of exciting new possibilities in hybrid cloud computing and will allow customers in Equinix facilities to leverage Amazon Web Services as if it was part of their own infrastructure.”

For Equinix customers, connectivity via AWS Direct Connect provides a wide range of new deployment scenarios, including:

• Integration with core-internal applications: High bandwidth, low latency connectivity allows customers to treat AWS instances as part of the data center LAN and integrate cloud strategy into more applications.
• High volume data processing: High bandwidth connectivity will allow Equinix customers to efficiently move much larger data volumes into and out of AWS, allowing for import/export of large-scale data sets for high peformance computing (HPC) applications.
• Direct storage connectivity: Private, high throughput access to AWS allows for more regular migration/replication of data from AWS into customer-managed storage solutions, allowing for more homogenous business continuity/disaster recovery policies, and retention strategies.
• Custom hardware integration: High throughput access to AWS allows for integration of custom hardware solutions into the AWS workflow, with real-time streaming of data to and from AWS.

Equinix has a page on its web site offering more information

Equinix (EQIX) provides colocation and interconnection services to more than 4,000 enterprises, cloud, digital content and financial companies and 650 network service providers. Equinix now operates 96 data centers in 38 strategic markets across the Americas, EMEA and Asia-Pacific.

• All Things Distributed – From Amazon.com CTO Werner Vogels: “Today marks another important milestone in our continuous cost reduction strategy; we’ve lowered prices over a dozen times in the past four years, and today we’re lowering them again. Bandwidth pricing will see a reduction that for many customers can easily lead to a 40% or 50% savings in bandwidth costs. Incoming bandwidth cost will drop to $0.00 in every region. And, in the US and Europe every outgoing tier will see price reductions. We are also adding new tiers to pass saving onto our very large bandwidth consumers.”
• GigaOm (Derrick Harris) – There are two likely – and non-monopolistic – explanations for why a company with such a dominant market position would continually drop prices like this, and they’re interrelated. One is that the economies of scale AWS achieves via its massive operation allow it to offer lower prices on everything from computing to bandwidth.The other reason is that cloud computing democratizes access to resources. AWS knows it can’t pull the Oracle strategy of locking customers in and then bending them over a barrel, so to speak. AWS has a seemingly insurmountable innovation lead among cloud providers, but it doesn’t always have the lowest prices.
• IT News – It is not the first time Amazon has felt compelled to offer free inbound data transfers. The company killed its free inbound data offer last October and replaced it a with a free micro Linux instance that would only attract charges if the application became popular.
• ZDNet (Mary Jo Foley) - Amazon wasn’t the first cloud provider to eliminate incoming bandwidth fees, as noted by Mary Jo Foley. “Microsoft plans to tweak its Windows Azure pricing, come July 1, to make the platform more attractive to users who want to migrate lots of data to the cloud. The change, announced on June 22, also will result in all inbound data transfers for both peak and off-peak times being free, company officials said, via a post to the Windows Azure blog.”

A look at the Amazon Perdix container, included in a presentation at Amazon Technology Day.

On Saturday The Hermiston Herald reported that Amazon subsidiary Vadata has deployed six modules in Umatilla, Oregon. The paper describes the facility as “innocuous, bland structures — a row of six shipping containers gussied up with a few doors and vents. ” That description matches images of a modular design called Perdix that Amazon’s James Hamilton discussed earlier this month at a technology open house.

Amazon.com joins major cloud builders Google, Microsoft and Yahoo in embracing factory-built components as a strategy to reduce the cost and deployment time for data center capacity. The Oregon construction is part of a larger effort by Amazon to prepare for a significant expansion of its data center capacity to accommodate the growth of its cloud computing business.

Amazon.com has submitted plans to build a similar facility with six modules in the Port of Morrow, and also has also resumed work at a site in Boardman, Oregon where the company bought land in 2008.

A look at the Amazon Perdix container, included in a presentation at Amazon Technology Day.

Amazon Web Services doesn’t say much about the data centers powering its cloud computing platform. But this week the company held a technology open house in Seattle, where AWS Distinguished Engineer James Hamilton discussed the company’s infrastructure. The presentation (PDF) included an image of a modular data center design used by Amazon, which is the first official acknowledgement that the company uses modular infrastructure.

Hamilton also shared a factoid that provides a sense of the rapid growth of Amazon’s cloud platform. “Every day Amazon Web Services adds enough new capacity to support all of Amazon.com’s global infrastructure through the company’s first 5 years, when it was a $2.76 billion annual revenue enterprise,” Hamilton states in one of the slides.

Growth Driving Data Center Expansion Plans

Even without the exact numbers, that’s an indicator of how rapidly Amazon’s infrastructure is growing, and why the company has recently began acquiring additional sites in Ireland, northern Virginia and Oregon for data center expansion.

In his research at Microsoft and now at Amazon Web Services, Hamilton has focused on cost models for operating hyper-scale data centers. His presentation at the Amazon open house reviewed cost assumptions for an 8 megawatt data center, which could include 46,000 servers.

Hamilton estimated the cost at $88 million (about $11 million per megawatt), and presented a pie chart outlining monthly operating costs for a facility, which is dominated by the cost for servers (57 percent), followed by power and cooling (18 percent) and electric power (13 percent).

These percentages are consistent with Hamilton’s earlier published research on data center costs. His example assumes power costs of roughly 7 cents per kilowatt hour and a Power Usage Effectiveness (PUE), which both suggest that the example data center is a composite  of Amazon’s global footprint rather than its best-performing data center.

Amazon and Modular Design

Hamilton was an early advocate of using shipping containers to deploy large volumes of servers in a tightly-controlled environment, first discussing this approach in a series of 2007 presentations that preceded Microsoft’s decision to use modular units to deploy its cloud computing infrastructure. When Hamilton moved to AWS, it prompted speculation that Amazon might also be using containers.

At Tuesday’s open house Hamilton displayed a slide of modular data centers, including the Amazon Perdix. The unit appears to be a custom-built unit that is wider and taller than standard ISO containers. While it’s hard to glean much from the exterior, vents at the side and top suggest cooling is managed in the upper section of the unit, which is air-cooled. Why Perdix? It’s the name of a character in Greek mythology known for inventing useful tools.

Is this Amazon’s current technology? Perhaps not. An Amazon affiliate that builds the company’s data centers has submitted building plans for a new facility in Umatilla, Oregon featuring six modules, according to local media. Plans show the structures will be about 20 feet wide and 108 feet long, situated side by side on the property, according to the East Oregonian.

Amazon continues to deploy infrastructure on raised floor, as seen in this photo shared by Hamilton.

A slide of a data center from a presentation at the Amazon Technology Open House.

While Zynga’s GagaVille persevered Monday it seems that a surprise one-day sale of the MP3 version of the album for 99 cents stalled Amazon servers by early afternoon. The surprise sale was an attempt to promote Amazon’s Cloud Drive where users get 20GB of storage space by purchasing the album. Amazon issued the statement,  ”Amazon is experiencing high volume and downloads are delayed. If customers order today, they will get the full Lady Gaga, Born This Way album for $0.99. Thanks for your patience.”

Amazon is trying to build as much momentum for their cloud service before Apple introduces their own this summer. Apple is reported to have signed the last major record label for its cloud-based music service, which has been the suggested use of their Maiden North Carolina data center.

Last month Amazon Web Services suffered big performance problems.  Embrace the Chaos Monkey.