Recent AWS outages didn't just keep folks from streaming Netflix movies or sharing pictures -- they may have also cost some people their happily ever after.
I am definitely not going to be using EC2 again. I will still be depending on the cloud, but I've learned my lesson.
During the two-hour AWS outage on June 14, staff at the Las Vegas-based bid-for-dates community fielded nearly 1,000 complaints from users desperately trying to make that special connection.
The outage occurred near the end of the day on a Thursday, a time when many users were trying to log onto the site to finalize plans for dates later that night.
"We had a couple complaints that users would have to stand up their dates because they couldn't get contact information," said Brandon Wade, CEO of WhatsYourPrice.com. "It seems to be pretty mission-critical at that point."
The company apologized to its forlorn customers, and doesn't want to have to do it a third time.
"Having to apologize once is probably enough," he said. "But having to … a second time for the exact same issue is probably too much."
The June 2012 Amazon EC2 outages did more than just tarnish the reputation of WhatsYourPrice.com and leave its customers without dates -- it cost the site thousands.
"In terms of lost revenue, a two-hour outage on those days roughly accounts for anywhere between $2,000 to $4,000," Wade noted. "I would quantify it as a maximum of $8,000."
Now, the company is on the market for a new cloud service provider.
"I am definitely not going to be using EC2 again," Wade said. "I will still be depending on the cloud, but I've learned my lesson and I'm not going to solely depend on just the cloud."
Wade isn't completely jilted by the idea of cloud, though. WhatsYourPrice.com is out there -- playing the field.
"We've been approached by a number of cloud providers, including Rackspace," Wade noted. "They have a solution in which they can provide an almost, if not 100%, availability SLA [service-level agreement]."
The newly released Google Compute Engine, an IaaS platform, also looks good on paper.
Cloud customers forgive -- but don't forget -- Amazon's transgressions
It's easy for me to get mad at [Amazon], but the truth is, our apps should have been built better.
CIO of HubSpot
As an Amazon cloud customer since 2007, Cambridge, Mass.-based HubSpot has also weathered its share of outages.
"People called it the 'Amazon Apocalypse of 2010' … we've been through a couple of other major outages with them," said Jim O'Neill, CIO of the hosted marketing software provider.
HubSpot hasn't put a definitive price on the two recent Amazon outages, but O'Neill speculates it cost them upwards of 10 peoples' time for two to three days. A smaller portion of the IT staff was involved for a longer timeframe after that.
"For the June 29 power outage, it was maybe 20 to 45 minutes worth of physical unavailability of given instances, but it took us 24 to 48 hours to recover all the data," O'Neill said. "Amazon did help -- they did a lot of things behind the scenes to help us out, but the scale was so large they didn't even have the resources to do it in a shorter time window."
Cloud consumers to Amazon: It's not you, it's me
HubSpot's O'Neill feels partly to blame for the outage's aftermath. Moving to cloud requires some re-engineering and apps need to be built for failure.
"I think we found areas of the app that we thought we had engineered better for failure, that we hadn't," he said. "It's easy for me to get mad at [Amazon], but the truth is, our apps should have been built better," O'Neill added.
Meanwhile, WhatsYourPrice.com was hosted in multiple zones to avoid downtime during outages, but that didn't help.
"Apparently there were some glitches, so we were unable to launch availability and new instances in other zones that were not impacted," Wade said.
After the first June outage, IT teams at WhatsYourPrice.com pulled resources in-house and eventually moved them off Amazon EC2 to a Las Vegas-based hosting facility.
"The long-term plan is for us to have a distributed environment running the website on servers that are hosted in two data centers," Wade noted. "We were still going to depend on the cloud in case we have a burst [in site traffic]. We just had to put that into action a lot sooner than we expected."
O'Neill will keep his options open, too -- though it isn't because of Amazon's outages.
"Every year we seriously look at trying to unify [back-end and front-end resources] to a single cloud provider," he said. "We do it every year based on how big we are and where we think we're heading."
Michelle Boisvert is Senior Site Editor for SearchCloudComputing.com. Contact her at firstname.lastname@example.org.