After two public failures in so many weeks, one being a distributed denial-of-service attack (DDOS) that took 18 hours to detect and the other a spanking from antispam advocates Spamhaus, Amazon Web Services (AWS) looked out of control. Its groundbreaking cloud computing services had been shown up by an inability to respond quickly to customer complaints traditionally viewed as routine, leaving users angry and uncertain about AWS's...
"Their abuse reporting system was inadequate," said frustrated Amazon user Richard Jowsey. Jowsey was hard particularly hit by the email blockade; he runs Death2Spam, a spam filtering service, entirely from within Amazon. When the blacklist went into place, his customers, largely enterprise and governmental agencies, found that their email was being falsely labeled as spam. For Jowsey, it was a nightmare.
"We naturally assumed [AWS] had done their bloody homework," said Jowsey.
The expectation was that when Spamhaus, a non-profit, voluntary service, listed all of Amazon's IP addresses as spammers, the online retail giant would respond as other hosting services do and shut down the offending spammers. Jowsey said that AWS failed to pay attention to the problem until it was too late.
Spamhaus' automatic notifications of trouble in the Amazon cloud went unnoticed and unresponded as well, he said, leading to the general ban. Jowsey said he was left without an explanation to give his customers, which put a severe strain on his credibility. His company guarantees service-level agreements (SLA) that mandate a free month of service for every hour of outage -- and he's hoping they'll be lenient with him.
In this case, he said his staff performed "emergency brain surgery" on his servers and opened a webmail portal, so that users who were denied normal email traffic could read mail being sent in. Once that was completed, he tried in vain to communicate with AWS support staff, as well as Spamhaus. Only one side responded.
While he calls himself proud to be a part of Amazon's cloud and an early believer in the technology, he thinks issues such as these hurt the whole cloud concept more than Amazon itself.
"It really damages the reputation of the cloud as viable for carrier-grade, enterprise-grade uses," he said.
Spamhaus CIO Richard Cox said via email that the problem was a fumble by Amazon, specifically due to its lack of response. He said that Spamhaus had notified AWS many times in the past that their service was being used for malicious purposes, and they have also repeatedly blacklisted portions of Amazon's IP address space. In this case, said Cox, malicious traffic increased one hundred-fold and prompted the blanket ban. Cox said AWS wasn't the only cloud service targeted; Amazon-competitor Rackspace, however, was able to shut down their abusers very quickly. "As a result, the listing was removed as soon as Rackspace had the situation under control, which was impressively fast," he said.
Cox said he regretted having to impose the blacklist, but Spamhaus had no other option once their warnings were ignored. He said he hoped Amazon would adopt policies to more closely monitor their cloud for abusers.
Other users, such as Shlomo Swidler, co-founder of start-up MyDrifts.com, think this is par for the course at Amazon. He feels it won't dent interest much for their base, who tend to be smaller, free-wheeling developers and start-ups willing to take risks. He believes that those most affected will be users like Jowsey, who run email-based businesses from within the cloud, and that larger entities will simply choose not to use EC2 for anything requiring email, mixing and matching their needs to Amazon's hands-off, fend-for-yourself approach. "Their handling of the situation could perhaps have been better coordinated, but Amazon's reputation has not suffered for this," said Swidler.
For those in the know, this may be the case, but AWS has been marketing its services as easy, cheap and available to all. The disconnect between that kind of perception and reality will cause the true damage to AWS's reputation, according to Christofer Hoff, security analyst and director of cloud computing at Cisco. Large organizations won't be satisfied with unexplained outages and Amazon's legendary silence, especially when very basic issues, such as shutting down a spam operator or load balancing a minor DDOS attack, take days instead of hours to resolve.
"I don't fault Amazon -- it's a fantastic service -- if you understand what you're getting into," Hoff said. Amazon is trying to have its cake and eat it too on self-service and automation, he said. They've built the first commercially successful compute cloud and popularized the concept. Now, they are feeling growing pains from too much early success, according to Hoff.
"Given the marketing, you'd think these problems don't exist," he said. Amazon has been so successful with its automated service and delivery that they can't respond fast enough when embarrassing outages occur. They simply aren't used to the level of transparency and accountability demanded by IT users. "There's no way to express how protection is provided or how that protection is expressed," he said, something Amazon will clearly need to fix.
Amazon seems to be taking steps to make sure the Bitbucket.org and Spamhaus debacles don't reoccur, and even though it seems to be gradually thawing out in public relations, the AWS cloud remains a black box in ways that are too much for many potential customers. A book or a pair of baby shoes can always be sent back or the purchase price refunded—not so for IT operations and fragile business reputations.
Carl Brooks is the Technology Writer for SearchCloudComputing.com. Contact him at firstname.lastname@example.org.