Intuit suffers debilitating outage
What led up to Wednesday's catastrophic outage for Intuit? Nobody knows, and the company is still in crisis mode as it attempts to restore critical services to users of its popular online accounting services and development tools.
The last news around Intuit was an ominous report of struggles with a major new customer on Monday. One day later, Intuit, with about four million business customers, had gone dark.
The outage began Tuesday night during a "routine maintenance procedure" at the firm's flagship San Diego data center, which the company owns and operates. It took down every part of Intuit's public online presence from the main site to the portals, services and access to users' hosted data until late Wednesday evening. Services remain choppy today. A notice on the Intuit community site (Editor's note: The notice may change) gave scant details:
"An accidental power failure during that procedure affected both our primary and backup systems, taking a number of Intuit websites and services offline. While power was quickly restored, we're working diligently to validate our systems and bring them back into full operation," it said.
Intuit CIO Ginny Lee issued a personal apology and promised regular updates on its site and social media outlets like Twitter.
The notice failed to detail how a power outage of a large enough magnitude to knock out entire swaths of infrastructure was allowed to occur; modern data center operations segment electrical service delivery carefully to limit the amount of damage an outage can have.
Usually events like this are preceded by external catastrophes -- other cloud outages in recent memory include Rackspace's outage in Dallas last year; a truck took out external power and generators failed to kick in. A lightning strike took out an Amazon data center in similar fashion. One thing is certain; this is a failure of "the cloud," and customers are extremely irate.
Customers launch complaints at Intuit
"As a new online customer, I'm extremely upset with this matter. You would think a company such as yours would have a backup plan. I run my entire business off of your online service," read one comment in part. Another said that they had been sold on a promise that this kind of outage was covered by disaster recovery plans that did not appear to have kicked in.
"I was told when I signed up for this that Intuit had a remote server running a mirror copy that they could switch to if this kind of problem occurred. Where is it?????? I think somebody lied to me!!" read the comment from 'CustomGraphix'.
Other posters detailed lost business, stalled payroll for hundreds of employees and bounced checks. Many asked for compensation in return for lost funds. Intuit has disclaimers on its site that all online services are "subject to occasional downtime," and it is likely that any service-level agreements (SLAs) disclaim financial penalties for outages, but that is an unknown at this time.
Intuit's online services are a little different from most cloud computing providers. It is based around the most critical of all business data -- financial data -- and users are virtually guaranteed to suffer adverse impacts. Whereas a website or Web application running on, say, Amazon, may suffer downtime without permanent damage except to reputation, and Amazon is very clear upfront that users are responsible for their own data.
Intuit has quietly grown from a software maker into a Platform as a Service (PaaS) and Web services delivery firm that tightly binds its hosted, online services with its desktop accounting software products. An army of developers use the venerable QuickBase online database service and now the Intuit Partner Platform to integrate Intuit hosted customer data and application functions into third-party software and services; startup Recurly is doubtlessly regretting its Monday launch of a service based on Intuit Online.
Intuit responds to outage
Intuit just announced it has restored service to customer websites affected by this week's outage. The Intuit sites, including TurboTax Online, QuickBooks Online, Quicken and QuickBase, are live today following work that continued throughout the night. In some isolated cases, customers may need to refresh their browsers to connect to the sites.
"We're carefully monitoring the sites and the applications that support them to ensure we provide the services that customers expect. And they're performing well at this time," said Ginny Lee, Intuit's chief information officer. "We're deeply sorry for the disruption to businesses and consumers and appreciate their patience as we worked to resolve this problem.
"With service restored, it's our priority to work with those affected and resolve any issues caused by this outage."
Dig Deeper on Cloud disaster recovery