If you asked Google's App Engine team about the biggest challenges facing developers moving to the cloud, working with a nonrelational distributed database architecture would surely be near the top of the list.
"I think the toughest thing for developers coming from a traditional IT development stack is they are used to having a relational model," said Fred Sauer, App Engine developer advocate at Google. "And with that comes the single hardened instance where you might have a database server or a failover. But everything is centralized."
In a relational model, even when developers are working in a locally distributed grid, they are usually working with centralized data stores and machines they can tweak to optimize access performance. Not so in the cloud.
Sauer said learning how to develop under a distributed model is not necessarily more difficult, as much as it is a matter of relearning how to store and call data.
"With App Engine, you're talking about a platform that by its very nature is very distributed," said Sauer. "Every time your browser makes a request, you may hit a different server or a different part of the data store."
To ease Web application building
In other words, if you fetch 100 rows of data, the requests might go out to a few dozen machines, and still more machines might actually return the data. This forces developers to think about the ways they read, write and store data in ways that suit those patterns, Sauer said. But the distributed model used in App Engine can find those rows of data and bring them back all at once, in parallel, rather than in some relational pecking order.
[Editor's note: Google's view of a distributed nonrelational data model is best exemplified by its MapReduce and Hadoop data implementations. A recently reported Yale computer science project seeks to combine characteristics of both Hadoop and relational database management systems.]
Sauer said App Engine was built because Google realized that building Web applications was just too difficult a process.
"There was too much work that a developer would have to do in getting the application ... to scale," said Sauer. "The three goals for the App Engine were [it had to be] easy to start, easy to scale and easy to manage."
Make mine Python
Sauer said three of the favorite languages at Google are Python, Java and C++. But the Python developers were the ones who first gravitated toward the App Engine project.
"One of our biggest challenges was building out a service that could be used by non-Googlers," said Sauer. "It's one thing to make an internal tool and another to open up a platform where you have no control over the application code that's being submitted. We had to harden the environment."
The last thing a company offering a cloud computing development platform wants is for requests coming from one application to hinder the response times of another application. To prevent this, App Engine has some limitations. Sauer said developers are not allowed access to file systems on the cloud machines or to open arbitrary network sockets or requests.
In terms of application portability, Sauer said App Engine tends to be more welcoming of native code written in Java than Python. In April, Google App Engine added support for Java to its existing support for Python, which proved to be a popular language among the troops as the Mountain View, Calif.-based giant grew. With Google's cloud-based application development platform now open to a more mainstream language, many developers are taking a closer look.
Farewell, proprietary API
"We made it a first-class goal for application developers to implement their entire application with open standards without the use of proprietary APIs [application programming interfaces]," said Sauer.
But things do get complicated for developers wishing to port very complex, non-standards-based applications from a relational to a distributed model, Sauer said. As a general rule, he said, it's easier to build an application in App Engine and port it into a relational system than the other way around.
More than 100,000 applications have been built on App Engine, Sauer claimed. They include social applications such as BuddyPoke and smaller apps that monitor keywords on Twitter. Developers have built games and small Web sites, while businesses have written applications for internal use.