NASDAQ puts market data in the cloud

The NASDAQ stock exchange is turning its data assets into a true cloud service and putting Amazon's cloud storage to unique use.

The NASDAQ stock exchange is taking its vast store of public market data to everyone by making it available as a cloud computing service. It's also proving out another interesting use for cloud storage in the way it has put Amazon's Simple Storage Service (S3) to work.

This year, we've seen [cloud computing] pick up in a big way.

Chas Cooper, marketing director at Xignite

NASDAQ collects and sorts hundreds of thousands of pieces of information about the stocks it trades every day for its Market Replay service, an online viewing tool for researchers and analysts.

"We have every single quote that comes out of the NASDAQ securities information processor and the SIAC processor for the last three years," said Jeff Kimsey, associate vice president for product management at NASDAQ OMX. The data is Level I stock information, a count of what stock was traded and for how much, and the new service was a whole new way to think about accessing and using that data.

The NASDAQ Data-On-Demand service will allow users to connect programmatically to NASDAQ's massive data store instead of having to either look at it online or download giant chunks of raw data and sort it themselves. An application programming interface (API) will let advanced users (financial analysts are known to tinker with computing) run queries against the data directly, so existing analytical tools can simply tap into it directly.

A customer could get a specific targeted piece of information for researching compliance, training or their own historical information, Kimsey said. It will give algorithmic traders the ability to "back-test" new methods of gaming the stock market by running their automated trading schemes against years of real performance data, something that will be a boon to those who study the market in detail.

"If you're creating a historical application, all you really need to do is flip a switch," he said.

NASDAQ delivers this data as a cloud service, but that's possible because it's already got that data on S3. The technical challenge lies in making sense of the flood of information that comes in every day.

Flat files speed transaction handling
Some heavily-traded stocks might have many megabytes worth of transactions recorded in a day, while others will have much less. In order to break the data into manageable, easily catalogued chunks, it is stored in 10-minute increments in flat files on S3. Search and query start from there. This was the most efficient way to store the data quickly and cheaply, Kimsey said.

"Our application was really designed to get to a specific instance in time," he said, "and because we wanted to make the application really quick to get to, we store it in 10-minute [intervals] -- that allows for quick, easy access to data that's meaningful."

"That was the nature of the challenge: we're dealing with terabytes of data in each database, how we do that without creating a million-dollar database?" Kimsey asked.

Since NASDAQ has so much historical data, the costs to store and use the data on Amazon Web Services (AWS) can add up to hundreds or possibly thousands of dollars per month; spare change from an enterprise perspective on operating costs. Kimsey said the service will be available in a few weeks. At that time, users will be able to sign up and get started without assistance, or they can work with NASDAQ to get greater access.

NASDAQ worked with Xignite for the new service, which specializes in on-demand data products for the finance world. Chas Cooper, marketing director at Xignite, said the financial firms and information processors like the NASDAQ are starting to feel the pressure to get with the times.

"This year, we've seen [cloud computing] pick up in a big way. We're kind of a bellwether for financial services in the cloud," he said.

If you're creating a historical application, all you really need to do if flip a switch.

Jeff Kimsey, associate vice president for product management at NASDAQ OMX

Cooper said Xignite delivered a Web services framework on top of the data that NASDAQ collected. The company had constructed a number of actions that could be performed against the data, and Cooper added that it was relatively straightforward to make more if a user had new ideas for queries. "We provide Web services at the API layer where you would plug in the data access layer," he said.

Cooper thinks this is part of a long-term trend; in the financial sector, where so much important information is publicly available and highly regulated, something he calls the "market data cloud" will eventually be fairly standard.

NASDAQ's Jeff Kimsey agrees. He's watching his business go from processing information to distributing it.

"Having less and less data stored in different spaces [and more in central repositories] is where we're going," he said.

Carl Brooks is the Technology Writer at Contact him at

Dig Deeper on Public cloud and other cloud deployment models

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.