This content is part of the Essential Guide: The CIO cloud blueprint: A strategic planning guide
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Cloud data integration: THINKstrategies offers advice, best practices

Editor's note: Following is a transcript of a presentation on cloud integration by Jeffrey M. Kaplan, managing director of the THINKstrategies cloud consulting firm. This transcript has been edited for clarity and editorial style.

View All Videos

Transcript - Cloud data integration: THINKstrategies offers advice, best practices

I think this is a very timely topic because, as we'll see as we proceed, there are a lot of market forces that are coming together to increase the importance of integration in the cloud.

Before I get into some of those trends, let me talk a little bit more about myself and THINKstrategies so that you'll have a better understanding of my point of view and the perspective that we bring to this topic.

I've been focused on the IT marketplace for the past 30 years, looking at how it's evolved from a product-centric to a services-driven model, in the telecom industry and the IT industry, and now looking at the software and services industries as well. THINKstrategies has been working with a variety of clients over the past 12 years to help them take advantage of this shift, whether they are solution providers who are trying to change the way they go to market to better deliver services, or IT or corporate executives who are trying to better understand how they can take advantage of these services to meet their corporate objectives. Even the investment community is trying to figure out which ponies to bet on and how to get horses across the finish line in today's rapidly changing market environment.

The bottom line is: We believe that the cloud is in fact not only changing the way in which technology and software is delivered, but the way in which organizations can take advantage of those solutions to meet their corporate objectives. The good news is: For the most part, it's working.

What we're seeing is a growing demand for these kinds of services. In fact, Gartner has suggested that new IT spending over the next couple of years will continue to shift towards cloud-computing platforms and applications, with nearly half of large enterprises having cloud deployments by the end of 2017. The Technology Business Research Group expects enterprises to spend almost $70 billion by 2018 on the deployment of private-cloud solutions to meet their own specific needs.

At the same time, it's an increasingly connected world that we live in. Cisco has been trying to measure that connected environment over the past few years with what it calls its Global Cloud Index, which tries to measure and predict what's happening in the way of global cloud traffic. They're expecting that traffic to grow four and one-half times over the next five years and ultimately reach 5.3 zettabytes of traffic by the end of 2017.

What we're talking about here is an explosive growth in a connected environment, which many people are now referring to as "the Internet of Things." That's brought about not only by the fact we have this new ability to connect various kinds of hardware and software together via the cloud, but also, of course, because the explosive growth of mobile end-user devices allowing us to access to those services on a day-to-day basis, not only for our consumer purposes but increasingly for commercial and corporate purposes as well.

So what we're seeing as a result is, along with the availability of a wide array of cloud services to provide computing power, an explosive growth in the adoption of a variety of Software as a Service or cloud applications … from office applications and customer relationship management systems on down to supply-chain and [human resources] capabilities.

In almost every one of these areas, we're seeing significant growth. In fact, THINKstrategies believes that the growth that’s measured by Gartner has really been dwarfed by the actual reality of the growth that's taking place. Sometimes [that's happening] in a clandestine fashion within organizations, where it's being driven not only by IT, but increasingly, by organizational needs throughout the strategic business units beyond IT.

So you can bet that in many cases this growth is happening more rapidly than even Gartner has been able to measure because IT organizations themselves are having a hard time keeping up with it. The point of all of this, though, is that it's creating a real challenge for IT and organizations as a whole to transform the way in which they manage not only the adoption process, but the ongoing organizational management of these applications to meet the corporate requirements as well.

Integration in the 'brave new world' of cloud

In the old world of the traditional data center, there was a highly defined method for adopting technology and software applications. This created an increasingly complex environment of highly siloed operational capabilities that required very customized solutions to tie them all together in an integrated fashion. Anybody who's worked in those traditional environments knows that while they may have gotten the job done, it was never easy. It was never a lot of fun to take advantage of traditional applications to get your job done. In fact, too often, those applications got in the way of your day-to-day activity.

In the brave new world of the cloud, the promises were that we're not only going to be better connected, but will have greater access to the applications we need in order to get our jobs done on a daily basis. Now that promise entails a series of integration steps in order to make it happen. What's proving to be a challenge is continuing to integrate this new layer of capabilities that are being delivered via the cloud into a legacy environment of software and systems that already exist on-premises and behind the firewall in the traditional data centers. So integration is a critical part of capitalizing on today's cloud-based alternatives.

What we've found is that not only are end users and corporate executives facing these challenges and recognizing the importance of them, but, increasingly, so are the service providers of Software as a Service and Platform as a Service and Infrastructure as a Service. [In one] survey, which THINKstrategies conducted in conjunction with MuleSoft, one of the integration players, we asked SaaS vendors themselves how they felt they were doing when it came to satisfying the SaaS integration needs of their customers. Unfortunately, they felt they were heading in the wrong direction. That is, in 2011, about 70% or so felt they were satisfying the needs of their customers. But a year later, in 2012, that number had dropped to about 55%. That's not the direction that you would hope we would be going at a time in which more and more people are depending upon Software as a Service and other cloud solutions.

There is a similar bit of information that came out of that same survey, which shows that the integration issue itself is becoming increasingly important. In fact, less than 10% of the SaaS vendors who we surveyed said that integration was not important in the consideration of Software as a Service solutions and other cloud capabilities.

Adopting a new approach to cloud integration

With all of this as the backdrop, what we're suggesting is that there needs to be a different approach to addressing the integration needs of the cloud. The good news is that the pieces, the parts, are beginning to come together. In fact, they're developing quite rapidly. They're evolving from the legacy world of the on-premises environments in a way in which we're seeing some increasing success.

The rapid growth of the cloud and SaaS marketplace would not be possible if it weren't for the on-demand alternatives. So where the operating systems of the past were really the underpinnings of the systems and software that existed in the on-premises world, it's all about Web services in the cloud. While data sources are absolutely an essential part of the architecture of integration, in the old world those data sources, obviously, are key determinants of how you go about integrating services and solutions in the cloud as well.

Where things get very interesting, though, is at the middleware level. There was a variety of proprietary middleware options available to us in the traditional world of on-premises software and solutions. In today's cloud environment there is a growing economy of APIs --application programming interfaces -- which really are the touch points for integrating off-the-shelf applications across the market's landscape and ensuring that these applications are, in fact, accessible and able to be used effectively by end-user organizations as the presentation-layer user interface that has been developed for today's applications as well.

So let's talk a little bit about this new idea of the API economy and what it means to today's cloud environment. What we're seeing is these capabilities being developed and supported by a wide array of integration tools, vendors and software developers, as well as professional service firms who are providing the skills to deploy them across a variety of corporate environments.

In the integration-tools segment of the marketplace, there is a wide assortment of vendors. We have some major players, like Informatica, who have been in the integration tools business for quite a while. We have specialized integration firms, like Scribe, who made their name in the past focused on integrating third-party applications capabilities into the Microsoft environment and increasingly are doing the same kind of thing in [the] world.

We have companies like SnapLogic, who are providing connectors that go from high-volume data-resource requirements to new big-data application opportunities. We have companies like MuleSoft, who are focusing in on API application integration tool sets that are hosted in the cloud.

It's interesting to note that even the major players like IBM and Dell have made acquisitions in this integration space to embed these tools into their broader set of capabilities. In the case of Dell, they acquired a company called Boomi and IBM [acquired] a company called Cast Iron Systems.

So the good news is that there are a variety of tool vendors who are offering API-oriented integration capabilities to help organizations address these cloud-to-cloud and cloud-to-on-premises integration requirements.

There are also a growing number of Platform as a Service alternatives that permit these APIs to be embedded into the software development environment so that application developers can more readily integrate their capabilities into third-party services and software solutions. The most prominent players in this space [start with], who has been pioneering this idea for a while with this Platform as a Service. Of course, Microsoft has been offering its Windows Azure capability as a combined infrastructure and Platform as a Service offering. Google has been doing the same kind of thing with its app engine capabilities. Oracle is increasingly trying to evolve its traditional development tools into a set of cloud enablement and development capabilities as well.

An interesting player in this market is Intuit, who has for a very long time, of course, been a major supplier of software solutions for [small and midsize businesses (SMBs)] and who has increasingly been pushing their products and customers towards the cloud. Intuit recently has made a series of acquisitions that enabled it to provide integration as a part of its platform through the form of an API marketplace to encourage integration across its third-party applications.

So what's great to see is that integration is being embedded into the development process to ensure greater availability of services across the marketplace. But there's still a need for skills to put these kinds of services in place. There's a new breed of systems integrator who's focused entirely on this challenge in the cloud, in Software as a Service. Companies like Appirio, Bluewolf and Cloud Sherpas are now competing with the established players in the market like Accenture, CapGemini and others who have traditionally been providing these kinds of services in the data-center environment for enterprises.

Capitalizing on the cloud

What we have as result is the opportunity to capitalize on the cloud by taking advantage of a new set of solutions that can help to ensure data integration across these cloud deployments. It's important to be able to overcome the complexity that's been created by adding this new layer of cloud services on top of the traditional on-premises environments -- and to be able to do it using easier-to-deploy tools that can fit into legacy systems and software, and then tie together those data sources to meet the day-to-day application needs of enterprises and SMBs.

The key piece of all of this is the APIs and the Web services that support them. The way I like to put it is: We haven't been able to eliminate the integration challenge but we've been able to shorten the distance between the two dots. With the added skills of today's new breed of systems integrators, we can accelerate that integration process and develop new applications more easily that can fit into a hybrid environment.

As you're thinking about these capabilities going forward, I believe the key criteria for selection is the following: What are the APIs that the application developer and service provider have to offer, and what is their ecosystem of third-party relationships by which they're able to deliver and deploy those capabilities?

These are the thoughts that we have about what's happening in the cloud in terms of integration, the issues you need to be aware of and some of the tools and techniques you should consider as you're moving forward. I hope this gives you a good starting point for addressing these increasingly complex integration requirements in the cloud. I can assure you that the industry is doing all it can to better support the integration needs -- but it will be up to you to find the right solutions to meet your specific requirements.

+ Show Transcript