Welcome!

Big Data Journal Authors: Carmen Gonzalez, Liz McMillan, Victoria Livschitz, Elizabeth White, Pat Romanski

Related Topics: Cloud Expo, Java, SOA & WOA, Virtualization, Web 2.0, Big Data Journal, SDN Journal

Cloud Expo: Article

Creating Harmony When Cloud and On-Premise Worlds Collide

Integrating data across diverse SaaS applications with existing on-premise solutions has proved exceptionally challenging

In recent years, IT departments have been confronted with the convergence of several highly disruptive trends that have fundamentally altered the enterprise IT landscape, particularly when it comes to how data and applications are managed. Mobility and the rise of BYOD (bring your own device), as well as the growth of social media and the electronic information it generates, have each proved transformative. But perhaps no shift has been more seismic than the adoption of cloud and SaaS-based applications led by CIOs who see the value proposition associated with outsourcing many complex IT operations.

However, integrating data across diverse SaaS applications with existing on-premise solutions has proven exceptionally challenging. To streamline this integration without slowing adoption, IT stakeholders are turning to cloud-based integration solutions that can curtail complexity and IT oversight while enabling organizations to better leverage their information capital to drive business objectives. Indeed, according to a recent report by analyst firm MarketsandMarkets, the global Cloud Brokerage Services (CSB) market is on track to grow from $1.57 billion in 2013 to $10.5 billion by 2018, a compound annual growth rate of more than 45% over the five year period.

In this article, we will provide advice to IT leaders for creating sustainable environments using hybrid integration between SaaS technologies and existing on-premise applications. We will also explore the top considerations for building out a successful cloud integration strategy that offers the scalability and flexibility to withstand fluctuations in enterprise data management needs.

Start by Asking the Right Questions
Over the past few years, "Cloud" has transformed from the buzzword of the moment - all the rage but lacking concrete definition - to an efficient, widely recognized enabler of scalable IT operations. Despite the increasing ubiquity and viability of the cloud delivery model, it's important to remember that cloud is not "IT in a box." No one cloud service provider can meet all the complex IT needs of a single organization. By and large, enterprises evaluate and onboard an array of purpose-built solutions from diverse cloud providers. As a result, the need to successfully integrate them not only with each other, but also with traditional on-premise application-to-application (A2A) and business-to-business (B2B) systems is critical. The multitude of complex integrations - A2A, B2B, and on-premise applications to SaaS/cloud applications, and cloud-to-cloud (C2C) - requires a clear-cut integration strategy.

A critical first step in developing an integration strategy is to ask and answer a few key questions, the first of which is "what problem is the integration solving?" While achieving streamlined integration between cloud-based systems like Magento, NetSuite, SAP, Ariba, and salesforce.com is one aspect of a full-fledged strategy, it's important to remember the challenge extends beyond cloud-to-cloud integration. In reality, what many people today refer to as "cloud integration" is actually hybrid integration - integration not only between cloud systems, but between cloud and on-premise applications. Determining the specific integration goal - whether it is strictly cloud-to-cloud, or a larger hybrid model - ensures the strategy scales to both immediate and long-term integration needs.

Once you consider what problem the integration will solve, it's important to consider how integration will solve the problem. As the number of systems to be integrated grows, the number of potential interface points expands exponentially, and traditional, manually driven point-to-point integration can quickly become overwhelming. Each time an individual application is altered, or a trading partner changes its specification interface, IT must review all external connections for potential impact. An upgrade cycle for a large ERP system may spawn dozens, hundreds, or even thousands of integration projects across several departments and external trading partners.

Continuing to rely on this point-to-point integration model will become untenable as cloud adds another layer of complexity to the integration landscape. In order to avert chaos, enterprises are actively leveraging integration to create an interconnected web that holistically addresses data management and integration challenges across all of these disparate systems and applications. If an integration strategy is designed with a broader goal in mind, it is much more likely that the same strategy can be leveraged not only to solve immediate integration challenges, but future demands as well.

Identifying where integration is needed and how it can benefit an organization is an important first step. But once the decision has been made to move forward, there are a few key considerations that CIOs must take into account to successfully build out a strategy with staying power.

Reading the Signs: Spotting and Addressing Complexity
Anticipating the areas in which integration complexity is most likely to arise is crucial to the development of a flexible, cost-effective integration strategy. The following are two of the usual suspects of which CIOs should be aware:

  1. SaaS APIs: Many cloud providers promise to deliver a simple-to-use web API, but this is rarely the reality. Specifications for many SaaS APIs can run into the dozens, if not hundreds, of pages long, and can be a major headache for internal teams unfamiliar with the nuances of integration. Moreover, APIs often evolve over time as SaaS applications evolve, generating a source of ongoing complexity.
  2. Data Translation: The potential for complexity, however, does not end once the APIs are successfully integrated. Translating data between different SaaS applications, as well as between SaaS and on-premise systems, can be challenging, and this translation should be factored into the complexity calculus. Data that is not properly translated will be rendered useless, and backtracking to fix the glitch can add time and expense to business-critical projects. As a general rule, a bug that costs one dollar to fix during development will cost 10 dollars to fix during quality assurance, and 100 dollars fix once in production. This backtracking approach can prove particularly brittle when new systems are added to the ecosystem.

A Long-Term Vision: Thinking Beyond the First Integration Project
Integration with cloud is often a daunting prospect, particularly for businesses just beginning to onboard cloud applications as part of their IT strategy. The immensity of a single cloud integration can produce tunnel vision for IT teams, who get so bogged down in an initial project that they fail to consider the long-term implications of the integration and how it will ultimately fit into the overarching IT architecture - a problem already amply demonstrated with the pitfalls of the point-to-point approach. However, the inevitable complexity of integrating multiple applications over time should be sufficient incentive to give any CIO pause before creating a strategy tailor-made for a single integration project.

Even though it will likely require greater upfront investment and effort, organizations must settle on a cohesive sourcing strategy for integration that meets their individual needs. There are three fundamental options for this strategy: a do-it-yourself (DIY) approach based solely on existing knowledge of on-premise software; a DIY approach using a customer-driven integration Platform-as-as-Service (iPaaS); or outsourcing integration entirely to a third-party integration brokerage provider. When determining which of these strategies to adopt, it is important to consider the following:

  1. First, consider the deployment timeline. As departments across the enterprise demand rapid access to new and greater functionality offered by diversifying SaaS applications, IT departments are under mounting pressure to test, procure and deploy these solutions. This is where a CSB can help speed things up based on their experience working with various customers, implementation scenarios and technologies. Even as deployment windows tighten, however, many businesses are only just beginning to build out core competency around integration. For those with the strictest timelines, the option to build out an internal integration function may have already passed, and it may become necessary to bring in a third-party integration provider. While some may initially view these external integration providers as a Band-Aid solution, working with a specialized integration broker can often be the best long-term solution, especially when it comes to cloud integration where existing IT teams may have less familiarity.
  2. Second, consider the cost for integration in the long term. As the complexity of cloud integration projects continues to increase, building out an internal team will require a capital investment in expert personnel and software. Although it requires greater initial investment, this relatively fixed capital expenditure may be a better use of resources for some organizations. For others, such a large capital expenditure may not be feasible or efficient. Outsourcing projects to an integration broker shifts the cost of integration as an operating expense, reducing or eliminating the up-front cost, and providing a more scalable, recurring cost-structure.
  3. Once these factors have been weighed, the next decision is: in-house or external? Although SaaS applications for both back-office systems and B2B processes can offer tremendous efficiencies, the coordination and integration required on the back end is no simple matter. While building out in-house integration capabilities is important for some organizations due to commercial or other business considerations, companies that choose this route must recognize it early and take a proactive approach to cultivating the expert staff and resources that will be required to effectively manage and complete integration projects. For those businesses that don't have compelling reasons to keep the integration function in-house, outsourcing may prove more efficient. Cloud Services Brokers (CSBs) have existing integration infrastructure that can be leveraged for rapid deployment, and can increase capacity on demand, offering scalability when and where it's needed most. CSBs also deliver experience and collective intelligence around integration that can offer efficiencies beyond what can be accomplished with internal resources alone.

The key criteria and requirements around data management continue to expand, and cloud integration is at the nexus of this expansion. By planning and executing a comprehensive integration strategy that can efficiently and consistently scale to the evolving integration requirements of the business - including traditional on-premise, back-office systems and cloud-based applications - IT can help ensure the long-term scalability and business success. Whether the decision is to bring integration capabilities in-house, outsource integration needs, or use some combination of both, the time to start developing a plan is now.

More Stories By Rob Fox

Rob Fox is Vice President of Application Development for Liaison Technologies, and the architect for several of Liaison’s data integration solutions. Liaison Technologies is a global provider of cloud-based integration and data management services and solutions. He was an original contributor to the ebXML 1.0 specification, is the former chair of marketing and business development for ASC ANSI X12, and a co-founder and co-chair of the Connectivity Caucus. Connect with Rob on Twitter: @robert_fox1

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@BigDataExpo Stories
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
The 4th International DevOps Summit, co-located with16th International Cloud Expo – being held June 9-11, 2015, at the Javits Center in New York City, NY – announces that its Call for Papers is now open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's large...
SAP is delivering break-through innovation combined with fantastic user experience powered by the market-leading in-memory technology, SAP HANA. In his General Session at 15th Cloud Expo, Thorsten Leiduck, VP ISVs & Digital Commerce, SAP, discussed how SAP and partners provide cloud and hybrid cloud solutions as well as real-time Big Data offerings that help companies of all sizes and industries run better. SAP launched an application challenge to award the most innovative SAP HANA and SAP HANA...
The security devil is always in the details of the attack: the ones you've endured, the ones you prepare yourself to fend off, and the ones that, you fear, will catch you completely unaware and defenseless. The Internet of Things (IoT) is nothing if not an endless proliferation of details. It's the vision of a world in which continuous Internet connectivity and addressability is embedded into a growing range of human artifacts, into the natural world, and even into our smartphones, appliances, a...
Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water,...
The Internet of Things is tied together with a thin strand that is known as time. Coincidentally, at the core of nearly all data analytics is a timestamp. When working with time series data there are a few core principles that everyone should consider, especially across datasets where time is the common boundary. In his session at Internet of @ThingsExpo, Jim Scott, Director of Enterprise Strategy & Architecture at MapR Technologies, discussed single-value, geo-spatial, and log time series dat...
How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happe...
An entirely new security model is needed for the Internet of Things, or is it? Can we save some old and tested controls for this new and different environment? In his session at @ThingsExpo, New York's at the Javits Center, Davi Ottenheimer, EMC Senior Director of Trust, reviewed hands-on lessons with IoT devices and reveal a new risk balance you might not expect. Davi Ottenheimer, EMC Senior Director of Trust, has more than nineteen years' experience managing global security operations and asse...
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
"SAP had made a big transition into the cloud as we believe it has significant value for our customers, drives innovation and is easy to consume. When you look at the SAP portfolio, SAP HANA is the underlying platform and it powers all of our platforms and all of our analytics," explained Thorsten Leiduck, VP ISVs & Digital Commerce at SAP, in this SYS-CON.tv interview at 15th Cloud Expo, held Nov 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Enthusiasm for the Internet of Things has reached an all-time high. In 2013 alone, venture capitalists spent more than $1 billion dollars investing in the IoT space. With "smart" appliances and devices, IoT covers wearable smart devices, cloud services to hardware companies. Nest, a Google company, detects temperatures inside homes and automatically adjusts it by tracking its user's habit. These technologies are quickly developing and with it come challenges such as bridging infrastructure gaps,...
"Cloud consumption is something we envision at Solgenia. That is trying to let the cloud spread to the user as a consumption, as utility computing. We want to allow the people to just pay for what they use, not a subscription model," explained Ermanno Bonifazi, CEO & Founder of Solgenia, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Ar...
SAP is delivering break-through innovation combined with fantastic user experience powered by the market-leading in-memory technology, SAP HANA. In his General Session at 15th Cloud Expo, Thorsten Leiduck, VP ISVs & Digital Commerce, SAP, discussed how SAP and partners provide cloud and hybrid cloud solutions as well as real-time Big Data offerings that help companies of all sizes and industries run better. SAP launched an application challenge to award the most innovative SAP HANA and SAP HANA...
Vichara Technologies in Hoboken, New Jersey is expanding its capabilities in big data from origins on Wall Street into other areas, and thereby demonstrating the growing marketplace for advanced big-data analytics services. The next BriefingsDirect deep-dive big data benefits case study interview explores how Vichara Technologies in Hoboken, New Jersey is expanding its capabilities in big data from origins on Wall Street into other areas, and thereby demonstrating the growing marketplace for ad...
The definition of IoT is not new, in fact it’s been around for over a decade. What has changed is the public's awareness that the technology we use on a daily basis has caught up on the vision of an always on, always connected world. If you look into the details of what comprises the IoT, you’ll see that it includes everything from cloud computing, Big Data analytics, “Things,” Web communication, applications, network, storage, etc. It is essentially including everything connected online from ha...
Scene scenario: 10 am in a boardroom somewhere, second round of coffees served, Danish and donuts untouched, a quiet hush settles. “Well you know what guys? (and, by the use of the term guys I mean to include both sexes here assembled) – the trouble that we have as a company is that we are, to put it bluntly, just a little analytics poor,” said the newly appointed Chief Analytics Officer. That we should consider a firm to be analytically deficient or poor is a profound comment on our modern ag...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world.
SYS-CON Events announced today that Windstream, a leading provider of advanced network and cloud communications, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Windstream (Nasdaq: WIN), a FORTUNE 500 and S&P 500 company, is a leading provider of advanced network communications, including cloud computing and managed services, to businesses nationwide. The company also offers broadband, p...
The major cloud platforms defy a simple, side-by-side analysis. Each of the major IaaS public-cloud platforms offers their own unique strengths and functionality. Options for on-site private cloud are diverse as well, and must be designed and deployed while taking existing legacy architecture and infrastructure into account. Then the reality is that most enterprises are embarking on a hybrid cloud strategy and programs. In this Power Panel at 15th Cloud Expo (http://www.CloudComputingExpo.com...