news image

5 hidden challenges of Cloud data integration and how to overcome them

13 Apr 2016

Enterprises have been slow to move big data processing to the cloud, but not for lack of trying. We looked into the case to find out why and shared our insights on the matter

Most companies nowadays use the public cloud in some form, mostly for SaaS applications. However, even though their numbers are growing, according to a 2014 Gartner survey, less than half of organizations with big data programs reported using the cloud in any form. This means that enterprises have been slow to migrate big data and data warehousing to the cloud, despite cost, scalability and elasticity benefits. Security concerns are not good enough of a reason to miss out on the huge variety of great services available today. So why aren’t more companies using the cloud for data processing? Let’s make a quick reality check on why enterprises are still avoiding to use the cloud for big data.



Cloud vs data centers


While there are many young digital companies, whose businesses are entirely online and built with cloud-based systems from the start, there are also big enterprises with massive infrastructures and established on-premise data centers. For them using the cloud would require significant investment in learning, maintaining, and integrating two environments – technically, culturally, and operationally. What’s more, cloud services are set up, configured, and priced differently than standard servers in a data center. Also, budgeting and managing costs changes completely under the paradigm of cloud space rental versus the traditional model. It is well-known how extremely difficult it is to integrate cloud services with existing systems. And that’s just setting up basic services – never mind architecture, deployment, integration, and managing the cloud.



Cloud security vs lack of resources/expertise


Security problems often result from inconsistent, ad hoc use of the cloud, or poorly-defined cloud policies, leading to human mistakes. Gartner predicts that through 2020, 95 percent of cloud security failures will be the customer’s fault. Instead of being scared off by this statistic, enterprises simply need to formally address cloud security, leaving no room for interpretation. Companies must also learn to correctly use the security features offered by cloud providers, and choose vendors that can work with their existing management systems and policies.  Since 2016 security is officially no longer the number one challenge for cloud integration, according to the RightScale: State of the Cloud Survey.  The leading position is taken by insufficient resources and expertise. Companies are struggling to find the right cloud services provider for their specific business needs, especially since big data and cloud expertise is so difficult (and expensive) to acquire. It’s no secret that there is a major skills shortage, given the fast pace of global digitalization and technology innovation.



From experimentation to production


Getting from research and prototyping to production is a big leap. While it may be easy for a developer to spin up a quick experiment, it is notoriously challenging to integrate cloud services with existing systems. Many companies report that they have piloted the cloud for big data, however, they find it hard to make the cloud an integrated part of production data processes. In addition to the complex data and system integration work, adding a new cloud environment also brings change management and operational hurdles. Enterprises report it’s more difficult for them to determine and uphold service-level agreements (SLAs) for cloud services. How can two-second response times for business intelligent queries be delivered most cost-effectively, versus overnight batch processes? Overall, it’s a major challenge to support the cloud as they do other services. Keep in mind that large enterprises that have successfully developed their own cloud analytics capabilities typically invest several years and millions of dollars in the effort.



Data movement


The emphasis is also misplaced in the well-covered issue of moving data into the cloud. Many discussions focus on how giant petabyte-sized datasets move from data centers into the cloud. While In fact, significant volumes of data like this are usually moved in physical media like Amazon Snowball or by other ground transportation. But these are often one-time, massive transfers, not ongoing updates. The real challenge is making data movement to and from the cloud a seamless part of the enterprise data flow. Discussions about data movement to the cloud must focus on streaming, micro-batching updates, and data pipelines. For the cloud to make an impact on production processes, enterprises must consider ongoing, two-way data movement.



Cloud services standards


Emerging tech markets often produce wildly varying labels, capabilities, and pricing models. This is especially true for data processing in the cloud, where buyers must dig deep to understand the differences between numerous point solutions for specific technologies and broader managed service offerings. Some services are pay-as-you-go; some are flat-rate. They include different levels of monitoring, management, and SLAs. And cloud vendors vary in their levels of operational support. So the big question is how to find cloud providers that can fit comfortably with your existing processes.



Our advice


Now, is spite of everything said, none of these challenges should keep you from staying out of the cloud. There is a right approach to start using cloud services without making your business suffer and we will gladly share our business insights. First of all, it’s key to pick cloud projects carefully, choosing well-scoped endeavors that augment your overall capabilities. Find the right balance of the cloud services you source versus strategic skills you develop in-house. Afterwards, we recommend evaluating each service provider carefully with a similar requirements matrix to test their integration capabilities upfront and be able to determine how the solution will fit into your existing architecture. Understand exactly what you’re getting from your cloud provider for security, and delegate roles and responsibilities for cloud management. In the end, the benefit from these services is that they provide a great alternative to building a custom cloud data platform from scratch, allowing you to focus your resources on your data strategy and core business activity.



Reality - checked


The cloud is no longer a DIY environment for developers. While enterprises have been slow to move to the cloud, vendors have been busy developing new services to ease the transition. With its processing power, scale and economics, cloud is the future of big data analytics.  As savvy digital natives have proven, using the cloud for analytics enables a major competitive edge. We at Imperia Mobile are fully capable of assisting you in successfully moving big data processing to the cloud and we will help you rise above the challenges of today’s digital evolution.