This document provides an introduction to integration platform as a service (iPaaS) and SnapLogic. It discusses the drivers for iPaaS adoption including big data, hybrid cloud environments, and the need for faster integration. Ten requirements for modern integration are outlined. The document then introduces SnapLogic and its unified platform for connecting applications, data and APIs anywhere through a library of pre-built connectors. Four primary iPaaS use cases are described: hybrid application integration, cloud data warehousing/analytics, big data ingestion/transformation/delivery, and replacing legacy integration platforms.
Connecting applications or data from multiple sources is not new – ESB, SOA, ETL have been around for a long time. But the old ways are not keeping up with today’s realities…
We all know that “data is eating the world.” Not only data from new sources such as wearables and sensors, but the sheer volume. The digital universe is doubling every two years and should reach 40,000 exabytes, or 40 trillion gigabytes by 2020.
Gartner conducts an annual survey of IT executives worldwide regarding their adoption of Big Data tools and techniques and many report challenges on their Big Data journey. Each year three items have been cited among the top 10 challenges or hurdles to Big Data adoption:
Shortage of people with relevant skills
Problems integrating multiple data sources
Connecting Big Data technology with existing infrastructure
All of these challenges can be overcome with SnapLogic.
Most organizations today have “hybrid” environments with systems on premises and in the cloud – if integration isn’t seamless, these can become information silos.
Becoming more common: multicloud, where organizations are managing a combination of private and public clouds from different vendors. This presents integration challenges that legacy systems simply cannot support.
These new requirements have given rise to a new category of integration called integration platform as a service (iPaaS), which should be built from the ground up to address the new and legacy enterprise application and data integration needs.
Our founder, Gaurav Dhillon, saw this change coming. He founded Informatica and led it to a successful IPO, so he knew that these legacy technologies weren’t going to keep up.
He founded SnapLogic specifically to address the new era of data. Andreessen Horowitz and Ignition believed in his vision and are our lead investors.
And we have a growing roster of enterprises across a number of industries who have chosen SnapLogic as their platform of choice for data and application integration.
Leading enterprises choose SnapLogic because we help them connect data and applications faster.
We connect anything: sources including applications, APIs, things, or data
We connect anytime: in batches, streaming, or in real time
And we connect anywhere: on premises, in the cloud or a combination of both
SnapLogic’s modern, elastic architecture supports hybrid environments. We like to say that SnapLogic “respects data gravity” and runs as close to the data as need be. If you are integrating only cloud applications, it would make no sense to run your integrations behind the firewall. Similarly, if you’re doing ground to ground or cloud to ground, you may want to run your Snaplex on premises or on a private cloud.
With SnapLogic, you can run all of these integrations with a single platform.
This is just a sampling of the available technologies that may go into a data lake.
To date, most data lake deployments have been built through manual coding, open source tools and custom integration.
Manual coding of data processing applications is common because data processing is thought of in terms of application-specific work. Unfortunately, this manual effort is a dead-end investment over the long term because the underlying technologies are constantly changing.
Older data warehouse environments and ETL type integration tools are good at what they do, but they can’t meet many of the new needs. The new environments are focused on data processing, but require a lot of manual work.
The data lake must incorporate aspects of old data warehouse environments like connecting to and extracting data from ERP or transaction processing systems, yet do this without clunky and inefficient tools like Sqoop. The data lake also must support new capabilities like reliable collection of large volumes of events at high speed and
timely processing to make data available immediately. It must also support data coming from multiple sources in a hybrid model. This exceeds the abilities of traditional data integration tools.
SnapLogic accelerates development of a modern data lake through:
Data acquisition: collecting and integrating data from multiple sources. SnapLogic goes beyond developer tools such as Sqoop and Flume with a cloud-based visual pipeline designer, and pre-built connectors for 300+ structured and unstructured data sources, enterprise applications and APIs.
Data transformation: adding information and transforming data. SnapLogic minimizes the manual tasks associated with data shaping and makes data scientists and analysts more efficient. SnapLogic includes Snaps for tasks such as transformations, joins and unions without scripting.
Data access: organizing and preparing data for delivery and visualization. SnapLogic makes data processed on Hadoop or Spark easily available to off-cluster applications and data stores such as statistical packages and business intelligence tools.
SnapLogic for big data integration benefits include:
Improved Big Data Acquisition: Instantaneous access to hundreds of data sources from within Hadoop via pre-built connectors, called Snaps. Going beyond developer tools like Sqoop and Flume, SnapLogic allows Hadoop users to access and introspect cloud and on-premises data sources in an easy-to-use graphical interface.
Double Data Scientist Productivity: Most data scientists spend the majority of their time on cumbersome coding tasks, data gathering and data preparation. SnapReduce 2.0 doubles data scientist productivity by simplifying complex data shaping tasks such as transformations, joins and unions. We call it Hadoop for Humans.
Universal Big Data Delivery: Make Hadoop data and analytics results easily available to off-cluster applications and data stores such as statistical packages and business intelligence (BI) or visualization tools.
Unified: At SnapLogic, we’re 100% focused on delivering a single, unified platform that can handle ETL (or ELT) requirements and low-latency ESB requirements.
Productive: When we talk about connecting faster, a bit part of that is the productivity gains that our customers tell us about. Integration is a hard problem to solve, and we’re focused on simplifying it as much as possible with a very easy to use, drag and drop design interface for both citizen and advanced integrators.
Modern: the SnapLogic modern architecture goes back to our ability to handle both real-time, low-latency integration requirements as well as big data volume, variety and velocity.
And finally, Connected. You’ll see in the demonstration the breadth and depth of our Snaps. We have over 300 Snaps and a Snap Development Kit (SDK) for building custom Snaps.
According to a recent Gartner report: “Unnecessarily segregated application and data integration efforts lead to counterproductive practices and escalating deployment cost.”
When we talk about connecting faster, a bit part of that is the productivity gains that our customers tell us about. Integration is a hard problem to solve, and we’re focused on simplifying it as much as possible with a very easy to use, drag and drop design interface for both self-service and advanced integrators.
As you can see from Adobe, who has over 100 users of SnapLogic today, they’re able to do more in 2 hours with our cloud integration service than they could do in 2 days with traditional solutions.
Each Snaplex – whether a cloudplex, groundplex, or Hadooplex – can elastically expand and contract based on data traffic.
The unit of scalability inside Snaplex is a JVM.
Each Snaplex is initialized with a preconfigured number of JVMs (say one, for example). Once the utilization of that one JVM reaches a certain threshold (say 80%), a new JVM is automatically spun up to handle any additional workload. Once this excess data traffic has been processed and the second JVM is sitting idle, it gets torn down to scale back in to its original size.
And finally, Connected. You’ll see in the demonstration the breadth and depth of our Snaps. We have over 300 Snaps and a Snap Development Kit (SDK) for building custom Snaps.
Reference: http://www.snaplogic.com/snaps
Why do customers choose SnapLogic over the alternatives?
Unified: At SnapLogic, we’re 100% focused on delivering a single, unified platform that can handle ETL (or ELT) requirements and low-latency ESB requirements.
Self-service: When we talk about connecting faster, a bit part of that is the productivity gains that our customers tell us about. Integration is a hard problem to solve, and we’re focused on simplifying it as much as possible with a very easy to use, drag and drop design interface for both citizen and advanced integrators.
Modern: the SnapLogic modern architecture goes back to our ability to handle both real-time, low-latency integration requirements as well as big data volume, variety and velocity.
And finally, Connected. You’ll see in the demonstration the breadth and depth of our Snaps. We have over 300 Snaps and a Snap Development Kit (SDK) for building custom Snaps.