This presentation introduces the concept of the "Integrator's Dilemma" and reviews some of the challenges faced by traditional data and application integration technologies when it comes to keeping up with the new enterprise data, application and API connectivity and management requirements. We review the landscape and share examples of the steps more and more IT organizations are taking to improve business alignment through faster access to trusted data.
To learn more, visit http://www.snaplogic.com/ipaas
[2024]Digital Global Overview Report 2024 Meltwater.pdf
The Impact of SMACT on the Data Management Stack
1. The Impact of SMACT on the
Data Management Stack
Darren Cunningham, SnapLogic
2. About Me
• Responsible for Marketing @ SnapLogic
o Co-founded by Gaurav Dhillon, co-founder and former CEO of Informatica
o www.SnapLogic.com
• Prior to SnapLogic:
o 4 years at Informatica (Cloud Team)
o 2 years at LucidEra (Cloud BI)
o 1 year at Salesforce (Analytics / AppExchange)
o 7 years at Business Objects
3. The New Enterprise Integration Challenges
Big Data Access and Analytics Disconnected SaaS Silos
API Proliferation The Internet of Things
5. of line of business employees
admit to using non-approved
SaaS applications
in their jobs
The global Hadoop
Market is estimated to reach
$50.2 billion by 2020.
It was valued at $1.5 billion in
2012 and is expected to grow
at a CAGR of 58.2%
between 2013 to 2020
6. The Integrator’s Dilemma
Old Approaches Not Built for the New Data Challenges
Legacy EAI
• Not built for the web
• On-prem ESB (xml,
soap)
• Code-intensive
Legacy ETL
• Built for rows and columns
• Batch-oriented
• Struggles with real-time
7. The Integrator’s Dilemma is faced when legacy integration
solutions are no longer effective in the new world of Social,
Mobile, Analytics (Big Data), Cloud and the Internet of Things
2003 2014
10. So What’s Changed?
2 User (and Buyer) Expectations
Citizen integrators are proliferating,
in most cases outside of the control
or visibility of whoever in the
company is commissioned with
fulfilling integration.
11. So What’s Changed?
T3he Data (Volume, Velocity, Variety, right?)
Top 5 Use Cases:
1) Customer Analytics
2) Operational Analytics: Understanding
Machines, Devices and Human
Interactions
3) Fraud and Compliance
4) Data-Driven Products & Services
5) EDW Optimization
Source: Datameer
12. So What’s Changed?
C4loudification and “Data Gravity” (iPaaS, elastic scale, opex)
CRM
HRMS ITSM
SCM
Analytics
ERP?
15. Whitepaper on Cloud Integration
• From XML to JSON
• From SOAP to REST
• From ESB to iPaaS
Working on ETL Paper…stay tuned!
Buses Don’t Fly in the Cloud
Why the ESB Is the Wrong Approach
For Cloud Integration
19. Option
• Determine if you’re suffering from the “Integrator’s Dilemma”
o Multiple teams and tools for EAI, SOA, ESB / ETL, ELT; old tech used to solve new
challenges
• Review your ICC / COE practices
o Command and control won’t fly in the age of the “Citizen Integrator” and need for
speed
• Do an audit of your cloud applications currently in use
o Find out how/if they’re integrated and how it’s going (typically many SaaS silos)
• Dig into Hadoop
o The economics and scale will make it a key component of your data infrastructure
• Investigate AWS Redshift other cloud DW / BI options
21. SnapLogic: Connecting Data, Apps and APIs
• Experienced Team: Leadership from Informatica,
Salesforce, Microsoft
• Investors: Andreessen Horowitz & Ignition
• Advisory Board: AstraZeneca, HP, Symantec, Yahoo
• Headquarters: San Mateo, CA
• Customers: Adobe, Acxiom, Blackberry, Bloomin’
Brands, CapitalOne, Cisco, GE, iRobot, Netflix…
22. Why SnapLogic Elastic Integration?
Fast Multi-Point Modern
Easily Design Monitor,
Manage
Deploy in Days not Months
• EAI, ETL, APIs, Hybrid
Deployment
• Modern Standards: REST,
JSON
• Scale Out Architecture
REST
SOAP
WEB
APIs
23. Hybrid Architecture
• Streams: No data is
stored/cached
• Secure: 100%
standards-based
• Elastic: Scales out
& handles data and
app integration use
cases
Metadata
Data
25. SnapReduce and the Hadooplex
Acquire, Prepare, Deliver Big Data
YARN
YARN
MapReduce
MapReduce
MapReduce
MapReduce
MapReduce
MapReduce
MapReduce
MapReduce
Snaplex YARN Application
MapReduce Generation
Snaplogic iPaaS + Hadoop
= Snaplex Container
26. Common SnapLogic Use Cases
Cloud App Integration
• Workday: HR On-
Boarding
• Salesforce: CRM
Back Office
• Eliminate SaaS Silos
Digital
Marketing
• AWS Redshift
• Tableau, Social, CRM
• Cloud Analytics
Big Data Analytics Enterprise
Platform
• Data Access
• Data Preparation
• Data Delivery
• Self Service
• Data, Apps, APIs
• Integrator’s Solution
27. We looked at SnapLogic as an opportunity to think differently
about integration. With a document-centric processing
approach and schema-less platform, we’ve been able to
eliminate some of the rigidity and time-consuming tasks related
to traditional integration patterns.
– Jim Teal, Information Architect, iRobot
$555M revenue, 534 employees
28. Designs and builds robots that
make a difference in people’s
lives
Challenges Faced:
• Geographically distributed manufacturing
• Wanted to improve data quality and eliminate FTP
and VPN channels
• Growing cloud app adoption
Why SnapLogic?
• Flexibility: Control plane in Boston, data planes in
China, schema-less approach
• Connectivity: Standards connectivity, eliminated
VPNs, etc.
• Productivity: 68 to 6 pipelines
29. Largest apparel brand ever
built on the web in the United
States.
• 1 month vs. 6m-1yr
• Reduction in headcount to administer and maintain
• Lowers the barrier to entry for advanced analytics
Speed
Flexibility
Scale
Amazon
Redshift
Reporting Platform
Analytics Platform
Data Integration Platform
Real-Time Dashboards
30. 50+ stores serving needs of
Hispanic community in Southern
CA and AZ
Challenges Faced:
How to integrate all of the company's applications
and keep its 24/7 operations running smoothly.
Why SnapLogic?
Speed: Northgate was able to achieve a significant
improvement in sales order throughput and
accuracy.
Productivity: In a matter of months, the company
went from 30 pipelines to 64, all while adding 14
additional stores
Connectivity: Modernized and connected the ERP
and WMS systems in only 12 weeks
31. The Impact of SMACT on the
Data Management Stack?
• Presents an opportunity to re-think your approach to data and
application integration
• Means you need to accept / embrace self-service data access
(citizen integrators)
• Data, App, API integration converging quickly
• Innovation is back in the DI market (so don’t settle for same old,
same old – SO SO Integration)
This session will introduce the concept of the "Integrator's Dilemma" and review some of the challenges faced by traditional data and application integration technologies when it comes to keeping up with the new enterprise data, application and API connectivity and management requirements. We'll review the landscape and share examples of the steps more and more IT organizations are taking to improve business alignment through faster access to trusted data.
This need for both cloud integration power and the need for speed and simplicity has lead to an “Integrator’s Dilemma” for many of the IT organizations we talk to today.
The good news is there’s now an awareness that without the right approach to integration, the promise of SaaS and cloud computing will not be met.
The bad news is that their existing legacy middleware technologies were conceptualized and built before the SMAC stack became an enterprise reality – Social, Mobile, Analytics/Big Data and Cloud Computing.
When it comes to Legacy Enterprise App Integration tools, “buses don’t fly.” On-premises enterprise service bus (ESB) are known to be brittle and code-intensive. They were designed to speak XML and SOAP, not the more modern web protocols like JSON and REST. They weren’t designed to run at cloud speed.
When it comes to Legacy Extract, Transformation and Loading tools, they’re great for large batches of structured data – rows and columns. This works well for initial migrations and periodic data loading requirements, but customers inevitably want both real-time and batch application and data integration capabilities.
According to a recent Gartner report, (read the quote on the slide) - “Organizations are increasingly turning to iPaaS offerings because of their close affinity with SaaS and the anticipated greater ease of use, lower costs and faster time-to-integration than traditional integration platforms”.
http://www.snaplogic.com/dilemma
.
http://datascience.berkeley.edu/what-is-big-data/
Use Case #1: Customer Analytics
To improve customer conversion rates, personalize campaigns, reduce customer churn, and more, marketers need to analyze data from lots of customer interaction points like mobile, social media, stores, and e-commerce sites. With big data analytics from Datameer, you can aggregate and analyze all of this data at once, yielding insights you never had before – for example, who are your high-value customers, what motivates them to buy, how they behave, and how to best reach them.
Use Case #2: Operational Analytics: Understanding Machines, Devices and Human Interactions
Manufacturing, operations, service and product executives face intense pressure to optimize asset utilization, budgets, performance and service quality. IT executives can help by using big data analytics to unlock insights buried in log, sensor and machine data and structured CRM, ERP, other data. Datameer customers are detecting outliers; running time series and root cause analyses; and parsing, transforming and visualizing data for insights that improve asset management decisions and ROI.
Use Case #3: Fraud and Compliance
Data-driven insights can help you uncover what’s hidden and suspicious – and in time to mitigate risks. With big data analytics, you can combine, integrate and analyze all of your risk-related data at once to generate the insights and metrics needed to detect fraud and compliance issues. For example, you can perform time series analysis, data profiling and accuracy calculations, data standardization, root cause analysis, breach detection, and fraud scoring, identity verifications, risk profiles, and data visualizations.
Use Case #4: Data-Driven Products and Services
Savvy companies are leveraging big data analytics to create new, data-driven products and services that differentiate their business and drive revenue. For example, a media company is using Datameer to provide brands and advertisers with reports about how customers behave using mobile apps so they can optimize ads and boost responses. And a leading provider of enterprise cloud applications uses Datameer to provide customers with reports on how end users are actually using their software.
Use Case #5: EDW Optimization
EDWs are critical business and IT resources today – but as the size and complexity of the data to be analyzed increases, you’ll eventually hit the limits of traditional data warehouses. By offloading the most challenging data management and analytics activities to big data analytics solutions like Datameer that runs on Hadoop, you can cost effectively scale to any volume of data and store and analyze any and all data types together – both structured and unstructured.
http://www.datanami.com/2014/10/06/top-five-big-data-analytics-use-case-yield-high-returns/
Fix into colums
“We’ve been able to eliminate some of the rigidity and time-consuming tasks related to traditional integration patterns.”
– Jim Teal, Information Architect, iRobot
Architecture diagram - SnapLogic pulls data from all of the services, CSVs, APIs, databases and pushes it into all of the BI tools
Good Data is the primary BI tool
Easy to use web interface (minimal training) easy to maintain
Tableau for deeper business analysts – internal case studies
Real-time reporting – Gecko Board (API calls for quick dashboards)
Everything goes into Redshift
Good Data is subset, more highly aggregated
Use Python to do the data science on Redshift
Predictive, product recommendation
“I would rather work with the business than with the hardware!”
– David Glueck, Sr. Director of Data Science and Engineering, Bonobos
. "Future development is going to be faster and cheaper because we can reuse the Snaps," Lewis said. The IT staff is small -- only one person is fully focused on integration -- and that works because the SnapLogic support staff fills in the gaps,
“Working with SnapLogic has enabled us to make better use of both our on-premise and cloud applications and drive greater benefits from our overall investments.”
– Harrison Lewis, CIO, Northgate Markets