M.C. Shivas's presentation was part of a panel discussion on Stream Processing Systems on January 20th, 2016 led by Ben Lorica (O'Reilly Media) with panelists: Jay Kreps (Confluent), Karthik Ramasamy (Twitter), Nikita Shamgunov (MemSQL), Ram Sriharsha (Hortonworks)

1. © 2014 MapR Technologies 1© 2014 MapR Technologies

2. © 2014 MapR Technologies 2
American express American express
MapR at the Core of Big Data
700+ CustomersCloud Leadership
American
Express
Wells Fargo
UnitedHealth
Group

3. © 2015 MapR Technologies 3
The Growing Need For Platform Thinking
As it
happens
Near real
time
Micro batch
Batch
Developer Production Aspirational Goal
Batch
Micro
batch
New System
of Record
AND
Operational
Intelligence
Operational
analytics
Scale
Reliability
Trust
Latency
Multi-tenancy
Streams

4. © 2015 MapR Technologies 4
Their Strategy: Customer Do-It-Yourself
APACHE HADOOP
Security
YARN
Spark
Streaming
Storm
StreamingNoSQL &
Search
Juju
Provisioning
&
coordination
Savanna
h
ML,
Graph
Mahout
MLLib
GraphX
EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS
Workflow
& Data
Governanc
e
Pig
Cascadin
g
Spark
Batch
MapRedu
ce v1 & v2
Tez
HBase
Solr
Hive
Impala
Spark
SQL
Drill
SQL
Sentry Oozie
ZooKeepe
r
Sqoop
Flume
Data
Integration
& Access
HttpFS
Hue
Search
cluster
HA
duplicate
Kafka
cluster
NAS
storage
NoSQL
cluster
Not pre-integrated
More expensive
Latencies everywhere
Unreliability grows

5. © 2015 MapR Technologies 5
BIG STORAGE
What Makes A Big Data Platform?
Hadoop SQL Search Messaging
BIG PROCESSING
NoSQL

6. © 2015 MapR Technologies 6
MapR CONVERGED DATA PLATFORM
APACHE HADOOP
Security
YARN
Spark
Streaming
Storm
StreamingNoSQL &
Search
Provisioning
&
coordination
Sahara
ML, Graph
Mahout
MLLib
GraphX
EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS
Workflow
& Data
Governance
Pig
Spark
Batch
MapReduce v1
& v2
HBase
Solr
Hive
Impala
Spark SQL
Drill
SQL
Sentry Oozie ZooKeeperSqoop
Flume
Data Integration
& Access
HttpFS
Hue
HDFS APIs
MapR STORAGE PLATFORM
Unlimited Scale
Multi
Data-center
Global Namespace
Data
Protection
Unified Security

7. © 2015 MapR Technologies 7
MapR CONVERGED DATA PLATFORM
MapR-
DB
JSON
Search
RealTime
Streams
Apache Drill : ANSI SQL as the Unified Access Layer
File
System
POSIX
APACHE HADOOP
Security
YARN
Spark Streaming
Storm
StreamingNoSQL & Search
Provisioning
&
coordination
Sahara
ML, Graph
Mahout
MLLib
GraphX
EXECUTION ENGINES
DATA GOVERNANCE AND OPERATIONS
Workflow
& Data
Governance
Pig
Spark
Batch
MapReduce v1 &
v2
HBase
Solr
Hive
Impala
Spark SQL
Drill
SQL
Sentry Oozie ZooKeeperSqoop
Flume
Data Integration
& Access
HttpFS
Hue
HDFS APIs
MapR STORAGE PLATFORM
Unlimited Scale
Multi
Data-center
Global Namespace
Data
Protection
Unified Security

8. © 2015 MapR Technologies 8
MapRCONVERGEDDATAPLATFORM
MapR
DB
Search
MapR
Streams
APACHE HADOOP AND OSS ECOSYSTEM
Security
YARN
Spark
Streaming
Storm
StreamingNoSQL &
Search
Provisioning
&
coordination
Sahara
ML, Graph
Mahout
MLLib
GraphX
EXECUTION ENGINES
DATA GOVERNANCE AND OPERATIONS
Workflow
& Data
Governance
Pig
Spark
Batch
MapReduce
v1 & v2 HBase
Solr
Hive
Impala
Spark SQL
Drill
SQL
Sentry Oozie ZooKeeperSqoop
Flume
Data
Integration
& Access
HttpFS
Hue
File
System
POSIX
Apache Drill: Unified Access Layer
MapR STORAGE PLATFORM
Unlimited Scale
Multi
Data-center
Global Namespace
Data
Protection
Unified Security
DATABASES&APPLICATIONS
EMail
HPC
Document
Mgmt
Custom
Apps
Custom
Apps

9. © 2015 MapR Technologies 9
MapR Converged Data Platform
Open Source Engines & Tools Commercial Engines & Applications
Utility-Grade Platform Services
DataProcessing
Enterprise Storage
MapR-FS MapR-DB MapR Streams
Database Event Streaming
Global Namespace High Availability Data Protection Self-healing Unified Security Real-time Multi-tenancy
Search &
Others
Cloud &
Managed
Services
Custom
Apps
UnifiedManagementandMonitoring

10. © 2015 MapR Technologies 10
MapR Streams – Publish/Subscribe
Listeners
Publish billions of messages per sec to a topic in a stream.
Reliable delivery to all consumers. Immediately.
JSON data format, direct data access from analytics frameworks
L
Simple, standard API (Kafka).
Topi
c
Stream
Tie together geo-dispersed clusters. Worldwide.
Topic
Producers
Consumers

11. © 2014 MapR Technologies
Records
JSON messages allow SQL queries by Apache Drill
Pub1
{
“event” : “CLICK”,
“url” : “/about/”,
“user” : “Will”,
“ip” : “192.168.1.1”,
“loggedin” : true
}
/stream/web:events
[
{“seq001”: { … } },
{“seq002”: { … } },
…
]
SELECT user, ip, url FROM marlin.`/stream/web:events` WHERE event=CLICK
user ip url
--------------------------------------------------------------------------------------------
Will 192.168.1.1 /about/
Neeraja 192.168.1.2 /logout/
Mitesh 192.168.1.3 /prod01/buy

12. © 2015 MapR Technologies
MapR Streams: Global Messaging Fabric
...Edge
Aggregation
...
Sensors Apps Analytics
● Arbitrary interconnection of thousands of clusters
● Seamless disconnection/reconnection
● Globally synchronized sequence numbers &
cursors: Listener & Producer failover
● Global applications
● Data filtering/aggregation at the edge
● Distributed analytics
NYC LONDON
CHI TOK