As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture? To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.

1. Building Real-Time Data Pipelines
Through In-Memory Architectures
Ben Lorica, Chief Data Scientist, O'Reilly Media
@bigdata
Eric Frenkiel, CEO & Co-Founder, MemSQL
@ericfrenkiel

2. What’s In Store
Why In-Memory for Real Time
Using an In-Memory Database with Spark and Kafka
Real-Time Use Cases and Demonstrations
About MemSQL

3. Going Real-Time is the Next Phase for Big Data
More
Sensors
More
Interconnectivity
More
User Demand
…and companies are at risk of being left behind

4. Expensive
Not scalable
Batch only
SAN-burdened
1%

5. Success will
be driven by
real-time
analytic
applications

6. What’s In Store
Why In-Memory for Real Time
Using an In-Memory Database with Spark and Kafka
Real-Time Use Cases and Demonstrations
About MemSQL

7. Speed
Serving
Batch Fast Updates
Unified queries, full SQL
Fast Appends
A Fresh Look at Lambda Architectures

8. Comprehensive Architecture
Transactions

9. Comprehensive Architecture
Real Time
Speed/Streaming Layer
Fast Updates
Rowstore
Transactions

10. Comprehensive Architecture
Real Time
Speed/Streaming Layer
Fast Updates
Rowstore
Analytics
Transactions

11. Comprehensive Architecture
Real Time
Speed/Streaming Layer
Fast Updates
Rowstore
Historical
Batch Layer
Fast Appends
Columnstore
Analytics
Transactions

12. Comprehensive Architecture
Real Time
Speed/Streaming Layer
Fast Updates
Rowstore
Historical
Batch Layer
Fast Appends
Columnstore
Analytics
Transactions
Execution engine that spans the data spectrum

13. Comprehensive Architecture
Real Time
Speed/Streaming Layer
Fast Updates
Rowstore
Historical
Batch Layer
Fast Appends
Columnstore
Analytics
Transactions

14. Simplified Lambda Architectures with MemSQL
Layer Traditional Lambda MemSQL Lambda
Batch Hadoop MemSQL Column Store
Speed Storm, Spark Kafka > Spark > MemSQL
Serving Cassandra, HBase MemSQL

15. Designing the Ideal Real-Time Pipeline
Message Queue Transformation Speed/Serving Layer
End-to-End Data Pipeline Under One Second

16.  A high-throughput
distributed messaging
system
 Publish and subscribe to
Kafka “topics”
 Centralized data transport
for the organization
Kafka

17.  In-memory execution
engine
 High level operators for
procedural and
programmatic analytics
 Faster than MapReduce
Spark

18.  In-memory, distributed
database
 Full transactions and
complete durability
 Enable real-time,
performant applications
MemSQL

19. Lambda Applies to Real-Time Data Pipelines
Message
Queue
Batch
Inputs DatabaseTransformation Application

20. Kafka, Spark, and MemSQL Make it Simple
Batch
Inputs Application

21. Put Apache Spark in the fast lane
with MemSQL Streamliner

22.  One click deployment of
integrated Apache Spark
 Put Spark in the Fast Lane
• GUI pipeline setup
• Multiple data pipelines
• Real-time transformation
 Eliminates batch ETL
 Open source on GitHub
Introducing the MemSQL Streamliner

23. Simple Deployment Process
Application

24. Cluster
1. Deploy MemSQL
In-Memory | Distributed | Relational
Application

25. Cluster
2. Deploy Spark
Application

26. Cluster
Kafka Connects to Each Node
Application

27. Streamliner Architecture
First of many integrated Apache Spark solutions
Other
Real-Time Data
Sources Application
Apache Spark
Future Solution
Future Machine
Learning Solution
STREAMLINER

28. Streamliner ETL Detail
Other
Real-Time Data
Sources Application
Apache Spark
Future Solution
Future Machine
Learning Solution
STREAMLINER
Custom
Future Extractor
JSON
Custom
Future Transformer
STREAMLINER
Extract Transform Load

29. Streamliner

30. Extract

31. Transform

32. Load

33. Streamliner: Dynamic Resource Management
Without Streamliner With Streamliner
Pipeline 1
Spark Worker
Pipeline 2
Spark Worker
Executor
(P2 only)
Executor
(P2 only)
Executor
(P1 only)
Executor
(P1 only)
Driver
(P1 only)
Driver
(P2 only)
All Pipelines
Streamliner Driver
…
…
Spark WorkerSpark Worker
Executor
(P1 or P2)
Executor
(P1 or P2)
Executor
(P1 or P2)
Executor
(P1 or P2)

34. What’s In Store
Why In-Memory for Real Time
Using an In-Memory Database with Spark and Kafka
Real-Time Use Cases and Demonstrations
About MemSQL

35. One Architecture
for Many Applications

36. Monitoring real-time Xfinity programming and video health

37.  Collect streaming data at scale
(hundreds of MemSQL
machines)
 Proactively diagnose issues
 Query ad-hoc and in real-time
with full SQL
From 30 minutes to less than 1 second
Real-time
Analytics

38. Real-Time
Trend Analytics

39. Massive Ingest and Concurrent Analytics
 Instant accuracy to the latest repin
 Build real-time analytic applications
Real-time
analytics

41. Real-Time
Segmentation

42. Using Real-Time for Personalization
Ad Servers
EC2
Real-time
analytics
PostgreSQL
Legacy reports
Monitoring S3 (replay)
HDFS
Data Science
Vertica
Operational Data Store (ODS)
Star Schema MictoStrategy
 Reach overlap and ad optimization
 Over 60,000 queries per second
 Millisecond response times

43. MemCity
Capturing data from 1.4 million households
Total AWS hardware costs at $2.35 per hour

44. Subscribing to Kafka
(2015-07-06T16:43:40.33Z, 329280, 23, 60)
0111001010101111101111100000001010
111100001110101100000010010010111…
Publish to Kafka Topic
0111001010101111101111100000001010
111100001110101100000010010010111…
1110010101000101010001010100010111
111010100011110101100011010101000…
0101111000011100101010111110001111
011010111100000000101110101100000…
Event added to message queue

45. Enrich and Transform the Data
Spark polling Kafka for new messages
(2015-07-06T16:43:40.33Z, 329280, 23, 60)
(2015-07-06T16:43:40.33Z, 329280, 94110, 23,
‘kitchen_appliance’, 60)
Deserialization
Enrichment
0111001010101111101111100000001010
111100001110101100000010010010111…

46. Persist and Prepare for Production
RDD.saveToMemSQL()
INSERT INTO memcity_table ...
time house_id zip
device
_id
device_type watts
2015-
07-
06T16:4
3:40.33
Z
329280 94110 23
‘kitchen_app
liance’
60
… … … … … …

47. Go to Production
Compress development
timelines
SELECT ... FROM memcity_table ...

48. Building Real-Time Data
Pipelines and Predictive
Applications

49. Adding Real-Time Scoring to Predictive Applications
Streamliner
Input
User Jar
SAS Generated PMML
Industrial
Equipment
Sensor Data
S1 S2 S3 P1 P2 P3
Scoring Real-Time Data
with Predictive Models
Sensor 1 Predictive Model 1

50. What’s In Store
Why In-Memory for Real Time
Using an In-Memory Database with Spark and Kafka
Real-Time Use Cases and Demonstrations
About MemSQL

51. MemSQL at a Glance
• Enable every company to be a real-time
enterprise
• Founded 2011, based in San Francisco
• Founders are ex-Facebook, SQL Server
engineers
• Deliver a database technology for modern
architecture
Enterprise Focus

52. The Real-Time Database for Transactions and Analytics
In-Memory Distributed Relational
Data CenterSoftware Cloud

53. MemSQL for the Spectrum of Transactions
Each Transaction Paramount Transactional Aggregates Paramount
 Guarantee that every individual
transaction is persisted
 No individual transaction can be lost
• Financial credits and debits
• Inventory movement
• Employee status
 Capture massive event streams for
immediate analysis
 Transaction repetition/redundancy at
the device level
• Event data and clickstreams
• Sensor data, Internet of Things
• Mobile applications
• Real-time streams

54. Gartner Magic Quadrant for ODBMS
Leading Relational
Database in
Visionaries Quadrant

55. Forrester Wave: In-Memory Database Platforms
”“MemSQL Named
Strong Performer

57. GET YOUR FREE COPY:
memsql.com/oreilly