Zoomdata explains how it uses Spark for streaming interactive visualizations on big data.

1. Interactive Visualization of Data powered by
Spark

2. Streaming Data @ Zoomdata
Visualizations react to
new data delivered
Users start,
stop, pause
the stream
Users select a rolling
window or pin a start
time to capture
cumulative metrics

3. Drivers for Streaming Data
Data Freshness Time to Analytic Business Context

4. Challenges
● Time
● Frequency
● Retention
● Synchronization
● Order
● Updates

5. Addressing the Problem @ Zoomdata
Historical Revised
Receive Data JMS Kafka
Manipulate Stream Single JVM in Memory Spark Streaming
Hold Data in Buffer MongoDB Pluggable
Interact with Data Custom Code Pluggable

6. Technology Cast
● The Stream - Kafka, Kinesis, JMS
● Processing Fabric - Spark Streaming
● Landing Area - MemSQL, Solr, Kudu, Others

7. How it looks

8. With the rest of the app

9. Scale Out

10. Benefits
● Contextual Expressiveness with Streaming Data
● Independent scalability (scale-up, scale-around)
● Expressiveness powered by Spark -- using
Windowing (dataframe API with stream)

11. Side Benefits
● Separation of concerns
● Disaster Recovery, COOP, other Data management
concerns
● Restatements
● Options!

12. Demo
● Twitter Producer
● Spark Streaming
● MemSQL & Solr Sinks

13. Future Work
● Cross Stream Synchronization & Fusion
● On-demand scale out and resource management via
Mesos
● Schema Evolution
● Storage Tiering

14. Thanks
For more information contact:
ruhollah@zoomdata.com
quan@zoomdata.com