Zeppelin at Twitter - Prasad Wagle, Technical Lead in the Data Platform team - Twitter Prasad will talk about how Zeppelin is used at Twitter, the development work they did before release and the features and enhancements they are working on to increase adoption.

1. Zeppelin at Twitter
Prasad Wagle
Technical Lead, Data Platform
twitter.com/prasadwagle
May 18, 2016
Apache Zeppelin Meetup
MapR, San Jose, CA

2. Twitter Data Pipeline Overview
Production
systems
Presto
Vertica
MySQL
Scalding
Spark
Custom Dashboards
Tableau
Zeppelin
Command line tools
HDFS
Analytics Engines
Analytics Front-ends

3. One company-wide server
530 notes
2300 paragraphs
1000 Vertica, 800 Presto, 200 MySQL
250 Markdown
50 Hive, Scalding, Spark
550 users
Zeppelin Usage Metrics

4. Field of Dreams

5. Report Creators
Product managers (dashboards, product analytics)
Data scientists
Sales analysts
Engineers and SREs
Report Viewers
Anyone in the company
Zeppelin Users

6. My mind is absolutely blown away by the ease of use, speed, and power of
Zeppelin. I've been wanting a tool like this at Twitter my entire time
working here.
started playing with @ApacheZeppelin. amazingly addictive!
Thanks for all the updates to Zeppelin - Fabric has fallen in love with it fast
(and we're even using it for daily tracking of our OKRs amongst all the
other metrics)
Zeppelin Testimonials

7. Very easy to create and share reports
Web based
Works seamlessly with analytics engines
JDBC
Non-JDBC - Scalding, Spark
Flexible
Open source (easy to add features)
Reasons for adoption

8. Drag and drop report builder
Can create complex queries without SQL knowledge (e.g. Top N)
Polished UI (for executives)
Filters and other transformations work on extracts
no new database queries (fast)
Row level permissions (for sales reports)
Tableau

9. Areas:
Security
Stability
Operations
Interpreter
Work Done Before Production

10. Authentication
Integrated with Twitter’s homegrown single sign-on system
SSL
Integrated with Twitter’s homegrown key distribution system
Notebook authorization
Data source authorization
Work Done (Security)

11. Websocket deadlock issue with Jetty 8
reduce communication
remove synchronized block (risky, will move to Jetty 9)
Monitoring
Standby server
Backups
Work Done (Stability, Operations)

12. Scalding
Hive
Create JDBC connection for every query to avoid Vertica closed
connection issue
Work Done (Interpreter)

13. Notebook authorization
Data source authorization
Run scheduled notes with a user
Scalding interpreter
Reduce websocket communication
Paragraph footer
Row level permissions
Work Contributed to Apache Project

14. Stability, Scalability
Jetty9
Interpreters
Scalding and Spark
Multiuser scalability, authentication, integration with Twitter
sources
R (authentication)
Use JDBC interpreter instead of Hive
Future / Work in Progress

15. Features and UX
Notebook organization (folders)
Email reports and alerts
Row level permissions like tableau
Operations
Monitoring (end-to-end query)
Admin (view/stop running jobs, resource usage)
Failover
Continuous Integration
Future / Work in Progress