Use Spark, Apache Flume and ElasticSearch-Kibana to unleash the power of your Code Review and Continuous Delivery Pipeline logs. Jenkins and Gerrit Code Review are the sources of your Analytics KPI dashboards.

1. #JenkinsWorld
Speed up Continuous Delivery
with BigData Analytics
Luca Milanesio

2. @gitenterprise @gerritreview #JenkinsWorld
• Founded in the UK
• HQ in London, Offices in the UK and Italy
• Committed to OpenSource
About GerritForge

3. @gitenterprise @gerritreview #JenkinsWorld
• Co-founder and Director of
GerritForge
• over 20 years in Agile
Development and ALM
• Contributor to many OpenSource
projects
• Mind and hands of @gitenterprise
and @gerritreview Twitter feeds
About Luca Milanesio

4. @gitenterprise @gerritreview #JenkinsWorld
NOT a lecture on Continuous Delivery
NOT a lecture on Code Review
NOT a lecture on BigData
What NOT to expect?

5. @gitenterprise @gerritreview #JenkinsWorld
SO WHAT?
#!@$!

6. @gitenterprise @gerritreview #JenkinsWorld
SHOW
a Continuous Delivery scenario

7. @gitenterprise @gerritreview #JenkinsWorld
LEARN
together from its problems

8. @gitenterprise @gerritreview #JenkinsWorld
EXPERIMENT
a new approach using BigData

9. @gitenterprise @gerritreview #JenkinsWorld
DISCUSS
results together and iterate

10. @gitenterprise @gerritreview #JenkinsWorld
Continuous Delivery Pipeline
Push Build
Unit Tests
Push Build
Unit Tests
Automated
Tests
Push Build
Unit Tests
Automated
Tests
Beta Users
Tests
User's feedback Publish to all Users
Failed
Failed

11. @gitenterprise @gerritreview #JenkinsWorld
Code Review & CD Validation
A1
refs/for/master
master
C2 C2*
A1 C2 (local)
master
C2*
push
Verified: -1
(build failed)
push
Verified: +1
(build OK)
Reviewed: +2
CI Build: OK
C2*
CI Build: OK
amend

12. @gitenterprise @gerritreview #JenkinsWorld
Git/Gerrit
(Code Review)
Components
Jenkins
(Continuous
Integration & Delivery)

13. @gitenterprise @gerritreview #JenkinsWorld
DEMO
See the pipeline in action

14. @gitenterprise @gerritreview #JenkinsWorld
We use micro-services
–100+ Components
–200+ Repositories
–300+ People and 100,000+ of review comments
–Cross-components [implicit] dependencies
–Lots of build chains
–10,000+ of builds
Now feel the pain :-(

15. @gitenterprise @gerritreview #JenkinsWorld
Problem #1 : Tons of logs slows us down
"Build was so fast a week ago … why is it getting so
slow now?"
100Ks of logs are tough for Jenkins

16. @gitenterprise @gerritreview #JenkinsWorld
"It was working until 5 mins ago … then someone
changed something"
 Rings any bell?
Problem #2 : Ripple effect

17. @gitenterprise @gerritreview #JenkinsWorld
In order to deliver Feature ABC
I NEED
Ver. N of Service-A with
Ver. M of Service-B and
Ver. K of Service C
Problem #3 : Dependencies

18. @gitenterprise @gerritreview #JenkinsWorld
Release small changes is GOOD
Release often every change is GOOD
BUT
How long it takes the "Big Picture" and End-to-End?
Where is my bottleneck?
Problem #4 : Cycle time

19. @gitenterprise @gerritreview #JenkinsWorld
Problems #1 + #2 + #3 + #4 =
DevOps meltdown 
Problem #6 : all together

20. @gitenterprise @gerritreview #JenkinsWorld
• Collect all review events
• Collect all logs
• Channel them to a central store
• Crunch and Crunch continuously
• Never delete
• Process, inspect and learn
BigData to the rescue

21. @gitenterprise @gerritreview #JenkinsWorld
Continuous Delivery – Analytics Pipeline
Data
Aggregation
Jobs

22. @gitenterprise @gerritreview #JenkinsWorld
Continuous Delivery – Analytics Dimensions
Projects Commits
People Reviews System Metrics

23. @gitenterprise @gerritreview #JenkinsWorld
Continuous Delivery – Tools Dimensions
SCM / Code Review
CI / CD

24. @gitenterprise @gerritreview #JenkinsWorld
DEMO
Continuous Delivery Analytics

25. @gitenterprise @gerritreview #JenkinsWorld
• Collect more data (System, JVM, Review scores)
• Real-time statistics
• Events correlation
• Feedback loop to Code-Review
• Production data harvest
• More?
Future directions

26. @gitenterprise @gerritreview #JenkinsWorld
Want to know more?
www.gerritforge.com/contact

27. @gitenterprise @gerritreview #JenkinsWorld
• GerritForge's blog
www.gitenterprise.me
• Gerrit Code Review
www.gerritcodereview.com
• Jenkins
www.jenkins.io
• RabbitMQ
rabbitmq.com
Resources
• Apache Flume
flume.apache.org
• Hadoop HDFS
hadoop.apache.org
• Apache Spark
spark.apache.org
• Kibana
github.com/elastic/kibana