This talk will be a step by step walkthrough of a developer troubleshooting a real performance issue we had at MyDrive, from the very first steps diagnosing the symptoms, through looking at metric charts down to CQL queries, the Ruby CQL driver, and Ruby code profiling.

1. Case Study: Troubleshooting Production Issues as a Developer

2. © 2015. All Rights Reserved.
1 Introductions
2 The problem
3 Show me the code!
4 Conclusions
5 Q & A

3. Carlos Alonso - About me
3© 2015. All Rights Reserved.
• Spanish Londoner
• MSc at Salamanca University, Spain
• Software Engineer @ MyDrive Solutions
• Enjoying Cassandra since 2014
• Cassandra Certified Developer
• DataStax Cassandra MVP 2015
• @calonso / http://mrcalonso.com

4. MyDrive Solutions - About
4© 2015. All Rights Reserved.
• World leading driver profiling company.
• Using technology and data to understand
how to improve driving behaviour.
• Enjoying Cassandra since 2012
• Recently acquired by The Generali Group.
• @_MyDrive
• http://www.mydrivesolutions.com
• We are hiring!

5. @calonso @_MyDrive© 2015. All Rights Reserved. 5
Troubleshooting performance issues in production
as developer

6. @calonso @_MyDrive© 2015. All Rights Reserved. 6
trips.csv
driver_id start_time end_time start_location end_location
123456 2011-04-18
12:23:36 +0100
2011-04-18
14:11:22 +0100
-11.06238284,
-126.71490131
86.02409727,
72.13493018
…
import.rb
The program

7. @calonso @_MyDrive© 2015. All Rights Reserved. 7
CREATE KEYSPACE drivers WITH replication = {
'class': 'SimpleStrategy',
'replication_factor': '2'
};
• 2 environments
• 3 nodes clusters
• Cassandra 1.2.9
The Setup

8. @calonso @_MyDrive© 2015. All Rights Reserved. 8
CREATE TABLE trips (
driver_id int,
end_time timestamp,
end_location text,
start_location text,
start_time timestamp,
PRIMARY KEY (driver_id, end_time)
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};

9. @calonso @_MyDrive© 2015. All Rights Reserved. 9
Tools
Ruby-Prof https://github.com/ruby-prof/ruby-prof
Cassanity https://github.com/jnunemaker/cassanity

11. @calonso @_MyDrive
Conclusions
• Measure everything
• Metrics and monitoring let us know we were facing an
unexpected performance issue
• Keep calm, read to the end
• On first profiling report I could have spotted that the issue was
on Cluster#connect but due to my eagerness to find a fix I
made a wrong assumption that meant more time (~2 days!).
© 2015. All Rights Reserved. 11

12. @calonso @_MyDrive
Thank the community
• We have great tools such as:
• Cassandra: Apache and DataStax
• Statsd, Graphite.
• RubyProf: All members of RubyProf Organisation
• Special thanks to Patrick McFadin.
© 2015. All Rights Reserved. 12

13. Thank you