SlideShare ist ein Scribd-Unternehmen logo
1 von 28
Downloaden Sie, um offline zu lesen
Measure everything

                             Ruby Underground, August 2012
                                       @arikfr




Tuesday, February 12, 13
Today’s plan

                     • Why measure
                     • How to measure
                     • Graphite / StatsD
                     • Using Graphite/StatsD with Ruby and Rails

Tuesday, February 12, 13
Questions to answer
                     • How fast is my system?
                     • Is it faster than last month?
                     • Did our last deploy affect database
                           performance?
                     • How much time do we spend calling
                           external web services?


Tuesday, February 12, 13
More questions

                     • How many errors do we have a day?
                     • How many failed logins?
                     • How many successful logins?
                     • How many orders without house number?

Tuesday, February 12, 13
And more questions!
                     • How many orders did we have today?
                     • How many orders did we have today from
                           Android version 2.056?
                     • How many rejected orders did we have?
                     • How many rejected orders due to lack of
                           coverage vs. lack of taxis?


Tuesday, February 12, 13
To answer all of this,
                            you need a way to
                             track different
                               numbers.

Tuesday, February 12, 13
So now we know the
                                   why.
                                But how...?


Tuesday, February 12, 13
Tuesday, February 12, 13
But that’s not enough

                     • Not real time enough
                     • Hard to control what’s being collected
                     • Pricey for big deployments


Tuesday, February 12, 13
The Alternative

                     • Graphite (Whisper, Carbon, Graphite Web)
                     • StatsD
                     • CollectD
                     • (there are other options -- OpenTSDB,
                           Liberato, home grown)



Tuesday, February 12, 13
Benefits
                     • Easy to install
                     • Highly scalable
                     • Practically zero cost to measure anything:
                      • efficient storage
                      • UDP packets to send data in
                     • Ecosystem
Tuesday, February 12, 13
Whisper
                     • Default settings:
                      • 6 hours of 10 second data
                      • 1 week of 1 minute data
                      • 5 years of 10 minute data
                     • That’s amounts to ~3.2MB per metric.
                     • Configurable.
Tuesday, February 12, 13
Types of data

                     • Counters - number of orders per sec
                     • Gauges - total orders today
                     • Timers - time to make an order
                      • (with additional values, such as: count,
                           mean, 90th percentile, max, min, etc)



Tuesday, February 12, 13
Sending a number to
                                Graphite

                     • metric number timestamp
                     • example.ruby.under_ground 20 1346075634
                     • echo "example.ruby_underground 20 `date
                           +%s`" | nc graphite.yourcorp.com 2003




Tuesday, February 12, 13
Sending a number to
                                 StatsD




                               ./statsd-client.sh 'my_metric:100|g'
Tuesday, February 12, 13
Naming Convention
                     • Whatever makes sense to you, just
                           remember that it’s a tree.
                     • We use:
                      • {env}.{metric}.{region}.{hostname}
                     • You can use globs, when querying:
                      • app.orders.daily.completed.israel.*
                      • app.orders.daily.completed.*.*
Tuesday, February 12, 13
Ruby Examples



Tuesday, February 12, 13
$ gem install statsd-ruby
                           (https://github.com/reinh/statsd)




Tuesday, February 12, 13
Track Deployments




Tuesday, February 12, 13
Measure controllers




Tuesday, February 12, 13
Measure Specific stuff




Tuesday, February 12, 13
Getting Data out



Tuesday, February 12, 13
Graphite web




Tuesday, February 12, 13
Tasseo




Tuesday, February 12, 13
Gdash




Tuesday, February 12, 13
Graphene




Tuesday, February 12, 13
Down sides
                     • Hard to track user analytics
                      • You can tell how many orders were done
                             today
                           • You can’t tell (easily) how many unique
                             users did those orders
                     • The tree of metrics is sometimes annoying

Tuesday, February 12, 13
Questions?



Tuesday, February 12, 13

Weitere ähnliche Inhalte

Ähnlich wie Measure Everything

Big Data - architectural concerns for the new age
Big Data - architectural concerns for the new ageBig Data - architectural concerns for the new age
Big Data - architectural concerns for the new ageDebasish Ghosh
 
GitHub Notable OSS Project
GitHub  Notable OSS ProjectGitHub  Notable OSS Project
GitHub Notable OSS Projectroumia
 
Dan node meetup_socket_talk
Dan node meetup_socket_talkDan node meetup_socket_talk
Dan node meetup_socket_talkIshi von Meier
 
Dragging old data forward: finding yourself an RDA Helper
Dragging old data forward:  finding yourself an RDA HelperDragging old data forward:  finding yourself an RDA Helper
Dragging old data forward: finding yourself an RDA HelperTerry Reese
 
Why Speed Matters
Why Speed MattersWhy Speed Matters
Why Speed Mattersdarinrs
 
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services 2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services PHP Conference Argentina
 
Escalando una PHP App con DB sharding - PHP Conference
Escalando una PHP App con DB sharding - PHP ConferenceEscalando una PHP App con DB sharding - PHP Conference
Escalando una PHP App con DB sharding - PHP ConferenceMatias Paterlini
 
Statsd backends presentation
Statsd backends presentationStatsd backends presentation
Statsd backends presentationDraco2002
 
Fast Mobile UIs
Fast Mobile UIsFast Mobile UIs
Fast Mobile UIsWooga
 
Lisa12 methodologies
Lisa12 methodologiesLisa12 methodologies
Lisa12 methodologiesBrendan Gregg
 
Infrastructure Migration
Infrastructure MigrationInfrastructure Migration
Infrastructure MigrationMatt Simmons
 
Location Location Location
Location Location LocationLocation Location Location
Location Location LocationGavin Heavyside
 
Performance for Product Developers
Performance for Product DevelopersPerformance for Product Developers
Performance for Product DevelopersMatthew Wilkes
 
Torchbox University Accessibility
Torchbox University AccessibilityTorchbox University Accessibility
Torchbox University Accessibilitymynameismartin
 

Ähnlich wie Measure Everything (20)

Qcon talk
Qcon talkQcon talk
Qcon talk
 
Big Data - architectural concerns for the new age
Big Data - architectural concerns for the new ageBig Data - architectural concerns for the new age
Big Data - architectural concerns for the new age
 
GitHub Notable OSS Project
GitHub  Notable OSS ProjectGitHub  Notable OSS Project
GitHub Notable OSS Project
 
Dan node meetup_socket_talk
Dan node meetup_socket_talkDan node meetup_socket_talk
Dan node meetup_socket_talk
 
Dragging old data forward: finding yourself an RDA Helper
Dragging old data forward:  finding yourself an RDA HelperDragging old data forward:  finding yourself an RDA Helper
Dragging old data forward: finding yourself an RDA Helper
 
Why Speed Matters
Why Speed MattersWhy Speed Matters
Why Speed Matters
 
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services 2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
 
Escalando una PHP App con DB sharding - PHP Conference
Escalando una PHP App con DB sharding - PHP ConferenceEscalando una PHP App con DB sharding - PHP Conference
Escalando una PHP App con DB sharding - PHP Conference
 
Cassandra at scale
Cassandra at scaleCassandra at scale
Cassandra at scale
 
Statsd backends presentation
Statsd backends presentationStatsd backends presentation
Statsd backends presentation
 
Hadoop & Hep
Hadoop & HepHadoop & Hep
Hadoop & Hep
 
Drupal for enterprise
Drupal for enterpriseDrupal for enterprise
Drupal for enterprise
 
Fast Mobile UIs
Fast Mobile UIsFast Mobile UIs
Fast Mobile UIs
 
Lisa12 methodologies
Lisa12 methodologiesLisa12 methodologies
Lisa12 methodologies
 
[Phind] Miracle
[Phind] Miracle[Phind] Miracle
[Phind] Miracle
 
Infrastructure Migration
Infrastructure MigrationInfrastructure Migration
Infrastructure Migration
 
Location Location Location
Location Location LocationLocation Location Location
Location Location Location
 
Ruby on hadoop
Ruby on hadoopRuby on hadoop
Ruby on hadoop
 
Performance for Product Developers
Performance for Product DevelopersPerformance for Product Developers
Performance for Product Developers
 
Torchbox University Accessibility
Torchbox University AccessibilityTorchbox University Accessibility
Torchbox University Accessibility
 

Kürzlich hochgeladen

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Alexander Turgeon
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 

Kürzlich hochgeladen (20)

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 

Measure Everything

  • 1. Measure everything Ruby Underground, August 2012 @arikfr Tuesday, February 12, 13
  • 2. Today’s plan • Why measure • How to measure • Graphite / StatsD • Using Graphite/StatsD with Ruby and Rails Tuesday, February 12, 13
  • 3. Questions to answer • How fast is my system? • Is it faster than last month? • Did our last deploy affect database performance? • How much time do we spend calling external web services? Tuesday, February 12, 13
  • 4. More questions • How many errors do we have a day? • How many failed logins? • How many successful logins? • How many orders without house number? Tuesday, February 12, 13
  • 5. And more questions! • How many orders did we have today? • How many orders did we have today from Android version 2.056? • How many rejected orders did we have? • How many rejected orders due to lack of coverage vs. lack of taxis? Tuesday, February 12, 13
  • 6. To answer all of this, you need a way to track different numbers. Tuesday, February 12, 13
  • 7. So now we know the why. But how...? Tuesday, February 12, 13
  • 9. But that’s not enough • Not real time enough • Hard to control what’s being collected • Pricey for big deployments Tuesday, February 12, 13
  • 10. The Alternative • Graphite (Whisper, Carbon, Graphite Web) • StatsD • CollectD • (there are other options -- OpenTSDB, Liberato, home grown) Tuesday, February 12, 13
  • 11. Benefits • Easy to install • Highly scalable • Practically zero cost to measure anything: • efficient storage • UDP packets to send data in • Ecosystem Tuesday, February 12, 13
  • 12. Whisper • Default settings: • 6 hours of 10 second data • 1 week of 1 minute data • 5 years of 10 minute data • That’s amounts to ~3.2MB per metric. • Configurable. Tuesday, February 12, 13
  • 13. Types of data • Counters - number of orders per sec • Gauges - total orders today • Timers - time to make an order • (with additional values, such as: count, mean, 90th percentile, max, min, etc) Tuesday, February 12, 13
  • 14. Sending a number to Graphite • metric number timestamp • example.ruby.under_ground 20 1346075634 • echo "example.ruby_underground 20 `date +%s`" | nc graphite.yourcorp.com 2003 Tuesday, February 12, 13
  • 15. Sending a number to StatsD ./statsd-client.sh 'my_metric:100|g' Tuesday, February 12, 13
  • 16. Naming Convention • Whatever makes sense to you, just remember that it’s a tree. • We use: • {env}.{metric}.{region}.{hostname} • You can use globs, when querying: • app.orders.daily.completed.israel.* • app.orders.daily.completed.*.* Tuesday, February 12, 13
  • 18. $ gem install statsd-ruby (https://github.com/reinh/statsd) Tuesday, February 12, 13
  • 22. Getting Data out Tuesday, February 12, 13
  • 27. Down sides • Hard to track user analytics • You can tell how many orders were done today • You can’t tell (easily) how many unique users did those orders • The tree of metrics is sometimes annoying Tuesday, February 12, 13