SlideShare ist ein Scribd-Unternehmen logo
1 von 14
HOW TO MANAGE 1 BILLION+ INSERTS A DAY INTO A MYSQL RESOURCE
                                                        Sam Lambert, October, 2011




    PHP & Continuous Data Processing, PHPNW 2011
1
SMITH ELECTRIC VEHICLES & TELEMETRY

    • Worlds largest manufacturer of Commercial, all-electric vehicles
    • Smith Link – on-board vehicle telematics system, capturing over 2500 data
      points each second on the vehicle and broadcasting them over mobile network
    • ~400 telemetry enabled vehicles on the road
    • Worlds largest telemetry project outside of F1
    • Number of inserts per day increases by around 250,000,000 per month
    • Record on truck all the way to our web interface in a maximum of 30 seconds




                         How to manage 1 billion + inserts a day into a MySQL resource
2
SYSTEM ARCHITECTURE




    How to manage 1 billion + inserts a day into a MySQL resource
3
ORIGINAL PROBLEMS

Unable to insert data quick enough

Problems with concurrently processing data

Data processing scripts taking an unacceptable amount of time

No obvious path for scaling


                     How to manage 1 billion + inserts a day into a MySQL resource
4
INNODB

Huge scaling potential

Transactional

Tuneable

Well supported/Strong community


                     How to manage 1 billion + inserts a day into a MySQL resource
5
MEASURE

Measure, measure, measure

Always measure as many metrics as possible

De ne a scope

Decide on metrics that can best determine performance improvements


                    How to manage 1 billion + inserts a day into a MySQL resource
6
TOOLS
Sysbench

Cacti (mysql-cacti-templates)

Iostat

Iotop

mysqltuner.pl
                     How to manage 1 billion + inserts a day into a MySQL resource
7
RESOURCES



Large amount of RAM (all machines 100GB+) Our latest server roll outs
have minimum of 144GB

innodb_buffer_pool_size=???




                     How to manage 1 billion + inserts a day into a MySQL resource
8
STRIKING A BALANCE
Maximum safe RAM utilisation

Make the sure buffer pool never swaps




                     How to manage 1 billion + inserts a day into a MySQL resource
9
LOAD DATA INFILE
Our application pulls messages from the queue and creates les for
importing

Have a well optimised perl application that loads the data into MySQL

Inserting data in a batch gave great improvements




                     How to manage 1 billion + inserts a day into a MySQL resource
10
OTHER SETTINGS
innodb_ le_per_table=1

innodb_ ush_log_at_trx_commit=2

innodb_ ush_method=O_DIRECT

innodb_log_ le_size=1G



                   How to manage 1 billion + inserts a day into a MySQL resource
11
SAN
Open-iSCSI settings:

node.session.cmds_max

node.session.queue_depth




                       How to manage 1 billion + inserts a day into a MySQL resource
12
MESSAGE QUEUE
Makes it much easier to plan downtime

Can comfortably tweak the live system

Can pull data in different directions for testing etc

Fast way to receive data



                       How to manage 1 billion + inserts a day into a MySQL resource
13
SUMMARY

             www.samlambert.com

                     @isamlambert



                        Questions?



     How to manage 1 billion + inserts a day into a MySQL resource
14

Weitere ähnliche Inhalte

Kürzlich hochgeladen

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Kürzlich hochgeladen (20)

[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 

Empfohlen

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 

Empfohlen (20)

AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk

  • 1. HOW TO MANAGE 1 BILLION+ INSERTS A DAY INTO A MYSQL RESOURCE Sam Lambert, October, 2011 PHP & Continuous Data Processing, PHPNW 2011 1
  • 2. SMITH ELECTRIC VEHICLES & TELEMETRY • Worlds largest manufacturer of Commercial, all-electric vehicles • Smith Link – on-board vehicle telematics system, capturing over 2500 data points each second on the vehicle and broadcasting them over mobile network • ~400 telemetry enabled vehicles on the road • Worlds largest telemetry project outside of F1 • Number of inserts per day increases by around 250,000,000 per month • Record on truck all the way to our web interface in a maximum of 30 seconds How to manage 1 billion + inserts a day into a MySQL resource 2
  • 3. SYSTEM ARCHITECTURE How to manage 1 billion + inserts a day into a MySQL resource 3
  • 4. ORIGINAL PROBLEMS Unable to insert data quick enough Problems with concurrently processing data Data processing scripts taking an unacceptable amount of time No obvious path for scaling How to manage 1 billion + inserts a day into a MySQL resource 4
  • 5. INNODB Huge scaling potential Transactional Tuneable Well supported/Strong community How to manage 1 billion + inserts a day into a MySQL resource 5
  • 6. MEASURE Measure, measure, measure Always measure as many metrics as possible De ne a scope Decide on metrics that can best determine performance improvements How to manage 1 billion + inserts a day into a MySQL resource 6
  • 7. TOOLS Sysbench Cacti (mysql-cacti-templates) Iostat Iotop mysqltuner.pl How to manage 1 billion + inserts a day into a MySQL resource 7
  • 8. RESOURCES Large amount of RAM (all machines 100GB+) Our latest server roll outs have minimum of 144GB innodb_buffer_pool_size=??? How to manage 1 billion + inserts a day into a MySQL resource 8
  • 9. STRIKING A BALANCE Maximum safe RAM utilisation Make the sure buffer pool never swaps How to manage 1 billion + inserts a day into a MySQL resource 9
  • 10. LOAD DATA INFILE Our application pulls messages from the queue and creates les for importing Have a well optimised perl application that loads the data into MySQL Inserting data in a batch gave great improvements How to manage 1 billion + inserts a day into a MySQL resource 10
  • 11. OTHER SETTINGS innodb_ le_per_table=1 innodb_ ush_log_at_trx_commit=2 innodb_ ush_method=O_DIRECT innodb_log_ le_size=1G How to manage 1 billion + inserts a day into a MySQL resource 11
  • 12. SAN Open-iSCSI settings: node.session.cmds_max node.session.queue_depth How to manage 1 billion + inserts a day into a MySQL resource 12
  • 13. MESSAGE QUEUE Makes it much easier to plan downtime Can comfortably tweak the live system Can pull data in different directions for testing etc Fast way to receive data How to manage 1 billion + inserts a day into a MySQL resource 13
  • 14. SUMMARY www.samlambert.com @isamlambert Questions? How to manage 1 billion + inserts a day into a MySQL resource 14

Hinweis der Redaktion

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n