SlideShare ist ein Scribd-Unternehmen logo
1 von 39
Downloaden Sie, um offline zu lesen
Building a Cloud Culture at Yelp
Jim Blomo – Engineering Manager, Yelp
November 15, 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Why Cloud?
Yelp!
Yelp Data
Why Cloud?
specificity & optimization
generality & abstraction
Make the Trade-Off!
How Cloud?
Back to the Past
Logging – an aside

{user_id: 5,
request_id: "b1946ac92492d2347c6235b4d2611184",
search_query: "farm to table",
city: "Portland, OR",
results: [17289, 8230452, 825429, 184312,...],
timestamp: 1384469660
}
Hadoop Trade-Offs
Riddle: Success?
• How do you know
when your
infrastructure is a
success?
• Starts failing under
heavy load
• "Too many" users
overloading the system
Hadoop Issues
EMR Solutions

S3

• Clusters up for limited
amount of time
• Upgrades handled by
Amazon
• Multiple clusters
means no capacity
coordination
Trade-Offs
Trade-Offs
Trade-Offs
Standard configs
Resource consumption tracking
testing
No Cargo Cults
Standard Configs
mrjob Configs
# standard

# memory intensive

# cpu intensive
Resource Tracking

python -m mrjob.tools.emr.terminate_idle_job_flows
-c mrjob.conf
Testing
--runner emr

mr_canary.py
mrjob is Open Source
Adoption
Cloud Calculations
• 5 days with 10
machines = 1 day
with 50 machines
• (On demand pricing
simplification)
Cost
Cost
Cost Control
Data Availability

• s3mysqldump
Overview
in: logs & DB dumps

in: JSON logs

out: CSV, JSON, MyISAM
in: s3mysqldump
S3 out: LOAD DATA or rsync
Pitfalls
Leaky Abstractions
Closed Source

bootstrap-actions/configure-hadoop
mapred.reduce.tasks.speculative.execution=false
Data Explosion

Tron
Wanted
Next Up: Services
Cloud Strategy
Target
generality & abstraction

easiest way
Tandem
We’re Hiring

yelp.com/careers
Please give us your feedback on this
presentation

BDT305 – Cloud @Yelp
As a thank you, we will select prize
winners daily for completed surveys!

Weitere ähnliche Inhalte

Andere mochten auch

AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
Amazon Web Services
 

Andere mochten auch (19)

Understanding AWS Security
Understanding AWS SecurityUnderstanding AWS Security
Understanding AWS Security
 
Understanding AWS Security
Understanding AWS SecurityUnderstanding AWS Security
Understanding AWS Security
 
Journey Through the Cloud - Security Best Practices on AWS
Journey Through the Cloud - Security Best Practices on AWSJourney Through the Cloud - Security Best Practices on AWS
Journey Through the Cloud - Security Best Practices on AWS
 
AWS Deployment Best Practices
AWS Deployment Best PracticesAWS Deployment Best Practices
AWS Deployment Best Practices
 
AWS 201 - A Walk through the AWS Cloud: AWS Security Best Practices
AWS 201 - A Walk through the AWS Cloud: AWS Security Best PracticesAWS 201 - A Walk through the AWS Cloud: AWS Security Best Practices
AWS 201 - A Walk through the AWS Cloud: AWS Security Best Practices
 
Optimizing for Cost in the AWS Cloud - 5 Ways to Further Save - AWS Summit 20...
Optimizing for Cost in the AWS Cloud - 5 Ways to Further Save - AWS Summit 20...Optimizing for Cost in the AWS Cloud - 5 Ways to Further Save - AWS Summit 20...
Optimizing for Cost in the AWS Cloud - 5 Ways to Further Save - AWS Summit 20...
 
Advanced Security Best Practices Masterclass
Advanced Security Best Practices MasterclassAdvanced Security Best Practices Masterclass
Advanced Security Best Practices Masterclass
 
Introduction to AWS Security
Introduction to AWS SecurityIntroduction to AWS Security
Introduction to AWS Security
 
(SEC201) How Should We All Think About Security?
(SEC201) How Should We All Think About Security?(SEC201) How Should We All Think About Security?
(SEC201) How Should We All Think About Security?
 
AWS Security Best Practices and Design Patterns
AWS Security Best Practices and Design PatternsAWS Security Best Practices and Design Patterns
AWS Security Best Practices and Design Patterns
 
Security Best Practices on AWS
Security Best Practices on AWSSecurity Best Practices on AWS
Security Best Practices on AWS
 
Security and Compliance in the Cloud
Security and Compliance in the Cloud Security and Compliance in the Cloud
Security and Compliance in the Cloud
 
Cloud security best practices in AWS by: Ankit Giri
Cloud security best practices in AWS by: Ankit GiriCloud security best practices in AWS by: Ankit Giri
Cloud security best practices in AWS by: Ankit Giri
 
AWS Security – Keynote Address (SEC101) | AWS re:Invent 2013
AWS Security – Keynote Address (SEC101) | AWS re:Invent 2013AWS Security – Keynote Address (SEC101) | AWS re:Invent 2013
AWS Security – Keynote Address (SEC101) | AWS re:Invent 2013
 
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
 
Architecting for End-to-End Security in the Enterprise (ARC308) | AWS re:Inve...
Architecting for End-to-End Security in the Enterprise (ARC308) | AWS re:Inve...Architecting for End-to-End Security in the Enterprise (ARC308) | AWS re:Inve...
Architecting for End-to-End Security in the Enterprise (ARC308) | AWS re:Inve...
 
How to Build a Successful AWS Consulting Practice
How to Build a Successful AWS Consulting PracticeHow to Build a Successful AWS Consulting Practice
How to Build a Successful AWS Consulting Practice
 
Best Practices for Managing Security Operations in AWS - AWS July 2016 Webina...
Best Practices for Managing Security Operations in AWS - AWS July 2016 Webina...Best Practices for Managing Security Operations in AWS - AWS July 2016 Webina...
Best Practices for Managing Security Operations in AWS - AWS July 2016 Webina...
 
(SEC304) Architecting for HIPAA Compliance on AWS
(SEC304) Architecting for HIPAA Compliance on AWS(SEC304) Architecting for HIPAA Compliance on AWS
(SEC304) Architecting for HIPAA Compliance on AWS
 

Ähnlich wie Building a Cloud Culture at Yelp (BDT305) | AWS re:Invent 2013

Go2Group's presentation on ConnectALL at HP Discover 2014.
Go2Group's presentation on ConnectALL at HP Discover 2014.  Go2Group's presentation on ConnectALL at HP Discover 2014.
Go2Group's presentation on ConnectALL at HP Discover 2014.
Go2Group, Inc.
 
Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...
Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...
Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...
SlideTeam
 
Is10 innovative technology 8
Is10 innovative technology 8Is10 innovative technology 8
Is10 innovative technology 8
hicksonj
 

Ähnlich wie Building a Cloud Culture at Yelp (BDT305) | AWS re:Invent 2013 (20)

How Cloud Providers are Playing with Traditional Data Centers
How Cloud Providers are Playing with Traditional Data CentersHow Cloud Providers are Playing with Traditional Data Centers
How Cloud Providers are Playing with Traditional Data Centers
 
Hadoop Migration Partner Videov2.pptx
Hadoop Migration Partner Videov2.pptxHadoop Migration Partner Videov2.pptx
Hadoop Migration Partner Videov2.pptx
 
Damag - EmPower your BI Architecture
Damag - EmPower your BI ArchitectureDamag - EmPower your BI Architecture
Damag - EmPower your BI Architecture
 
Edmunds.com: Migrating, Deploying & Managing On-Premises Web Property (DMG205...
Edmunds.com: Migrating, Deploying & Managing On-Premises Web Property (DMG205...Edmunds.com: Migrating, Deploying & Managing On-Premises Web Property (DMG205...
Edmunds.com: Migrating, Deploying & Managing On-Premises Web Property (DMG205...
 
Go2Group's presentation on ConnectALL at HP Discover 2014.
Go2Group's presentation on ConnectALL at HP Discover 2014.  Go2Group's presentation on ConnectALL at HP Discover 2014.
Go2Group's presentation on ConnectALL at HP Discover 2014.
 
Your Move to the Cloud: Everything You Need to Know
Your Move to the Cloud: Everything You Need to KnowYour Move to the Cloud: Everything You Need to Know
Your Move to the Cloud: Everything You Need to Know
 
Performance Testing: Eliminate System Outages and Save Millions
Performance Testing: Eliminate System Outages and Save MillionsPerformance Testing: Eliminate System Outages and Save Millions
Performance Testing: Eliminate System Outages and Save Millions
 
Prescriptive Cloud Services for the Future Ready Enterprise
Prescriptive Cloud Services for the Future Ready EnterprisePrescriptive Cloud Services for the Future Ready Enterprise
Prescriptive Cloud Services for the Future Ready Enterprise
 
Sugar crm and ibm cast ironwebinar 1-1
Sugar crm and ibm cast ironwebinar 1-1Sugar crm and ibm cast ironwebinar 1-1
Sugar crm and ibm cast ironwebinar 1-1
 
Google Cloud Platform (GCP)
Google Cloud Platform (GCP)Google Cloud Platform (GCP)
Google Cloud Platform (GCP)
 
Don't Let Your Shoppers Drop; 5 Rules for Today’s eCommerce
Don't Let Your Shoppers Drop; 5 Rules for Today’s eCommerceDon't Let Your Shoppers Drop; 5 Rules for Today’s eCommerce
Don't Let Your Shoppers Drop; 5 Rules for Today’s eCommerce
 
How to Budget for Cloud-Based Disaster Recovery
How to Budget for Cloud-Based Disaster RecoveryHow to Budget for Cloud-Based Disaster Recovery
How to Budget for Cloud-Based Disaster Recovery
 
Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...
Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...
Cloud Computing Roadmap Public Vs Private Vs Hybrid And SaaS Vs PaaS Vs IaaS ...
 
Google developer group 2021 - Introduction to cloud computing
Google developer group 2021 - Introduction to cloud computingGoogle developer group 2021 - Introduction to cloud computing
Google developer group 2021 - Introduction to cloud computing
 
Cloud Computing
Cloud ComputingCloud Computing
Cloud Computing
 
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...
 
Is10 innovative technology 8
Is10 innovative technology 8Is10 innovative technology 8
Is10 innovative technology 8
 
Citihub IDC Event 2009 Cloud Mark Ellis
Citihub IDC Event 2009 Cloud Mark EllisCitihub IDC Event 2009 Cloud Mark Ellis
Citihub IDC Event 2009 Cloud Mark Ellis
 
Hadoop and the Relational Database: The Best of Both Worlds
Hadoop and the Relational Database: The Best of Both WorldsHadoop and the Relational Database: The Best of Both Worlds
Hadoop and the Relational Database: The Best of Both Worlds
 
Building Business Success from Buzz Words
Building Business Success from Buzz WordsBuilding Business Success from Buzz Words
Building Business Success from Buzz Words
 

Mehr von Amazon Web Services

Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
Amazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
Amazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
Amazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
Amazon Web Services
 

Mehr von Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Kürzlich hochgeladen

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Kürzlich hochgeladen (20)

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 

Building a Cloud Culture at Yelp (BDT305) | AWS re:Invent 2013