SlideShare ist ein Scribd-Unternehmen logo
1 von 44
Downloaden Sie, um offline zu lesen
MPI Cluster
   Programming with
Python and Amazon EC2
      Pete Skomoroch

           datawrangling.com
           juiceanalytics.com/writing
Outline
Netflix Prize & Amazon EC2
Python Parallel Programming Options
ElasticWulf
MPI basics in Python
Demo: ElasticWulf, mpi4py, Ipython1
ElasticWulf Performance
EC2 pointers + Q&A
The Netflix Prize

                               17K movies
                               500K users
                               100M ratings
                               2 GB raw data

                               A “big” computation?




10% improvement of RMSE wins $1 Million
Crunching the data at home...

               Use the Pyflix library to squeeze dataset
               into 600 MB

               http://pyflix.python-hosting.com

               Simon Funk iterative SVD algorithm runs
               on your laptop overnight

               “Timely Development” code gets us to
               4% improvement over Cinematch

               Use Numpy/Scipy, call C when needed
I needed more CPUs

Basic algorithm ties up your machine for 13 hours
Some algorithms take weeks...
Need runs over many sets of parameters
Need to run cross validation over large datasets
Bugs in long jobs suck
The best approach so far ( Bell Labs ) merged the
successful results of over 107 different algorithms
But this was my only machine...
How can I get a Beowulf cluster?
Cluster of nearly identical commodity machines
Employ message passing model of parallel computation
Often have shared filesystem
Amazon EC2



                __init__(                               )


      Launch instances of Amazon Machine
      Images (AMIs) from your Python code

      Pay only for what you use (wall clock time)


http://www.flickr.com/photos/steverosebush/2241578490/
EC2 Instance Types

                         Small 1x   Large 4x   ExtraLarge 8x

        RAM               1.7GB      7.5GB        15GB

         Disk             160GB     850GB         1.6TB

  CPU (1.7Ghz)            1 32bit    4 64bit      8 64bit

        I/O
                         Moderate    High          High
   Performance
        Price             $0.10      $0.40        $0.80
   (per instance-hour)
Driving EC2 using Python

yum -y install python-boto

Blog post: “Amazon EC2 Basics for Python Programmers”
- James Gardner

PyCon talk: “Like Switching on the Light: Managing an
Elastic Compute Cluster with Python”
	 	 George Belotsky; Heath Johns

http://del.icio.us/pskomoroch/ec2
Parallel Programming in Python

 MapReduce (Hadoop + Python)
 Python MPI options (mpi4py, pyMPI, ...)
 Wrap existing C++/Fortran libs (ScaLAPACK,PETSc, ...)
 Pyro
 Twisted
 IPython1

 Which one you use depends on your particular situation

 These are not mutually exclusive or exhaustive...
Introducing ElasticWulf
ElasticWulf - batteries included
Master and worker beowulf node Amazon Machine Images
Python command line scripts to launch/configure cluster
Includes Ipython1 and other good stuff...

The AMIs are publicly visible
Updated python scripts + docs will be on Google Code

BLAS/LAPACK       OpenMPI      Xwindows      Twisted
 Numpy/Scipy       MPICH         NFS        Ipython1
   ipython        MPICH2        Ganglia      mpi4py
  matplotlib        LAM         C3 tools      boto
What is MPI?

High performance message passing interface (MPI)
Standard protocol implemented in multiple languages
Point to Point - Collective Operations
Very flexible, complex...
We will just look at a use case involving a master process
broadcasting data and receiving results from slave
processes via simple primitives (bcast(), scatter(), gather(),
reduce())
MPI Basics in Python

Lets look at some code...
size attribute: number of cooperating processes
rank attribute: unique identifier of a process



We will cover some PyMPI examples, but I would recommend using mpi4py (moved over
to scipy, used by Ipython1)

http://mpi4py.scipy.org/

The program structure is nearly identical in either implementation.
MPI Broadcast




mpi.bcast()

broadcasts a value from the root process to all other processes
MPI Scatter




mpi.scatter()

scatters an array roughly evenly across processes
MPI Gather




mpi.gather()

Collects results from all processes back to the master process
MPI Reduce




mpi.reduce()

Apply a summary function across all nodes. Like Python reduce (at
least until Guido kills it)
A simple example
  Calculating Pi.
  Throw random darts at a dartboard.
Monte Carlo Pi


 Measure the distance
 from the origin, count
 the number of “darts”
 that are within 1 unit of
 the origin, and the total
 number of darts...
Throwing more darts




42 inside the circle
8 outside the circle
estimated pi is 3.360 ( 42*4 / (42+8) )
Serial Pi Calculation
Throwing darts in parallel
Parallel Pi Calculation
Starting up your ElasticWulf Cluster


1) Sign up for Amazon Web Services
2) Get your keys/certificates + define environment variables
4) Download ElasticWulf Python scripts
4) Add your Amazon Id to the ElasticWulf config file
5) ./start-cluster.py -n 3 -s ‘xlarge’
6) ./ec2-mpi-config.py
7) ssh in to the master node...
Demo: Start the cluster
 $ ./ec2-start-cluster.py
Demo: check progress

$ ./ec2-check-instances.py
Demo: configure the nodes
   $ ./ec2-mpi-config.py
Demo: login to the master node
Demo: start the MPI daemons
Demo: mpi4py code
Demo: run mpi4py code
Running Ganglia
Demo: start ipython1 controller
Demo: start engines with mpirun
Demo: parallel ipython1
Demo: parallel ipython1
Scaling KNN on ElasticWulf
Run pearson correlation between every pair of movies...
Use Numpy & mpi4py to scatter the movie ratings vectors
Spend $2.40 for 24 cpu hours, trade $ for time
Performance
Latency: 0.000492 (microseconds)




   Netpipe Benchmark
Elasticwulf Pycon Talk
EC2 “gotchas”

EC2 is still in Beta: instances die, freeze, be prepared
Relatively high latency
Dynamic subnet
ec2-upload-bundle --retry
/etc/fstab overwritten by default in bundle-vol
more: http://del.icio.us/pskomoroch/ec2+gotchas
Questions?

Weitere ähnliche Inhalte

Was ist angesagt?

Storm presentation
Storm presentationStorm presentation
Storm presentationShyam Raj
 
Introduction to Apache Storm - Concept & Example
Introduction to Apache Storm - Concept & ExampleIntroduction to Apache Storm - Concept & Example
Introduction to Apache Storm - Concept & ExampleDung Ngua
 
Paris DataGeek - SummingBird
Paris DataGeek - SummingBirdParis DataGeek - SummingBird
Paris DataGeek - SummingBirdSofian Djamaa
 
Custom object detection_gcp
Custom object detection_gcpCustom object detection_gcp
Custom object detection_gcpVaibhav Sahu
 
Apache spark presentation
Apache spark presentationApache spark presentation
Apache spark presentationMahboob Hussain
 
Luigi presentation NYC Data Science
Luigi presentation NYC Data ScienceLuigi presentation NYC Data Science
Luigi presentation NYC Data ScienceErik Bernhardsson
 
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
(CMP310) Data Processing Pipelines Using Containers & Spot Instances(CMP310) Data Processing Pipelines Using Containers & Spot Instances
(CMP310) Data Processing Pipelines Using Containers & Spot InstancesAmazon Web Services
 
SciPy 2019: How to Accelerate an Existing Codebase with Numba
SciPy 2019: How to Accelerate an Existing Codebase with NumbaSciPy 2019: How to Accelerate an Existing Codebase with Numba
SciPy 2019: How to Accelerate an Existing Codebase with Numbastan_seibert
 
Pytorch Ecosystem Day: The 珞 Ecosystem
Pytorch Ecosystem Day: The 珞 EcosystemPytorch Ecosystem Day: The 珞 Ecosystem
Pytorch Ecosystem Day: The 珞 EcosystemLysandre Debut
 
Monitoring MySQL with Prometheus, Grafana and Percona Dashboards
Monitoring MySQL with Prometheus, Grafana and Percona DashboardsMonitoring MySQL with Prometheus, Grafana and Percona Dashboards
Monitoring MySQL with Prometheus, Grafana and Percona DashboardsJulien Pivotto
 
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow managementIntro to Airflow: Goodbye Cron, Welcome scheduled workflow management
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow managementBurasakorn Sabyeying
 
Slide #1:Introduction to Apache Storm
Slide #1:Introduction to Apache StormSlide #1:Introduction to Apache Storm
Slide #1:Introduction to Apache StormMd. Shamsur Rahim
 
Luigi presentation OA Summit
Luigi presentation OA SummitLuigi presentation OA Summit
Luigi presentation OA SummitOpen Analytics
 
Building infrastructure as code using Terraform - DevOps Krakow
Building infrastructure as code using Terraform - DevOps KrakowBuilding infrastructure as code using Terraform - DevOps Krakow
Building infrastructure as code using Terraform - DevOps KrakowAnton Babenko
 
Community-Driven Graphs with JanusGraph
Community-Driven Graphs with JanusGraphCommunity-Driven Graphs with JanusGraph
Community-Driven Graphs with JanusGraphJason Plurad
 
Managing data workflows with Luigi
Managing data workflows with LuigiManaging data workflows with Luigi
Managing data workflows with LuigiTeemu Kurppa
 

Was ist angesagt? (20)

Storm presentation
Storm presentationStorm presentation
Storm presentation
 
Introduction to Apache Storm - Concept & Example
Introduction to Apache Storm - Concept & ExampleIntroduction to Apache Storm - Concept & Example
Introduction to Apache Storm - Concept & Example
 
Paris DataGeek - SummingBird
Paris DataGeek - SummingBirdParis DataGeek - SummingBird
Paris DataGeek - SummingBird
 
Custom object detection_gcp
Custom object detection_gcpCustom object detection_gcp
Custom object detection_gcp
 
Apache spark presentation
Apache spark presentationApache spark presentation
Apache spark presentation
 
Luigi presentation NYC Data Science
Luigi presentation NYC Data ScienceLuigi presentation NYC Data Science
Luigi presentation NYC Data Science
 
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
(CMP310) Data Processing Pipelines Using Containers & Spot Instances(CMP310) Data Processing Pipelines Using Containers & Spot Instances
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
 
SciPy 2019: How to Accelerate an Existing Codebase with Numba
SciPy 2019: How to Accelerate an Existing Codebase with NumbaSciPy 2019: How to Accelerate an Existing Codebase with Numba
SciPy 2019: How to Accelerate an Existing Codebase with Numba
 
Pytorch Ecosystem Day: The 珞 Ecosystem
Pytorch Ecosystem Day: The 珞 EcosystemPytorch Ecosystem Day: The 珞 Ecosystem
Pytorch Ecosystem Day: The 珞 Ecosystem
 
Apache Storm Internals
Apache Storm InternalsApache Storm Internals
Apache Storm Internals
 
Tensorflow vs MxNet
Tensorflow vs MxNetTensorflow vs MxNet
Tensorflow vs MxNet
 
Luigi future
Luigi futureLuigi future
Luigi future
 
Monitoring MySQL with Prometheus, Grafana and Percona Dashboards
Monitoring MySQL with Prometheus, Grafana and Percona DashboardsMonitoring MySQL with Prometheus, Grafana and Percona Dashboards
Monitoring MySQL with Prometheus, Grafana and Percona Dashboards
 
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow managementIntro to Airflow: Goodbye Cron, Welcome scheduled workflow management
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management
 
Slide #1:Introduction to Apache Storm
Slide #1:Introduction to Apache StormSlide #1:Introduction to Apache Storm
Slide #1:Introduction to Apache Storm
 
Luigi presentation OA Summit
Luigi presentation OA SummitLuigi presentation OA Summit
Luigi presentation OA Summit
 
Building infrastructure as code using Terraform - DevOps Krakow
Building infrastructure as code using Terraform - DevOps KrakowBuilding infrastructure as code using Terraform - DevOps Krakow
Building infrastructure as code using Terraform - DevOps Krakow
 
Terraform on Azure
Terraform on AzureTerraform on Azure
Terraform on Azure
 
Community-Driven Graphs with JanusGraph
Community-Driven Graphs with JanusGraphCommunity-Driven Graphs with JanusGraph
Community-Driven Graphs with JanusGraph
 
Managing data workflows with Luigi
Managing data workflows with LuigiManaging data workflows with Luigi
Managing data workflows with Luigi
 

Ähnlich wie Elasticwulf Pycon Talk

Unmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/InvokeUnmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/InvokeDmitri Nesteruk
 
PyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsPyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsHenry Schreiner
 
Euro python2011 High Performance Python
Euro python2011 High Performance PythonEuro python2011 High Performance Python
Euro python2011 High Performance PythonIan Ozsvald
 
Deep learning with kafka
Deep learning with kafkaDeep learning with kafka
Deep learning with kafkaNitin Kumar
 
Parallelism in a NumPy-based program
Parallelism in a NumPy-based programParallelism in a NumPy-based program
Parallelism in a NumPy-based programRalf Gommers
 
Packer Genetics: The selfish code
Packer Genetics: The selfish codePacker Genetics: The selfish code
Packer Genetics: The selfish codejduart
 
PyCon_India_2017_MicroPython_Ayan
PyCon_India_2017_MicroPython_AyanPyCon_India_2017_MicroPython_Ayan
PyCon_India_2017_MicroPython_AyanAyan Pahwa
 
GoSF Jan 2016 - Go Write a Plugin for Snap!
GoSF Jan 2016 - Go Write a Plugin for Snap!GoSF Jan 2016 - Go Write a Plugin for Snap!
GoSF Jan 2016 - Go Write a Plugin for Snap!Matthew Broberg
 
Golang Performance : microbenchmarks, profilers, and a war story
Golang Performance : microbenchmarks, profilers, and a war storyGolang Performance : microbenchmarks, profilers, and a war story
Golang Performance : microbenchmarks, profilers, and a war storyAerospike
 
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with SparkSparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Sparkfelixcss
 
20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo Omura
20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo Omura20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo Omura
20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo OmuraPreferred Networks
 
May2010 hex-core-opt
May2010 hex-core-optMay2010 hex-core-opt
May2010 hex-core-optJeff Larkin
 
Leveraging Elastic Web-Scale Computing with AWS
Leveraging Elastic Web-Scale Computing with AWSLeveraging Elastic Web-Scale Computing with AWS
Leveraging Elastic Web-Scale Computing with AWSAmazon Web Services
 
Deep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceDeep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceAmazon Web Services
 
Concurrency and Python - PyCon MY 2015
Concurrency and Python - PyCon MY 2015Concurrency and Python - PyCon MY 2015
Concurrency and Python - PyCon MY 2015Boey Pak Cheong
 

Ähnlich wie Elasticwulf Pycon Talk (20)

Unmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/InvokeUnmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/Invoke
 
PyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsPyCon2022 - Building Python Extensions
PyCon2022 - Building Python Extensions
 
Euro python2011 High Performance Python
Euro python2011 High Performance PythonEuro python2011 High Performance Python
Euro python2011 High Performance Python
 
Deep learning with kafka
Deep learning with kafkaDeep learning with kafka
Deep learning with kafka
 
Parallelism in a NumPy-based program
Parallelism in a NumPy-based programParallelism in a NumPy-based program
Parallelism in a NumPy-based program
 
Packer Genetics: The selfish code
Packer Genetics: The selfish codePacker Genetics: The selfish code
Packer Genetics: The selfish code
 
Docker meetup
Docker meetupDocker meetup
Docker meetup
 
PyCon_India_2017_MicroPython_Ayan
PyCon_India_2017_MicroPython_AyanPyCon_India_2017_MicroPython_Ayan
PyCon_India_2017_MicroPython_Ayan
 
Multicore
MulticoreMulticore
Multicore
 
PyCon Estonia 2019
PyCon Estonia 2019PyCon Estonia 2019
PyCon Estonia 2019
 
mpi4py.pdf
mpi4py.pdfmpi4py.pdf
mpi4py.pdf
 
GoSF Jan 2016 - Go Write a Plugin for Snap!
GoSF Jan 2016 - Go Write a Plugin for Snap!GoSF Jan 2016 - Go Write a Plugin for Snap!
GoSF Jan 2016 - Go Write a Plugin for Snap!
 
Golang Performance : microbenchmarks, profilers, and a war story
Golang Performance : microbenchmarks, profilers, and a war storyGolang Performance : microbenchmarks, profilers, and a war story
Golang Performance : microbenchmarks, profilers, and a war story
 
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with SparkSparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Spark
 
20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo Omura
20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo Omura20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo Omura
20180926 kubeflow-meetup-1-kubeflow-operators-Preferred Networks-Shingo Omura
 
May2010 hex-core-opt
May2010 hex-core-optMay2010 hex-core-opt
May2010 hex-core-opt
 
Cloud Talk
Cloud TalkCloud Talk
Cloud Talk
 
Leveraging Elastic Web-Scale Computing with AWS
Leveraging Elastic Web-Scale Computing with AWSLeveraging Elastic Web-Scale Computing with AWS
Leveraging Elastic Web-Scale Computing with AWS
 
Deep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceDeep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance Performance
 
Concurrency and Python - PyCon MY 2015
Concurrency and Python - PyCon MY 2015Concurrency and Python - PyCon MY 2015
Concurrency and Python - PyCon MY 2015
 

Mehr von Peter Skomoroch

Bridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder SupportBridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder SupportPeter Skomoroch
 
Managing Machines: The New AI Dev Stack
Managing Machines: The New AI Dev StackManaging Machines: The New AI Dev Stack
Managing Machines: The New AI Dev StackPeter Skomoroch
 
Product Management for AI
Product Management for AIProduct Management for AI
Product Management for AIPeter Skomoroch
 
Executive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkExecutive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkPeter Skomoroch
 
Building Competitive Moats With Data
Building Competitive Moats With DataBuilding Competitive Moats With Data
Building Competitive Moats With DataPeter Skomoroch
 
O'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data ExhaustO'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data ExhaustPeter Skomoroch
 
SF Data Science: Developing Data Products
SF Data Science: Developing Data ProductsSF Data Science: Developing Data Products
SF Data Science: Developing Data ProductsPeter Skomoroch
 
Skills, Reputation, and Search
Skills, Reputation, and SearchSkills, Reputation, and Search
Skills, Reputation, and SearchPeter Skomoroch
 
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social TaggingLinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social TaggingPeter Skomoroch
 
Developing Data Products
Developing Data ProductsDeveloping Data Products
Developing Data ProductsPeter Skomoroch
 
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, TokyoPractical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, TokyoPeter Skomoroch
 
Street Fighting Data Science
Street Fighting Data ScienceStreet Fighting Data Science
Street Fighting Data SciencePeter Skomoroch
 
Data Mashups -Data Science Summit
Data Mashups -Data Science SummitData Mashups -Data Science Summit
Data Mashups -Data Science SummitPeter Skomoroch
 
Geo Analytics Tutorial - Where 2.0 2011
Geo Analytics Tutorial - Where 2.0 2011Geo Analytics Tutorial - Where 2.0 2011
Geo Analytics Tutorial - Where 2.0 2011Peter Skomoroch
 
Rapid Data Exploration With Hadoop
Rapid Data Exploration With HadoopRapid Data Exploration With Hadoop
Rapid Data Exploration With HadoopPeter Skomoroch
 

Mehr von Peter Skomoroch (15)

Bridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder SupportBridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder Support
 
Managing Machines: The New AI Dev Stack
Managing Machines: The New AI Dev StackManaging Machines: The New AI Dev Stack
Managing Machines: The New AI Dev Stack
 
Product Management for AI
Product Management for AIProduct Management for AI
Product Management for AI
 
Executive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkExecutive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you think
 
Building Competitive Moats With Data
Building Competitive Moats With DataBuilding Competitive Moats With Data
Building Competitive Moats With Data
 
O'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data ExhaustO'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data Exhaust
 
SF Data Science: Developing Data Products
SF Data Science: Developing Data ProductsSF Data Science: Developing Data Products
SF Data Science: Developing Data Products
 
Skills, Reputation, and Search
Skills, Reputation, and SearchSkills, Reputation, and Search
Skills, Reputation, and Search
 
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social TaggingLinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
 
Developing Data Products
Developing Data ProductsDeveloping Data Products
Developing Data Products
 
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, TokyoPractical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
 
Street Fighting Data Science
Street Fighting Data ScienceStreet Fighting Data Science
Street Fighting Data Science
 
Data Mashups -Data Science Summit
Data Mashups -Data Science SummitData Mashups -Data Science Summit
Data Mashups -Data Science Summit
 
Geo Analytics Tutorial - Where 2.0 2011
Geo Analytics Tutorial - Where 2.0 2011Geo Analytics Tutorial - Where 2.0 2011
Geo Analytics Tutorial - Where 2.0 2011
 
Rapid Data Exploration With Hadoop
Rapid Data Exploration With HadoopRapid Data Exploration With Hadoop
Rapid Data Exploration With Hadoop
 

Kürzlich hochgeladen

9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 

Kürzlich hochgeladen (20)

20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 

Elasticwulf Pycon Talk

  • 1. MPI Cluster Programming with Python and Amazon EC2 Pete Skomoroch datawrangling.com juiceanalytics.com/writing
  • 2. Outline Netflix Prize & Amazon EC2 Python Parallel Programming Options ElasticWulf MPI basics in Python Demo: ElasticWulf, mpi4py, Ipython1 ElasticWulf Performance EC2 pointers + Q&A
  • 3. The Netflix Prize 17K movies 500K users 100M ratings 2 GB raw data A “big” computation? 10% improvement of RMSE wins $1 Million
  • 4. Crunching the data at home... Use the Pyflix library to squeeze dataset into 600 MB http://pyflix.python-hosting.com Simon Funk iterative SVD algorithm runs on your laptop overnight “Timely Development” code gets us to 4% improvement over Cinematch Use Numpy/Scipy, call C when needed
  • 5. I needed more CPUs Basic algorithm ties up your machine for 13 hours Some algorithms take weeks... Need runs over many sets of parameters Need to run cross validation over large datasets Bugs in long jobs suck The best approach so far ( Bell Labs ) merged the successful results of over 107 different algorithms
  • 6. But this was my only machine...
  • 7. How can I get a Beowulf cluster? Cluster of nearly identical commodity machines Employ message passing model of parallel computation Often have shared filesystem
  • 8. Amazon EC2 __init__( ) Launch instances of Amazon Machine Images (AMIs) from your Python code Pay only for what you use (wall clock time) http://www.flickr.com/photos/steverosebush/2241578490/
  • 9. EC2 Instance Types Small 1x Large 4x ExtraLarge 8x RAM 1.7GB 7.5GB 15GB Disk 160GB 850GB 1.6TB CPU (1.7Ghz) 1 32bit 4 64bit 8 64bit I/O Moderate High High Performance Price $0.10 $0.40 $0.80 (per instance-hour)
  • 10. Driving EC2 using Python yum -y install python-boto Blog post: “Amazon EC2 Basics for Python Programmers” - James Gardner PyCon talk: “Like Switching on the Light: Managing an Elastic Compute Cluster with Python” George Belotsky; Heath Johns http://del.icio.us/pskomoroch/ec2
  • 11. Parallel Programming in Python MapReduce (Hadoop + Python) Python MPI options (mpi4py, pyMPI, ...) Wrap existing C++/Fortran libs (ScaLAPACK,PETSc, ...) Pyro Twisted IPython1 Which one you use depends on your particular situation These are not mutually exclusive or exhaustive...
  • 13. ElasticWulf - batteries included Master and worker beowulf node Amazon Machine Images Python command line scripts to launch/configure cluster Includes Ipython1 and other good stuff... The AMIs are publicly visible Updated python scripts + docs will be on Google Code BLAS/LAPACK OpenMPI Xwindows Twisted Numpy/Scipy MPICH NFS Ipython1 ipython MPICH2 Ganglia mpi4py matplotlib LAM C3 tools boto
  • 14. What is MPI? High performance message passing interface (MPI) Standard protocol implemented in multiple languages Point to Point - Collective Operations Very flexible, complex... We will just look at a use case involving a master process broadcasting data and receiving results from slave processes via simple primitives (bcast(), scatter(), gather(), reduce())
  • 15. MPI Basics in Python Lets look at some code... size attribute: number of cooperating processes rank attribute: unique identifier of a process We will cover some PyMPI examples, but I would recommend using mpi4py (moved over to scipy, used by Ipython1) http://mpi4py.scipy.org/ The program structure is nearly identical in either implementation.
  • 16. MPI Broadcast mpi.bcast() broadcasts a value from the root process to all other processes
  • 17. MPI Scatter mpi.scatter() scatters an array roughly evenly across processes
  • 18. MPI Gather mpi.gather() Collects results from all processes back to the master process
  • 19. MPI Reduce mpi.reduce() Apply a summary function across all nodes. Like Python reduce (at least until Guido kills it)
  • 20. A simple example Calculating Pi. Throw random darts at a dartboard.
  • 21. Monte Carlo Pi Measure the distance from the origin, count the number of “darts” that are within 1 unit of the origin, and the total number of darts...
  • 22. Throwing more darts 42 inside the circle 8 outside the circle estimated pi is 3.360 ( 42*4 / (42+8) )
  • 24. Throwing darts in parallel
  • 26. Starting up your ElasticWulf Cluster 1) Sign up for Amazon Web Services 2) Get your keys/certificates + define environment variables 4) Download ElasticWulf Python scripts 4) Add your Amazon Id to the ElasticWulf config file 5) ./start-cluster.py -n 3 -s ‘xlarge’ 6) ./ec2-mpi-config.py 7) ssh in to the master node...
  • 27. Demo: Start the cluster $ ./ec2-start-cluster.py
  • 28. Demo: check progress $ ./ec2-check-instances.py
  • 29. Demo: configure the nodes $ ./ec2-mpi-config.py
  • 30. Demo: login to the master node
  • 31. Demo: start the MPI daemons
  • 35. Demo: start ipython1 controller
  • 36. Demo: start engines with mpirun
  • 39. Scaling KNN on ElasticWulf Run pearson correlation between every pair of movies... Use Numpy & mpi4py to scatter the movie ratings vectors Spend $2.40 for 24 cpu hours, trade $ for time
  • 41. Latency: 0.000492 (microseconds) Netpipe Benchmark
  • 43. EC2 “gotchas” EC2 is still in Beta: instances die, freeze, be prepared Relatively high latency Dynamic subnet ec2-upload-bundle --retry /etc/fstab overwritten by default in bundle-vol more: http://del.icio.us/pskomoroch/ec2+gotchas