Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Tensorflow in production
with AWS Lambda
Tensorflow Tokyo - 2016-09-15
THIS IS NOT
A MACHINE
LEARNING
TALK
Tensorflow in production with AWS lambda
What Will You Learn?
▸ What can you do with your trained model: MLOPS
▸ Why AWS l...
Tensorflow in production with AWS lambda
About Me
▸ Freelance Data Products Developper and Consultant

(data visualization...
So you have trained
a model?
Now what?
Tensorflow in production with AWS lambda
It is a product, not an ad-hoc analysis


Live Data
Historical Data
" "
Trained...
Tensorflow in production with AWS lambda
Many things may need to be done in production
▸ Batch processing
▸ Stream / event...
Tensorflow in production with AWS lambda
This needs to be scalable, resilient
And also:
▸ maintainable
▸ versioned
▸ easy ...
Why AWS Lambda
may help.
Tensorflow in production with AWS lambda
Some deployment solutions
▸ Tensor flow Serving:
▸ Forces you to create dedicated ...
Tensorflow in production with AWS lambda
Serverless architectures with AWS Lambda
▸ Serverless offer by AWS
▸ No lifecycle ...
Tensorflow in production with AWS lambda
Creating a function
Tensorflow in production with AWS lambda
Creating a function
Tensorflow in production with AWS lambda
Creating an “architecture” with triggers
Tensorflow in production with AWS lambda
Event / microbatch processing
▸ event based: db/stream update, new file on s3, web...
Tensorflow in production with AWS lambda
Batch processing
▸ cron scheduling
▸ let your function get some data and process ...
Tensorflow in production with AWS lambda
An API
▸ on API call
▸ returned response is your function return value
▸ manage A...
Tensorflow and
AWS Lambda in
practice.
Tensorflow in production with AWS lambda
How to save a TF model
▸ Use a saver object.
▸ It will save on disk:
▸ the graph ...
Tensorflow in production with AWS lambda
How to restore a TF model
▸ Restore the graph and variable values with a saver ob...
Tensorflow in production with AWS lambda
Setting up AWS Lambda for Tensorflow
Tensorflow needs to be compiled for the right ...
Tensorflow in production with AWS lambda
A tensorflow calling lambda function
▸ Accepts a list of input vectors: multiple p...
Tensorflow in production with AWS lambda
upload and test
▸ add your lambda function code and TF model to the environment z...
Tensorflow in production with AWS lambda
Where to put the model?
▸ with the function: easy, in particular when testing
▸ o...
Tensorflow in production with AWS lambda
caveats
▸ No GPU support at the moment
▸ model loading time: better to increase m...
Your turn!
Thanks
Questions?
@fabian_dubois
fabian@datamaplab.com
check denryoku.io
Text
references:
▸ http://docs.aws.amazon.com/lambda/latest/dg/current-
supported-versions.html
▸ tensor flow package url h...
Nächste SlideShare
Wird geladen in …5
×

Tensorflow in production with AWS Lambda

13.760 Aufrufe

Veröffentlicht am

How to put your Tensorflow machine learning model to work with AWS Lambda. Batch processing, stream processing or an API? you are covered.

Veröffentlicht in: Software
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Tensorflow and its supporting libraries eat upto 307MB of space. AWS Lambda not accepting as packages
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • We can use Kubernetes cluster for TensorFlow Serving to scale up the service automatically. To me it seems more reliable than rely on constant traffic. And if AWS Lambda keeps running because of a traffic, wouldn't we have high costs too?
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • If you have a reasonable amount of traffic to your API Lambda will keep it running - which means your model will remain in memory. So you can avoid re-loading it each time and reuse the graph between requests.
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • @Stathis Boii Thanks for the input. While individual queries response time will indeed be slower than with a continuously running server, it can still be more efficient in term of cost. I will explore possible optimisations.
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Tensorflow in production with AWS Lambda

  1. 1. Tensorflow in production with AWS Lambda Tensorflow Tokyo - 2016-09-15
  2. 2. THIS IS NOT A MACHINE LEARNING TALK
  3. 3. Tensorflow in production with AWS lambda What Will You Learn? ▸ What can you do with your trained model: MLOPS ▸ Why AWS lambda can be a solution ▸ AWS lambda with tensor flow: how it works
  4. 4. Tensorflow in production with AWS lambda About Me ▸ Freelance Data Products Developper and Consultant
 (data visualization, machine learning) ▸ Former Orange Labs and Locarise
 (connected sensors data processing and visualization) ▸ Current side project denryoku.io an API for electric grid power demand and capacity prediction.
  5. 5. So you have trained a model? Now what?
  6. 6. Tensorflow in production with AWS lambda It is a product, not an ad-hoc analysis   Live Data Historical Data " " Trained model Deployed model Prediction Model selection and training Production ▸ Needs to run on live data
  7. 7. Tensorflow in production with AWS lambda Many things may need to be done in production ▸ Batch processing ▸ Stream / event processing ▸ A prediction API ▸ Update and maintain the model
  8. 8. Tensorflow in production with AWS lambda This needs to be scalable, resilient And also: ▸ maintainable ▸ versioned ▸ easy to integrate ML+DevOps = MLOPS
  9. 9. Why AWS Lambda may help.
  10. 10. Tensorflow in production with AWS lambda Some deployment solutions ▸ Tensor flow Serving: ▸ Forces you to create dedicated code if you have more than a pure Tensorflow model ▸ doesn’t solve scalability issues ▸ forces you to manage servers ▸ Google CloudML ▸ Private Beta ▸ Likely limitations
  11. 11. Tensorflow in production with AWS lambda Serverless architectures with AWS Lambda ▸ Serverless offer by AWS ▸ No lifecycle to manage or shared state => resilient ▸ Auto-scaling ▸ Pay for actual running time: low cost ▸ No server, infra management: reduced dev / devops cost …events lambda function output
  12. 12. Tensorflow in production with AWS lambda Creating a function
  13. 13. Tensorflow in production with AWS lambda Creating a function
  14. 14. Tensorflow in production with AWS lambda Creating an “architecture” with triggers
  15. 15. Tensorflow in production with AWS lambda Event / microbatch processing ▸ event based: db/stream update, new file on s3, web hook ▸ classify the incoming data or update your prediction
  16. 16. Tensorflow in production with AWS lambda Batch processing ▸ cron scheduling ▸ let your function get some data and process it at regular interval
  17. 17. Tensorflow in production with AWS lambda An API ▸ on API call ▸ returned response is your function return value ▸ manage API keys, rate limits, etc on AWS gateway
  18. 18. Tensorflow and AWS Lambda in practice.
  19. 19. Tensorflow in production with AWS lambda How to save a TF model ▸ Use a saver object. ▸ It will save on disk: ▸ the graph model (‘filename.meta’) ▸ the variable values (‘filename’) ▸ Need to identify the placeholders that will be accessed later saver = tf.train.Saver() # # do the training # tf.add_to_collection('output', pred) tf.add_to_collection('input', x) save_path = saver.save(sess, "model-name.ckpt") python
  20. 20. Tensorflow in production with AWS lambda How to restore a TF model ▸ Restore the graph and variable values with a saver object saver = tf.train.import_meta_graph(filename + '.meta') with tf.Session() as sess: # Restore variables from disk. saver.restore(sess, filename) pred = tf.get_collection('output')[0] x = tf.get_collection('input')[0] print("Model restored.") # Do some work with the model prediction = pred.eval({x: test_data}) python
  21. 21. Tensorflow in production with AWS lambda Setting up AWS Lambda for Tensorflow Tensorflow needs to be compiled for the right environment # install compilation environment sudo yum -y update sudo yum -y upgrade sudo yum groupinstall "Development Tools" # create and activate virtual env virtualenv tfenv source tfenv/bin/activate # install tensorflow export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/ linux/cpu/tensorflow-0.10.0-cp27-none-linux_x86_64.whl pip install --upgrade $TF_BINARY_URL # zip the environment content touch ~/tfenv/lib/python2.7/site-packages/google/__init__.py cd ~/tfenv/lib/python2.7/site-packages/ zip -r ~/tf-env.zip . --exclude *.pyc cd ~/tfenv/lib64/python2.7/site-packages/ 1. Launch an EC2 instance and connect to it 2. Install TensorFlow in a virtualenv 3. Zip the installed libraries shell
  22. 22. Tensorflow in production with AWS lambda A tensorflow calling lambda function ▸ Accepts a list of input vectors: multiple predictions ▸ Returns a list of predictions import tensorflow as tf filename = 'model-name.ckpt' def lambda_handler(event, context): saver = tf.train.import_meta_graph(filename + '.meta') inputData = event['data'] with tf.Session() as sess: # Restore variables from disk. saver.restore(sess, filename) pred = tf.get_collection('pred')[0] x = tf.get_collection('x')[0] # Apply the model to the input data predictions = pred.eval({x: inputData}) return {'result': predictions.tolist()} python
  23. 23. Tensorflow in production with AWS lambda upload and test ▸ add your lambda function code and TF model to the environment zip. ▸ upload your function
  24. 24. Tensorflow in production with AWS lambda Where to put the model? ▸ with the function: easy, in particular when testing ▸ on s3: ease update or allows for multiple models to be used in parallel. ▸ function could be called with model ref as argument "… lambda function tensor flowlive data prediction $model
  25. 25. Tensorflow in production with AWS lambda caveats ▸ No GPU support at the moment ▸ model loading time: better to increase machine RAM (hence CPU) for fast API response time ▸ python 2.7 (python 3 doable with more work) ▸ request limit increase to AWS for more than 100 concurrent executions
  26. 26. Your turn!
  27. 27. Thanks Questions? @fabian_dubois fabian@datamaplab.com check denryoku.io
  28. 28. Text references: ▸ http://docs.aws.amazon.com/lambda/latest/dg/current- supported-versions.html ▸ tensor flow package url https://www.tensorflow.org/ versions/r0.10/get_started/os_setup.html

×