Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

All AI Roads lead to Distribution - Dot AI

706 Aufrufe

Veröffentlicht am

Dot AI talk in Paris, May 2018 by Jim Dowling, Logical Clocks

Veröffentlicht in: Internet
  • Do This Simple 2-Minute Ritual To Loss 1 Pound Of Belly Fat Every 72 Hours ◆◆◆ http://scamcb.com/bkfitness3/pdf
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Gehören Sie zu den Ersten, denen das gefällt!

All AI Roads lead to Distribution - Dot AI

  1. 1. All AI Roads Lead to Distribution jim_dowling dotAI Paris 2018
  2. 2. ©2018 Logical Clocks AB. All Rights Reserved 2/38 “Methods that scale with computation are the future of AI”* - Rich Sutton (Founding Father of Reinforcement Learning) * https://www.youtube.com/watch?v=EeMCEQa85tw
  3. 3. ©2018 Logical Clocks AB. All Rights Reserved Massive Increase in Compute for AI* Distributed Systems 3.5 month-doubling time *https://blog.openai.com/ai-and-compute 3/38
  4. 4. ©2018 Logical Clocks AB. All Rights Reserved Model Parallelism Data Parallelism GPU2 GPU3 GPU1Layers1-100 Layer100-199 Layer200-299 Training/Test Data Predictions GPU0 GPU1 GPU99 Distributed Filesystem 32 files 32 files 32 files Batch Size of 3200 One Big Model on 3 GPUs Copy of the Model on each GPU Aggregate Gradients New Model Barrier 4/38
  5. 5. ©2018 Logical Clocks AB. All Rights Reserved CNNs Distribution Facebook vs Google ImageNet Mini AI Arms Race 5/38
  6. 6. ©2018 Logical Clocks AB. All Rights Reserved ImageNet [Image from https://www.slideshare.net/xavigiro/image-classification-on-imagenet-d1l4-2017-upc-deep-learning-for-computer-vision ] 6/38
  7. 7. Improved Accuracy for ImageNet
  8. 8. ©2018 Logical Clocks AB. All Rights Reserved Improvements in ImageNet – Accuracy 81.5 82 82.5 83 83.5 84 84.5 GOOGLE-1 (MAY '17) GOOGLE-2 (MARCH '18) FACEBOOK (MAY '18) Top-1 Accuracy for ImageNet ImageNet Top-1 AutoML with RL AutoML with GA Facebook- https://goo.gl/ERpJyr Google-1: https://goo.gl/EV7Xv1 Google-2:https://goo.gl/eidnyQ More Compute 1 Bn Images More Data 8/38
  9. 9. ©2018 Logical Clocks AB. All Rights Reserved Reduced Generalization Error Hyper Parameter Optimization Larger Training Datasets Better Regularization Methods Better Optimization Algorithms Design Better Models Methods for Improving Model Accuracy Distribution 9/38
  10. 10. Faster Training of ImageNet Single GPU [Image from https://www.matroid.com/scaledml/2018/jeff.pdf] Clustered GPUs
  11. 11. ©2018 Logical Clocks AB. All Rights Reserved Improvements in ImageNet – Training Time June’17 Oct’17 FB - https://goo.gl/ERpJyr Google-1: https://goo.gl/EV7Xv1 Google-2:https://goo.gl/eidnyQ 0 10 20 30 40 50 60 FACEBOOK (JUNE '17) GOOGLE-1 (DEC '17) GOOGLE-2 (MARCH '18) Training Time (mins) ImageNet >75% top-1 Training Time (mins) 256 Nvidia P100s 256 TPUv2s 256 TPUv2s 11/38
  12. 12. ©2018 Logical Clocks AB. All Rights Reserved 0 20 40 60 80 100 120 140 160 FACEBOOK (JUNE '17) GOOGLE-1 (DEC '17) GOOGLE-2 (MARCH '18) 1000s Files/Sec Processed Files Processed (K/s) ImageNet – Files/Sec Processed June’17 FB - https://goo.gl/ERpJyr Google-1: https://goo.gl/EV7Xv1 Google-2:https://goo.gl/eidnyQ HDFS ~80K/s 12/38
  13. 13. ©2018 Logical Clocks AB. All Rights Reserved 0 200 400 600 800 1000 1200 FACEBOOK GOOGLE-1 GOOGLE-2 HOPSFS Distributed Filesystem Read Throughput Files Processed (K/s) Remove Bottleneck and Keep Scaling HopsFS - https://goo.gl/yFCsGc Scale Challenge Winner (2017) Hops PLUG for Hops 13/38
  14. 14. ©2018 Logical Clocks AB. All Rights Reserved All AI Roads Lead to Distribution Distributed Deep Learning Hyper Parameter Optimization Distributed Training Larger Training Datasets Elastic Model Serving Parallel Experiments (Commodity) GPU Clusters Auto ML 14/38
  15. 15. THEY WILL IGNORE YOU, DISTRIBUTION UNTIL THEY NEED YOU DISTRIBUTION
  16. 16. ©2018 Logical Clocks AB. All Rights Reserved Top 10 List: Why Big Data/Distribution sucks 1. I don’t know which is scarier: reading Stephen King’s horror novel IT or reading a file from HDFS in Python 2. MapReduce – how can a programming model with only 2 parts have O(N2) complexity? 3. I managed to install Hadoop – at the cost of my last relationship…the Spark died! 4. “Jupyter’s down again” - some service you never heard of stopped it working…. 5. I still don’t know which is worse - a dead service or a dead slow service. 16/38
  17. 17. ©2018 Logical Clocks AB. All Rights Reserved Top 10 List: why Big Data/Distribution sucks 6. How do I debug this thing on my laptop? 7. Enough with the Dockerfiles and YAML files, when can I start writing *real* code? 8. Isn’t it supposed to be faster when I add a new server? 9. Do I really have to install the library on *all* of the servers? 10.Distributed? I don’t even do multi-threaded! 17/38
  18. 18. KubeFlow GPU/TPU Services in the Cloud and On-Premise •Managed Cloud Platforms -RiseML -FloydHub -Google Cloud ML -Microsoft Batch AI -AWS Sagemaker •On-Premise/Cloud Platforms -KubeFlow -Hops 18/38
  19. 19. ©2018 Logical Clocks AB. All Rights Reserved Hops Open-Source Data Science Platform 19/38
  20. 20. ©2018 Logical Clocks AB. All Rights Reserved PySpark End-to-End ML Pipeline in Python on Hops 20/38 Data Collection Experimentation Training Serving Feature Extraction Data Transformation & Verification Test TfServingTensorFlow
  21. 21. FLIP-O-RAMA with Python and HDFS/Hops Left Hand Here Right Thumb Here Data Collection Experimentation Training Serving Feature Extraction Data Transformation & Verification Test
  22. 22. import hops.hdfs as hdfs features = ["Age", "Occupation", "Sex", …, "Country"] with open(hdfs.project_path() + "/TestJob/data/census/adult.data", "r") as trainFile: train_data=pd.read_csv(trainFile, names=features, sep=r's*,s*', engine='python', na_values="?") Data Ingestion with Pandas Right Thumb Here
  23. 23. import hops.hdfs as hdfs features = ["Age", "Occupation", "Sex", …, "Country"] h = hdfs.get_fs() with h.open_file(hdfs.project_path() + "/TestJob/data/census/adult.data", "r") as trainFile: train_data=pd.read_csv(trainFile, names=features, sep=r's*,s*', engine='python', na_values="?") Data Ingestion with Pandas on Hops Right Thumb Here
  24. 24. Google Facets 24/38 Data Collection Experimentation Training Serving Feature Extraction Data Transformation & Verification Test
  25. 25. train_data = pd.read_csv(…) test_data = pd.read_csv(…) facets.overview(train_data, test_data) facets.dive(test_data.to_json(orient='records')) Google Facets 25/38
  26. 26. Big Data Preparation with PySpark images = spark.readImages(IMAGE_PATH, recursive = True, numPartitions=10, sampleRatio = 0.1).cache() tr = (ImageTransformer().setOutputCol(“transformed”) .resize(height = 200, width = 200) .crop(0, 0, height = 180, width = 180) ) smallImages = tr.transform(images).select(“transformed”) dfutil.saveAsTFRecords(smallImages, OUTPUT_DIR) 26/38
  27. 27. Experimentation and Training Data Collection Experimentation Training Serving Feature Extraction Data Transformation & Verification Test
  28. 28. GridSearch with TensorFlow/Spark on Hops def train(learning_rate, dropout): [TensorFlow Code here] args_dict = {'learning_rate': [0.001, 0.005, 0.01], 'dropout': [0.5, 0.6]} experiment.launch(spark, train, args_dict) Launch 6 Spark Executors 28/38
  29. 29. Model Architecture Search on TensorFlow/Hops def train_cifar10(learning_rate, dropout, num_layers): [TensorFlow Code here] dict = {'learning_rate': [0.005, 0.00005], 'dropout': [0.01, 0.99], 'num_layers': [1,3]} experiment.evolutionary_search(spark, train, dict, direction='max', popsize=10, generations=3, crossover=0.7, mutation=0.5) 29/38 Launch 10 Spark Executors
  30. 30. ©2018 Logical Clocks AB. All Rights Reserved Differential Evolution in Tensorboard (1/4) CIFAR-10 30/38
  31. 31. ©2018 Logical Clocks AB. All Rights Reserved Differential Evolution in Tensorboard (2/4) CIFAR-10 31/38
  32. 32. ©2018 Logical Clocks AB. All Rights Reserved Differential Evolution in Tensorboard (3/4) CIFAR-10 32/38
  33. 33. ©2018 Logical Clocks AB. All Rights Reserved Differential Evolution in Tensorboard (4/4) CIFAR-10 33/38
  34. 34. Distributed Training with Horovod hvd.init() opt = hvd.DistributedOptimizer(opt) if hvd.local_rank()==0: [TensorFlow Code here] ….. else: [TensorFlow Code here] ….. 34/38
  35. 35. ©2018 Logical Clocks AB. All Rights Reserved Model Serving: Monitor + Retrain 35/38
  36. 36. Summary •The future of Deep Learning is Distributed https://www.oreilly.com/ideas/distributed-tensorflow •Hops is a new Data Platform with first-class support for Python / Deep Learning / ML / Data Governance / GPUs *https://twitter.com/karpathy/status/972701240017633281 “It is starting to look like deep learning workflows of the future feature autotuned architectures running with autotuned compute schedules across arbitrary backends.” Andrej Karpathy - Head of AI @ Tesla 36/38
  37. 37. ©2018 Logical Clocks AB. All Rights Reserved The Team Jim Dowling, Seif Haridi, Gautier Berthou, Salman Niazi, Mahmoud Ismail, Theofilos Kakantousis, Ermias Gebremeskel, Antonios Kouzoupis, Alex Ormenisan, Fabio Buso, Robin Andersson, August Bonds. Active: Alumni: Vasileios Giannokostas, Johan Svedlund Nordström,Rizvi Hasan, Paul Mälzer, Bram Leenders, Juan Roca, Misganu Dessalegn, K “Sri” Srijeyanthan, Jude D’Souza, Alberto Lorente, Andre Moré, Ali Gholami, Davis Jaunzems, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Steffen Grohsschmiedt, Qi Qi, Gayana Chandrasekara, Nikolaos Stanogias, Daniel Bali, Ioannis Kerkinos, Peter Buechler, Pushparaj Motamari, Hamid Afzali, Wasif Malik, Lalith Suresh, Mariano Valles, Ying Lieu, Fanti Machmount Al Samisti, Braulio Grana, Adam Alpire, Zahin Azher Rashid, ArunaKumari Yedurupaka, Tobias Johansson , Roberto Bampi, Roshan Sedar. www.hops.io @hopshadoop

×