Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

How we built a tools stack for the benchmarking AI and what happened next

76 Aufrufe

Veröffentlicht am

Origin story about testers team which transformed to a team of programmers writing custom software for ML and AI testing. How we started, what mistakes we did and how we solved them.

Veröffentlicht in: Ingenieurwesen
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

How we built a tools stack for the benchmarking AI and what happened next

  1. 1. Michał Łukaszewski Software Engineer
  2. 2. 2 Source: https://software.intel.com/en-us/articles/how-to-get-started-as-a-developer-in-ai
  3. 3. 3 Google trends Source: Google Trends https://goo.gl/dFRmtp
  4. 4. 4
  5. 5. 5 OpenVINO™ - Intel® DLDT • Common API for CPU, GPU, Movidius™, FPGA • Pre-trained models • And much more! https://software.intel.com/en-us/openvino-toolkit/ Intel® Movidius™ NCS Test it yourself! https://developer.movidius.com
  6. 6. 6 Source: Wikimedia; CC BY-SA 3.0
  7. 7. 7 https://aidc.gallery.video/detail/videos/day-2:-sessions/video/5790625585001/a-deep-learning-tool-for-fast-simulation-in-high-energy-physics
  8. 8. 8 Glossary • Training – process of teaching the neural network model • Inference – act of using trained network model to work • Benchmarking (aka „testing”) – research for the conditions in which it’s the fastest
  9. 9. 10 Intel® MKL-DNN
  10. 10. 11 Frameworks 8 Models 79 Datasets 26 Batch sizes 48 Platforms 17 Configuration
  11. 11. 0 2000 4000 6000 8000 10000 12000 Apr-17 Jun-17 Aug-17 Sep-17 Nov-17 Dec-17 Feb-18 Apr-18 May-18 Jul-18 Numberofconfigurations Month inferences trainings 12 Number of configurations
  12. 12. 13 Difficulties in benchmarking DL software • Lack of documentation • Quality of DL software • Working with dev branches • Influence of software environment • Influence of hardware environment • Client’s use case scenarios
  13. 13. Good Benchmarking === Truth 14
  14. 14. 16 Two Three towers Scheduling Execution Reporting
  15. 15. 17 Stage 1
  16. 16. 18
  17. 17. 19 Stage 2
  18. 18. 20
  19. 19. 21 Stage 3
  20. 20. 22 Stage 4
  21. 21. 23 Example of error rate for specific framework
  22. 22. 24 Stage 5
  23. 23. 25 Source: wikimedia; public domain
  24. 24. 26
  25. 25. 27 Stage 6
  26. 26. 28
  27. 27. 29 Source: wikimedia, CC BY 2.0; Robert Linsdell
  28. 28. 31
  29. 29. 32 © Daniel Kummer Source: https://danielkummer.github.io/git-flow-cheatsheet/
  30. 30. 33 Source: mediawiki, CC BY-SA 3.0; Immanuel Giel
  31. 31. 34
  32. 32. 35
  33. 33. 36
  34. 34. 37 yum clean all apt-get clean autoclean apt-get autoremove -y Pro tip add RUN with commands below to your Dockerfile
  35. 35. 38
  36. 36. 0 10000 20000 30000 40000 50000 60000 70000 Apr-17 Jun-17 Aug-17 Sep-17 Nov-17 Dec-17 Feb-18 Apr-18 May-18 Jul-18 Collectedresultsnumber Months inferences training 39 Performance of benchmarking
  37. 37. 40 Case study • Install software stack • Datasets sync • Enabled in Jenkins • New configuration in Scheduler • 1200 unique configurations tested • All remotly 24h
  38. 38. 41 Source: wikimedia; CC BY-SA 3.0
  39. 39. 42 We did it • We started as team of testers • Today we’re team of researchers and developers of unique software • Tests are going themself ;)
  40. 40. https://ai.intel.com 43

×