Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Ai powered images-mobileera

132 Aufrufe

Veröffentlicht am

Presented at Mobile Era, Oslo, Norway November 8, 2019

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Ai powered images-mobileera

  1. 1. Using AI to Power Your Images Doug Sillars Mobile Era November 8, 2019
  2. 2. Contact Me: @DougSillars Doug.Sillars@gmail.com www.dougsillars.com Doug Sillars Freelance Developer Relations Performance Audits: Web/Native Workshops: Performance/Images/Video/AR/ML http://bit.ly/HighPerformanceAndroidApps
  3. 3. Images Dominate the Web HTTPArchive mobile websites 04/2019
  4. 4. Image Prep for the Web
  5. 5. Image Prep for the Web
  6. 6. Image Prep for the Web
  7. 7. Image Prep for the Web
  8. 8. Image Prep for the Web
  9. 9. Shortcuts for Image Prep?
  10. 10. Shortcuts for Image Prep? Ensure You are delivering the optimal image
  11. 11. Shortcuts: Taking screenshots of Photoshop
  12. 12. Shortcuts for Image Prep?
  13. 13. Images on the web named screen*shot 426,000 screenshots live on the web April 2019
  14. 14. Images on the web named screen*shot 106,000 sites with Screen Shots April 2019
  15. 15. Images on the web named screen*shot 5M sites tested 106,000 sites with Screen Shots
  16. 16. Images on the web named screen*shot 5M sites tested 106,000 sites with Screen Shots 2.1% of the web! =
  17. 17. Image Formats – Screenshots -> JPG https://dougsillars.com/2019/02/10/using-screenshots-in-production/ Sometimes our Shortcuts have lasting effects
  18. 18. Shortcuts for Image Prep?
  19. 19. Shortcuts for Image Prep?
  20. 20. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps _tfzado.jpg
  21. 21. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps _tfzado.jpg 120 KB
  22. 22. Shortcuts for Image Prep?
  23. 23. Shortcuts for Image Prep? https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1080,w_1000,q_auto,f_auto/ v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg
  24. 24. Shortcuts for Image Prep Manual crop
  25. 25. Shortcuts for Image Prep Manual crops are not a shortcut
  26. 26. Shortcuts for Image Prep: Detection of Objects https://res.cloudinary.com/dougsillars/image/upload/g_auto,c_crop,w_1000,h_1000/ v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg
  27. 27. Image Cropping in Production: Twitter
  28. 28. Image Cropping in Production: Twitter https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html
  29. 29. Object Detection Using Machine Learning • ImageAI library • Tensorflow based • Need Image Model Library detector = ObjectDetection() detector.setModelTypeAsRetinaNet() detector.setModelPath( os.path.join(execution_path , "resnet50_coco_best_v2.0.1.h5")) detector.loadModel() detections = detector.detectObjectsFromImage(input_image=os.path.join(execution_path , "berwickontweed.jpg"), output_image_path=os.path.join(execution_path , "resnet.jpg")) resnet=plt.imread('resnet.jpg')
  30. 30. Object Detection Using Machine Learning
  31. 31. Object Detection Using Machine Learning
  32. 32. Object Detection with Machine Learning
  33. 33. Object Detection • Cropping
  34. 34. Object Detection • Cropping • Blurring objects
  35. 35. Object Detection • Cropping • Blurring objects • Object Removal http://bit.ly/bootsphoto
  36. 36. Object Detection • Cropping • Blurring objects • Object Removal • Alt Text Creation
  37. 37. Object Detection • Cropping • Blurring objects • Object Removal • Alt Text Creation Alt = “A Person and 2 dogs”
  38. 38. If the Object is not Modeled…
  39. 39. If the Object is not Modeled… •person : 99.72 •dog : 88.67 •dog : 96.92
  40. 40. …Try a new library • https://llama.vision
  41. 41. …Try a new library • https://llama.vision
  42. 42. ML ALT text in production
  43. 43. ML ALT text in production alt="Image may contain: bridge, sky, tree, outdoor, water and nature"
  44. 44. ML ALT text in production alt="Image may contain: dog"
  45. 45. …Or you can train a model
  46. 46. Object Detection Using Machine Learning
  47. 47. Google Object Detection
  48. 48. Google Object Detection: Model Creation
  49. 49. Google Object Detection: Model Creation
  50. 50. Google Object Detection: Model Creation
  51. 51. Google Object Detection: Model Creation
  52. 52. Train Your Model Precision: How often Model’s predictions were correct. In this case 1/6 is a false positive Recall: How often model found known objects. 1/6 is a missed prediction.
  53. 53. Deploy Model and Use
  54. 54. Deploy Model and Use
  55. 55. Model Results!!
  56. 56. Model Results!!
  57. 57. Image Fix 1: Cropping
  58. 58. Image Fix 1: Cropping
  59. 59. Image Fix 1: Cropping
  60. 60. Image Fix 2: Inpainting
  61. 61. Image Fix 2: Inpainting Create a mask of the area
  62. 62. Image Fix 2: Inpainting Create a mask of the area
  63. 63. Image Fix 2: Inpainting
  64. 64. Image Fix 2: Inpainting
  65. 65. InPaint: Works well on single color areas
  66. 66. InPaint: Works Not So well with lots of color
  67. 67. Tricking InPaint Remove “outlier” colors from the border Use the inside as inpainting area
  68. 68. Tricking InPaint Convert to Grayscale colors are 0 – 255 (1D) Kmeans to bucket the colors
  69. 69. Tricking InPaint
  70. 70. Tricking InPaint 10 most common gray colors found
  71. 71. Tricking InPaint 10 most common gray colors found Frequency
  72. 72. Tricking InPaint 1. Loop through Color list
  73. 73. Tricking InPaint 2. Ignore the White Center
  74. 74. Tricking InPaint 3. If Color is an outlier (lower than 25th percentile Higher than 75th) Mask and replace
  75. 75. Tricking InPaint
  76. 76. Tricking InPaint
  77. 77. Tricking InPaint
  78. 78. Sunglasses Project
  79. 79. Sunglasses Project
  80. 80. Manually edit 1,000s of glasses?
  81. 81. Sunglasses Project
  82. 82. Sunglasses Project
  83. 83. Sunglasses Project
  84. 84. Sunglasses Project 1. Remove Background
  85. 85. Sunglasses Project 1. Remove Background
  86. 86. Sunglasses Project 2. Make (slightly) transparent https://res.cloudinary.com/dougsillars/image /upload/e_make_transparent,o_70/v1558760 129/6658.png
  87. 87. Train a model!
  88. 88. Want to minimize frame capture
  89. 89. Train a model!
  90. 90. Beginning
  91. 91. Find the arms
  92. 92. Find the arms Finding an Odd number: Fail
  93. 93. Modify the border around the arms
  94. 94. Inpaint and upload
  95. 95. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/v1558855691/IMG_20181207_163533_ibfm7a.jpg
  96. 96. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120 7_163533_ibfm7a.jpg
  97. 97. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/v1558855691/IMG_2018120 7_163533_ibfm7a.jpg l_6558_f,o_80,e_make_transparent/
  98. 98. Putting it all together https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/ g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg
  99. 99. Demo Time https://res.cloudinary.com/dougsillars/image/upload/c_crop,g_face,w_1500,h_1500/l_6558_f,o_80,e_make_transparent/ g_adv_eyes,fl_layer_apply/v1558855691/IMG_20181207_163533_ibfm7a.jpg
  100. 100. Conclusion • Image processing with AI and ML can simplify image prep for the web • Cropping • Blurring • Object removal • Alt text
  101. 101. Love Building with Video and Images? Media Developer Expert mde-comm@cloudinary.com Become a

×