I/O Extendedで話した内容です。

1. Google I/O 2018
ML model + IoT Demo
Inside Story
2018/05/19
Satoru Nakamura
Groovenauts, Inc.

2. Who am I
● Satoru Nakamura
● GDE(Cloud)
●

3. QueryIt Smart
(Google Cloud Next ‘17 Playground Demo)
https://queryit.magellanic-clouds.com/

4. Device Location Detection
(Google Cloud Next ‘17 in Tokyo
Groovenauts’ presentation)

5. Outline of session demo

6. Google I/O 2018 Session
“ML models + IoT data
= a smarter world”
https://www.youtube.com/watch?v=avxpkFUXIfA

7. Regulations of this technical demo
● Adopt Cloud IoT Core
● Utilize ML model & Cloud ML Engine (training and inference)
● ML model inference on edge device is out of scope
(It’s included in Laurence’s part of the session)

8. Smart Shopping Navigator

9. Demo dashboard
Camera Image +
Object Detection Results
Recommended recipe
according to cart contents+α
Information about the foodstuffs
required for the recipe

10. Smart Shopping Navigator

11. Demo development timeline

12. 3/27
initial offer from
kaz-san
5/9
Google I/O Session

13. 3/27
initial offer from
kaz-san
〜4/5
Draw rough blueprint
of demo
〜4/27 5/9
Google I/O Session
Device (Raspberry Pi)
software and ML model
development deadline.

14. 3/27
initial offer from
kaz-san
〜4/5
Draw rough blueprint
of demo
〜4/13 4/27
Pivot!!
5/9
Google I/O Session
Device (Raspberry Pi)
software and ML model
development deadline.

15. 3/27
initial offer from
kaz-san
〜4/5
Draw rough blueprint
of demo
〜4/13
Pivot!!
4/27
Device (Raspberry Pi)
software and ML model
development deadline.
5/9
Google I/O Session
GW
(Japanese
Holidays)
kaz-san’s
departures to US

16. In fact only 2 weeks are available for
development
● Choose and purchase hardware setup (Raspberry Pi, camera module,
touch display etc...)
● Purchase imitation foodstuff (onion, tomato, potato, etc...)
● Device setup & demo application development
● Capture images for ML model training (Object Detection)
● Thinking of recipes
● Collect images for each recipe
● Generate dummy foodstuff sequence of shopper’s pickup.
● Training ML models (Object Detection & Next item prediction)
● Dashboard application

18. Technical Internal

19. Smart Shopping Navigator

20. Cloud IoT Core + ML Engine (TensorFlow)
Raspberry Pi
Camera
Touch
display
● upload telemetry (camera image)
● download configuration
(including recipe data)
● Object Detaction
● Next shopping item prediction

21. System Architecture
Raspberry Pi
dashboard
- Object Detection
- Cart Items Prediction
(Ingest Object Detection results)
touch display

22. ML models
● Based on Tensorflow Object Detection API
https://github.com/tensorflow/models/tree/master/research/object_detection
○ SSD + mobilenet v1
○ Transfer learning (upon COCO-trained model)
● Next shopping item prediction
○ 1D-Conv + MLP model
ML models are trained using

23. Smart Shopping Navigator

24. Smart Shopping Navigator

25. Smart Shopping Navigator

26. Smart Shopping Navigator

27. Difficulties
(except schedule)

28. ML inference should be executed on cloud
● We should think of reasons justify cloud inference
○ Edge inference was optimum in most scinario

29. The demo should be attractive and easy to
understand at glance
● “Small sensor device” is typical and realistic IoT system scenario
○ Temperature, Air pressure, Air dust etc..
● Attractive visualization is hard to develop
● Image recognition is intuitive and easy to visualize!
(But is it IoT thing?)

30. All images should be licensed under CC0
https://github.com/groovenauts/SmartShoppingNavigator/issues/11
● Collecting CC0 or Public Domain attractive images is time consuming stuff
● Deeply thanks to pixabay (https://pixabay.com/)

31. Recipes should be those which
US people are familiar with
● Culture gap?
● There are less typical cuisine with common name (really?)
● Cookpad (https://cookpad.com/us) show variety of recipes
But they are like “Bacon and potato”, “○○’s speciality”...

32. Smart Shopping Navigator
https://github.com/groovenauts/SmartShoppingNavigator/