review of factors affecting IoT system selection. for MVP phase and later phases. Computation, price, connectivity, open source support, development SDKs

1. Design considerations for IoT systems
Yossi Cohen
December 2020

2. Outline
• IoT Modern needs
• Deep Learning networks and operations
• CPU GPU TPU and NN
• Considerations
• Dev-boards and NPU accelerators
– RPI
– NVIDIA
– Rockchip
– Intel
• Cameras 2D / 3D
• Summary

3. IOT NEEDS

4. Basic Edge Processor operation
• Worked as controller
• Control motors / open gates
• Take pictures / basic image operations
• Send images to the cloud for further
processing/storage/decision
• Operate based on cloud decision

5. IoT Modern roles
• Operations done in IoT device
– Read numbers, license plates, road signs
– Detect faces (dogs, cats)
• Count people
• Segment cars, pedestrians
• Scan environment / Navigate
• Drive cars / Fly drones

6. Neural Networks
• Basic idea: Multiply each input by weights
and sum
• Exists for 60+ years !!
– Implemented with resistors as weights
• Deep Neural networks 100+ layers
– Hundreds of millions or more weights
– Heavy matrix operations

7. LSTM/RNN
• Used for
– sound detectors (gun shot detector)
– data series - number of clients in restaurant
– Activity detector – based on accelerometers
– Usually requires low processing compared to
CNNs

8. CNN - Classifiers
• Used for
– Is object X in the image
– Is there a cat in the Image ?
– Common implementation LaNet

9. BOX Detectors
• Box detectors detects where is the object
located
• Used to detect:
– People, face, corona mask
– car, road signs
– cats
• Known box detectors:
– YOLO3 YOLO5
– SSD

10. Segmenter and Multi-Instance Segmenter
• Segmentor
– defines class for each pixel
– Pixel(X,Y) belongs to cat
– FCN-8 basic implementation
– Does not separate cats
• Multi-instance segmentor
– Separate Cats
– Pixel(X,Y) belongs to cat42
– MRCNN – common implementation

11. CPU and CNNs
• X86, ARM – sequential instruction
processing
• Not suitable for matrix (DL) operations !
• Remedies
• +SIMD – MMX SSE
• +Multicore 2,4,8….
• Yet….

12. Moore law is dead??

13. GPU – more than a graphic processor
• GPU are using 100s-1000s of (simple)
cores
• Better in parallel processing than CPU
• Much better for large matrix operations
13

14. GPU Examples
• GPU Cards
– NVIDIA 20XX
– NVIDIA T4
• Can we run a GPU only IoT solution?

15. TPUs
• TPUs are built ONLT for deep learning
• Designed for single operation layer
processing
• First GPUs started 4 years ago
• Google Coral, Hauwei, NVIDIA, Intel
Movidius, Narvana,
• In Israel: Habana, Heilo
• In most modern phones: Samsung apple,
QCOM

16. TPU
All internal buses are 256B wide

17. Processor selection is obvious right?
• TPU is the best solution for Deep learning
activities

18. DEV BOARDS AND
ACCELERATORS

19. Edge TPU accelerators
• Intel Movidius Processor
– Edge AI processor / accelerator
– Either chip or USB
– Connects to RPI
• Google TPU (Coral)
– Chip or USB
• Rockchip RK 1808

20. Processors today are SoC
• SoC which usually include:
– 4-8 ARM cores
– GPU (ARM Mali, NVIDIA XX, Intel XX)
– Sometimes a TPU
• The selection is CPU/GPU/TPU
• Selection of solution is between SoC
• For MVP or development phases selection is
between SoC development boards !!

21. Considerations
• Price - $$
• Performance TFlops/Sec
• Power - Watts
• Longevity
• Existing code base
• Development boards
• Connectivity
• SDKs

22. Price
• 10-20$ Arduino
• 35-75$ RPI3 / RPI4 + extended memory
• 59-100$ Jetson nano 2-4GB
• 100$ Intel Movidius accelerator
• 115$ Google Coral dev board
• 175$ Rockchip 3399 Dev board
• 200$ Jetson TX2
• 400$ Jetson NX
• 700$ Jetson XVG

23. Performance
• Performance ratio between Jetson
processors

24. Operating systems
• Usually, ubuntu running on most devices
• Android on some boards
• Raspbian / Raspberry Pi OS for RPI

25. Longevity
• Important to check the processor lifespan
• For example, in NVIDIA jetson processors

26. RPI3 / 4 dev boards
• 35-75$ depending on GB
• Most common dev board
• Control/GPIO/basic image processing
• ~3-6 fps of TinyYOLO

27. NVIDIA Jetson nano
• 59-100$ dev board
• Relatively strong GPU
• 25 fps TinyYOLO
Model Application Framework
NVIDIA Jetson
Nano Raspberry Pi 3
Tiny YOLO V3
(416×416)
Object Detection Darknet 25 FPS 0.5 FPS
OpenPose
(256×256)
Pose Estimation Caffe 14 FPS DNR
VGG-19
(224×224)
Classification MXNet 10 FPS 0.5 FPS
Super Resolution
(481×321)
Image Processing PyTorch 15 FPS DNR
Unet
(1x512x512)
Segmentation Caffe 18 FPS DNR
Source: https://developer.nvidia.com/embedded/jetson-nano-dl-inference-benchmarks

28. Rockchip 3399 dev board
• 175 – 299$ depending on board
• USB2/3
• PCIe connectivity
• GPIO
• TPU accelerated

29. Intel Atom
• UP – boards
• 200-700$ depending on kit including
RealSense for some options
• Use OpenVino SDK for development and
optimization

30. Jetson NX
• 399$ dev board
• X4-5 stronger than the nano
• Same RPI-like form factor
and connectivity
• emmc + PCIe
• Can do:

31. Software / open source
• RPI has many open-source solutions
• NVIDIA uses DeepStream and ISAAC SDKs
to accelerate development
• Intel uses OpenVino to accelerate
development on its boards

32. 2D Camera
• MIPI / USB connectivity / HDMI
• Resolution / Formats YUV, JPEG, Bayer
• HDR / Pixel size
• Filters, IR Cut
• Lens quality, FOV, distortion
• ISP quality – very important for DL
• Not all cameras supported by all boards
• USB camera – UVC no need for driver on
Linux windows

33. Placing AI in the camera
• Many IoT solutions require depth camera
/ Lidar - Lots of processing
• To reduce processor cost done inside the
camera:
– Intel Realsense family
– Rockchip RMSL 2031000
– Veronica - Inutive

34. RealSense
• D455/D435/D415 cameras
– IR + RGB + Projector + dual cam
• Option for IMU for robots/drones
• On board processor
• Not fit for RPI in high-res(high data rate)

35. Inutive - Veronica
• Designed for robots drones AR
• Low latency to avoid AR motion sickness
• On chip SLAM
• Built in VPU
• Output depth map

36. ROCKchip RMSL203
• Rockchip depth camera solution
• Built in VPU/DSP
• Output depthmap
• Projector + IR + RGB

37. Placing AI in the camera
• Many IoT solutions require depth camera
/ Lidar - Lots of processing
• To reduce processor cost done inside the
camera:
– Intel Realsense family
– Rockchip RMSL 2031000
– Veronica - Inutive

38. RealSense
• D455/D435/D415 cameras
– IR + RGB + Projector + dual cam
• Option for IMU for robots/drones
• On board processor
• Not fit for RPI in high-res(high data rate)

39. Inutive - Veronica
• Designed for robots drones AR
• Low latency to avoid AR motion sickness
• On chip SLAM
• Built in VPU
• Output depth map

40. ROCKchip RMSL203
• Rockchip depth camera solution
• Built in VPU/DSP
• Output depthmap
• Projector + IR + RGB

41. Summary
• There isnt a wining solution
• Robotics / drones solutions differ from
video analytics
• First product MVP should prefer
development time & costs over
price/performance
• Select a product with good, well tried
SDKs and community

42. Cloud connectivity
• Why we need it?
– Heavy data/image/deep learning processing
– Data aggregation from many sensing points
– Logging
– Dashboard
– Health test
– Secure access
– OTA updates
• Both AWS and Azure has Cloud IoT solution which
seamless integrate with RPI3 and Jetson Nano

43. NVIDIA and AWS IoT (Core & Greengrass)
• Connecting to AWS IoT is easy
• Deepstream has already built in MQTT message
formatters and brokers designed for AWS
connectivity
Source: https://aws.amazon.com/blogs/iot/how-to-integrate-nvidia-deepstream-on-jetson-devices-with-aws-iot-core-and-aws-iot-
greengrass/

44. Cloud connectivity
• Why we need it?
– Heavy data/image/deep learning processing
– Data aggregation from many sensing points
– Logging
– Dashboard
– Health test
– Secure access
– OTA updates
• Both AWS and Azure has Cloud IoT solution which
seamless integrate with RPI3 Jetson products and
Intel products

45. Summary
• There isn't a wining solution
• Robotics / drones solutions differ from
video analytics
• First product MVP should prefer
development time & costs over
price/performance
• Select a product with good, well tried
SDKs and community
• Invest in a good camera / ISP

46. Contact
• Yossi Cohen CTO@DSP-IP
• +972-54-5313092 | yossi@dsp-ip.com
• What we do:
– Video Analytics
– Robotics
– Drones
– Autonomous cars
– Agritec