Dr Johan Barthelemy presented a seminar titled "Using AI and edge computing devices for traffic flow monitoring" as part of the SMART Seminar Series on 11th October 2018. More information: https://news.eis.uow.edu.au/event/using-ai-and-edge-computing-devices-for-traffic-flow-monitoring/ Keep updated with future events: http://www.uoweis.co/events/category/smart-infrastructure-facility/

1. UsingAI and edge computing
devices for traffic flow monitoring
Dr J. Barthélemy, Dr N. Verstaevel, Dr H. Forehead, Senior Prof. P. Perez,
Bilal Arshad, Dinh (Andrew) Che

2. Edge Computing with Jetson TX2
for Monitoring Flows of Pedestrian
and Vehicles
At SMART, we believe that People with good
information and good tools will make good
Decisions and change their world
2

3. The case of Liverpool
CBD is growing fast: new UoW campus, airport development,…
 What does it mean for the city and its community? What are the problems?
 Smart Cities and Suburbs Program: How can we use IoT to solve the problems?
3

4. Which problems?
Let’s ask the people! Pedestrians
- Where are they going?
- What are the most popular routes?
- What are the most congested locations?
- Impact of city activity?
Cyclists
- Which route are they taking?
- How can we improve bike usage?
Cars
- Live traffic?
4

5. Sensors locations
20 sensors
Image credit: OpenStreetMap
5

6. How about using CCTV?
6
Privacy!

7. Sensor requirements
• Mobile units + leveraging CCTV infrastructure
• Privacy is important!
– On board video analytics
– Only indicators transmitted (no raw data!)
• Real-time image processing
• LoRaWAN network
7

8. LoRaWAN: An IoT Network
Image credit: https://ttnmapper.org
• Long range
• Low power
• Low bandwidth (max 200 bytes/message)
• Open source
• Free for use by the community
8

9. History of the prototypes
9

10. The sensor
An edge computing device
Components
— NVIDIA Jetson TX2 for onboard processing
— Pycom LoPy 4 for data transmission on The Things Network
— Camera (USB webcam / existing CCTV)
10

11. The sensor
Actual state…
11

12. From the input to the dashboard
Image Acquisition
• CCTV/Webcam
Detection
• Deep Convolutional Neural Network
Tracking
• Kalman Filtering
Data transmission
• LoRaWAN/OneM2M
Dashboard/Database
Image credit: NVIDIA Corporation
Image credit: Pycom
12

13. Detection:YOLO v3
Inference: Jetson TX2Training: Titan Xp
• Fully convolutional DNN
• 106 hidden layers
• Detections at 3 scales
• 3 classes: person, bicycle, vehicles
• Pascal VOC and COCO datasets
cuDNN
FP16+ +
13

14. Detection:YOLO v3
Current training dataset
• 80.000 images from Pascal VOC, COCO
• 6 data augmentation techniques
480.000 images
14

15. Detection:YOLO v3
15

16. Tracking: Kalman Filtering
16
Prediction of O
in current frame
Position of O in previous frame
Detection D in current frame
𝐼𝑜𝑈 ≔ ≥ 𝛿
𝑥
𝑦
if D = O

17. Detection:YOLO v3
• Detecting locations of pedestrians and vehicles
• Number of objects of each type
17
VIRAT Video Dataset

18. Tracking: Kalman Filtering
• Associating IDs with the detections
• Trajectories
18
VIRAT Video Dataset

19. Final output
No image Privacy OK!
19

20. Liverpool Smart Pedestrian architecture
Fixed counters
Air quality x20
x5
x15
Mobile counters
Sensors IoT Core ApplicationsTransport
+
Forecasting
Analytics
Public
Dashboard
Liverpool IoT Core
Citizens app
20

21. ● Open source service platform for M2M interoperability
based on the oneM2M standard.
● Follows a RESTful approach GET/POST/DELETE/SUBSCRIBE
● Access control policies
● Plugins
● Can be decentralised with multiple nodes
● Support different technologies:
○ Lora
○ Zigbee
○ Sigfox
○ Bluetooth Low Energy
○ Z-wave
OM2M Eclipse
21

22. Liverpool IoT Core
MQTT TTN
AE - TheThingsNetwork
App Id …
DATA
IN_Node
METADATA
MN_Node1
MN_...
Subscriptions
HTTP Rest API
Database
Inject
HTTP GET
Publish data Collect data Manage and secure data Store data
POA
22

23. The Things Network application
AE-TheThingsNetwork
CNT- App Id
CNT - DATA
CIN – DATA_...
CNT - METADATA
CIN – METADATA_...
23

24. Visual Sensor application
MN- SensorName
CNT- DATA
CIN –...
CIN –...
CNT- DESCRIPTOR
CNT- CONFIGURATION
CIN –...
CNT- SYSTEM
CIN –...
CNT- COMMAND
CIN –...
24

25. Dashboard
25

26. Dashboard
Trajectories of the detections (inside a building)
26

27. Dashboard
Heatmap of the maximum number of the detections (inside a building)
27

28. Dashboard
28

29. Next steps: Inferring network dynamics
Image credit: OpenStreetMap Image credit: Google Maps
29

30. Next steps:Algorithm transferability
Testing in Vietnam
30

31. Conclusions
It’s only the beginning!
• VS able to perform real time object detection and tracking
• Privacy compliant… but meaningful information
• Open data for people centric approach
– citizen applications
– city and traffic planners
• IoT core scalable and interoperable
– adding new sensors: air quality, noise, …
Traffic
modelling
AnalyticsDashboardCitizens app
31

32. But it’s already successful!
Best Local Government Initiative
Document title32

33. Keep InTouch
johan@uow.edu.au
linkedin.com/company/smart-
infrastructure-facility-university-
of-wollongong
@SMART_Facility
smart.uow.edu.au
uowblogs.com/smartinfrastructureSMART Infrastructure Facility
33

34. Image credit: https://towardsdatascience.com
YOLO v3 architecture
34

35. 35
Image credit: Redmon, J. and Farhadi, A. (2018). YOLOv3: An Incremental Improvement
YOLO v3 performances

36. GPU and Edge computing
Image credit: NVIDIA Corporation
Dataset
Deep Neural Net
Trained model
deployment
Inference: Jetson
Image credit: NVIDIA Corporation
Training: Titan/DGX/Tesla
Image Acquisition
Detection
Tracking
Data transmission
Dashboard/Database
36

37. Architecture of the solution
Fixed
counters
Air quality x20
Noise level x20
x5
x15
Mobile
counters
Sensors IoT Core ApplicationsTransport
+
Private and
Public APIs Traffic
modelling
Analytics
Dashboard
Citizens app
37

38. Dashboard
Heatmap of the maximum number of detections
38

39. Dashboard
Trajectories of the detections
39