This is presentation slide for OpenStack Summit Austin 2016. // Abstraction Internet of Things is a hot topic today. Many companies are trying to create new business applications on a concept of IoT such as smart city, connected vehicle or smart grid. The platform for IoT applications has some unprecedented characteristics: (1) needs to accept huge number of connections simultaneously (2) needs to be highly reliable and secure (3) needs to be highly scalable. We have designed, prototyped and evaluated a highly reliable IoT platform for collecting and storing large-scale data. We explained our use case and architecture of IoT platform. We are tackling the following very high requirements during the process of prototyping and evaluating the platform. • Receiving and storing messages from over 10M clients concurrently • Highly reliable architecture of message broking without losing messages • Instant scale-out to process burst traffic rapidly We also discussed how we can adopt OpenStack to IoT backend and share ideas for its enhancement.

1. Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
April 28, 2016
Hiroshi Miura
Naoto Umemori
Yuji Hagiwara
Cloud Platform for IoT:
Designing and Evaluating Large-Scale Data Collecting
and Storing Platform
Any product name, service name, software name and other marks are trade mark or registered mark of corresponding companies.
記載されている会社名、商品名、又はサービス名は、各社の登録商標又は商標です。

2. 1Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
1. Any product name, service name, software name and other
marks are trade mark or registered mark of corresponding
companies.
2. This presentation is in a purpose of providing the knowledge
gained from our IoT projects.
3. A presenter and NTT DATA Corporation provide information in
as-is basis and have no responsiveness for results that you got
according to information in this presentation material.
1
Disclaimer

3. 2Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
About us
 Who are we?
• Hiroshi Miura : Manager
• Naoto Umemori : Platform Engineer
• Yuji Hagiwara : Platform Engineer
 OSS professional headquarter in NTT Data Corp.
 Our main target
• Cloud technology (OpenStack, Sheepdog, Docker,…)
• Automation of platforms
• IoT

4. 3Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Agenda
• Overview
• IoT platform
• IoT platform on OpenStack
3

5. Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016 4
Overview

6. 5Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
IoT: Internet of things
• Everything is connected to the Internet
Internet
Computers Smart devices Consumer electronics Cars Construction Machines
IoT enables data gathering, controlling over the Internet

7. 6Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Big data analysis with IoT
Gather data from anything, and
make value through analyzing them

8. 7Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Use case #1: Sensor network

9. 8Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Use case #2: Industry 4.0

10. 9Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Our Target
Our Target
Scale (Performance, Extensibility)
Social Influence (Reliability)
• Utility (Smart meter)
• Smart factory
• Connected vehicle
• Smart grid
• Smart City
(Smart traffic management)
High performance, extensibility and reliability
Huge impact to Social influence
Limited impact to Social influence
• Sensor network
• Mobile device management
• Vendor machine

11. 10Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
4 Key elements for IoT service
Business APs
Mobile
Sensor
Server
Car
W
A
N
Device
Data
Receiver
Edge
Computing
Map info.
Traffic info.
User info.
Stock info.
Platform Application
…
Security
Data
Lake
Data Analysis
Device + Network + Platform + Application = IoT service
Network
「IoT platform」

12. 11Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Our motivation for IoT platform
Business APs
Mobile
Sensor
Server
Car
W
A
N
Device
Data
Receiver
Edge
Computing
Map info.
Traffic info.
User info.
Stock info.
Platform Application
…
Security
Data
Lake
Data Analysis
「IoT platform」
Network
Key performance measure are
“Connection”, “Transaction” and “Accumulation”
1. Million concurrent
Connections
3. 100PB+ /month
2. 10K+ TPS

13. 12Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
The goal
Full-stacked IoT platform
on the OpenStack
Analysis
Data
Application
Middleware
Operating System
Server
Storage
Network
Rack
Data Center

14. Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016 13
IoT platform

15. 14Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
3 Key features of IoT data
Variety
100+ data formats
100+ analytic method
Volume 100+ Peta bytes
Velocity
10G+ BPS
50K+ TPS
1M+ Connections

16. 15Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Our goal
Business APs
Mobile
Sensor
Server
Car
W
A
N
Device
Data
Receiver
Edge
Computing
Map info.
Traffic info.
User info.
Stock info.
Platform Application
…
Security
Data
Lake
Data Analysis
「IoT platform」
Network
Goal: 1M Conn.
Goal: 50K TPS
Goal: 5GB/s
Key performance measure are
“Connections”, “Transactions” and “Accumulation”
1. Million concurrent
Connections
3. 100PB+ /month
2. 50K+ TPS

17. 16Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Testing environment
Developed and evaluated IoT platform on bare metal and cloud test bed
– Bare metal test bed: Hardware bench marking on physical servers
– Cloud test bed: Scale bench marking on a public cloud
Our Target
# of Nodes：18
# of Nodes：21
42
nodes
Minimum Physical Env.
Minimum Cloud Env.
Scale testing
Hardware sizing
x2
63
nodes
x3
Devices
Each nodes should be scalable
Testing Environment
1. Million concurrent
Connections
3. 100PB+ /month
2. 50K+ TPS
Goal: 1M Conn.
Goal: 50K TPS
Goal: 5GB/s

18. 17Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Cloud instances
Hardware devices
Devices # Categories Features Remarks
Servers 16 Server / Processor(s) HP PROLIANT DL360 GEN9 8C E5-2640V3 2.60GHz 2P/16C -
Memory 64GB 8GB * 8
Hard drives 300GB 15K SC2.5”12G SAS HDD (10 Disks) -
NIC • 1GBE (4 ports)
• 10GBE SFP+ (2 ports)
-
2 Server DELL PowerEdge R610 -
Processor(s) Intel Xeon E5530 @ 2.40GHz 2P/16C -
Memory 32GB -
Hard drives 1TB (500GB * 3Disks) Raid 5
NIC • 1GBE (4 ports)
• 10GBE SFP+ (2 ports)
-
L3SW 1 L3 Switch HP 5900AF-48XG-4QSFP+ Switch 10GBE, 48ports
L2SW 1 L2 Switch Cisco WS-C2960X-48TD-L SFP+ * 2ports
1 L2 Switch Arista 7050S-52 10GBE, 52ports
2 L2 Switch Cisco Catalyst 2970 -
Flavor # of instances vCPU Memory(GiB) I/O Network
performance
NW I/F Remarks
c4.8xlarge 21+ 36 60.0 Very high 10GBE 8 # of instances depends on
test cases
Testing environment

19. 18Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Elements Components Features
Data Receiver Connection
Gateway
•Accept huge number of connections
•Load balancing or distribution
•Multiple Protocol support (ex. MQTT, HTTP/2, CoAP, etc.)
Broker •Pub/Sub support
•Message Broking
•Persistence
•Data replication
Receiver •Broker Subscriber
•Cache data updater
Data Lake Cache (KVS) •Distributed cache
•Data replication
•Persistence
Converter &
Accumulator
•Data accumulation
Data store •Data storage (ex. HDFS, Swift, etc.)
Testing environment

20. 19Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Next step for us
IoT is just an use case on OpenStack?

21. 20Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Technical requests for OpenStack from our experience
QoS, I/O guaranteed
Feature of Message Broking
Management Integration
Application performance monitoring
Easy to manage data stores and ETL
1
2
3
4
5

22. 21Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Technical requests for OpenStack
1. QoS, I/O guaranteed
Network Storage
10G+ BPS 1K+ IOPS

23. 22Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Technical requests for OpenStack
2. Feature of Message Broking
Message Broker:
• Pub/Sub model (Distributed processing)
• Asynchronous I/O
• Retrying tasks if workers fail
Support multiple protocol for IoT:
• MQTT
• Kafka
• CoAP
• HTTP/2
QoS Control:
• Bandwidth
• Priority
Broker
ConnectionGateway
DataStore
Worker 1
Worker 2
Worker N
QoS
Data
source
Data
source
Internet
• MQTT
• Kafka
• CoAP
• HTTP/2 : A message from Data source

24. 23Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Technical requests for OpenStack
3. Management integration
Management integration includes:
・ Deploy ・ Automation ・ Authentication ・Monitoring
Platform Management
Mobile
Sensor
Server
Car
W
A
N
Device
OpenStack
Data
Receiver
Edge
Computing
Platform
Data
Lake
Data Analysis
Network
Device and Edge Management
Management

25. 24Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Technical requests for OpenStack
4. Application performance monitoring
Data Gathering and Visualizing for AP monitoring:
Devices
Data Receiver Data Lake
AP AP AP AP AP AP
Data Gathering
Data Visualizing
AP1 AP2 AP3 AP4 AP5
(ex.1) TPS
AP1 AP2 AP3 AP4 AP5
(ex.2) Queuing depth
Data Analysis AP AP APAP AP

26. 25Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Technical requests for OpenStack
5. Easy to manage data stores and ETL
Easy to manage various data stores
• RDB: mysql, PostgreSQL, …
• NoSQL: mongoDB, CouchDB, …
• KVS(cache): Redis, Couchbase, Cassandra, …
Easy to manage various data extract, transform and load
• (ex.1) KVS to NoSQL
• (ex.2) NoSQL to Object storage

27. 27Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Cloud Authentication
/ Authorization
IoT architecture in NTT DATA
Scalable cloud infrastructure
Cloud Monitoring
Device Authentication
/ Authorization
Gateway
Data
source
Data
source
Data
source
Gateway
Data
source
Data
source
Data
source
Broker Cache
RDB
NoSQL
Object
Storage
Data
Processing
(BT, RT)
Adaptor
Receiver
Converter&
Accumulator
L3Router,VPN
Device Monitoring
PaaS / Container
Application Application Application Application Application Application
Internet

28. 28Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
IoT architecture v2 in NTT DATA
Scalable cloud infrastructure
Edge
Gateway
Data
source
Data
source
Data
source
Edge
Gateway
Data
source
Data
source
Data
source
Broker
Cache
(KVS)
RDB
NoSQL
Object
Storage
Data
Processing
(BT, RT)
Adaptor
Receiver
Converter&
Accumulator
ConnectionGateway
PaaS / Container
Application Application Application Application Application Application
Cloud Authentication
/ Authorization
Cloud Monitoring
Device Authentication
/ Authorization
Device Monitoring
Edge Authentication
/ Authorization
Edge Monitoring
Application Application
New Component
Internet
ETL
ETL

29. Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016 29
IoT platform on OpenStack

30. 30Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
PaaS / Container
Application Application Application Application Application ApplicationApplication Application
IoT architecture v2 on OpenStack
Scalable cloud infrastructure
Edge
Gateway
Data
source
Data
source
Data
source
Edge
Gateway
Data
source
Data
source
Data
source
Broker
Cache
(KVS)
Data
Processing
(BT, RT)
Adaptor
Receiver
Converter&Accumulator
ConnectionGateway
Cloud Authentication
/ Authorization
Cloud Monitoring
Device Authentication
/ Authorization
Device Monitoring
Edge Authentication
/ Authorization
Edge Monitoring
Internet
Mature Testing Dev
Trove
Neutron
Ceilometer
Keystone
Neutron
Neutron
Heat, Magnum
Nova, Ironic
RDB
NoSQL
Object
Storage
ETL
ETL
Trove
TroveZaqar
Sahara
Swift

31. 31Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
To-do list for OpenStack
Private flavors with I/O guaranteed
Message Broker Support
Edge/Device Management
Performance Monitoring
ETLaaS: ETL as a Service
1
2
3
4
5

32. 32Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
To-do list for OpenStack
1. Private flavors with I/O guaranteed
To receive massive data require I/O guaranteed:
Network Storage
10G+ BPS 1K+ IOPS

33. 33Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
To-do list for OpenStack
2. Message Broker Support
Message Broker product
• Apache kafka
• Mosquitto
• IBM MessageSight
• RabitMQ
Data Streaming product
• Storm
• fluentd
• flume
• etc.

34. 34Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Cloud platform for IoT
To-do list for OpenStack
3. Edge/Device Management
Edge Gateway is used for Edge Computing
• PaaS / Container pushes an Application to Edge Gateway Management System
for message broking and processing on Edge Gateway
• Edge Gateway Management is needed to realize an pluggable data processing
Internet
Edge GatewayData
source
Data
source
Data
source
PaaS / Container
Edge Gateway Management System
Broker
Cache
Data
Processing
Immutable Infrastructure
GW info. Applications
Edge Monitoring
Edge Authentication / Authorization
Application
(for Broker and Data Processing)
Mobile Carrier
API

35. 35Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
To-do list for OpenStack
4. Performance Monitoring
Performance Monitor resources:
• Applications
• Virtual Machines
• Network
• Devices
Any integrated solutions?
• Horizon + Ceilometer + Monasca
• Elasticsearch + fluentd + Kibana + etc.

36. 36Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
To-do list for OpenStack
5. ETLaaS: ETL as a Service
Common ETL I/F for Data stores:
• Data Extract
• Data Transform
• Data Load
RDB
NoSQL
(KVS)
Object
Storage
Column
Storage
Block
Storage
ETL
ETL RDB
NoSQL
(KVS)
Object
Storage
Column
Storage
Block
Storage
ETL
I/F

37. Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016 37
Summary

38. 38Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016
Summary
• IoT technologies enable us to gather huge data over
the internet, and create new value with data
analysis.
• We updated our IoT architecture, and summarized
to-do list to make IoT platform on OpenStack.
IoT powered by OpenStack!

39. 39Copyright © 2016 NTT DATA Corporation OpenStack Summit | Austin 2016Copyright © 2016 NTT DATA Corporation
Any product name, service name, software name and other marks are trade mark or registered mark of
corresponding companies.
記載されている会社名、商品名、又はサービス名は、各社の登録商標又は商標です。