タイトル：NTT Docomo's Challenge looking ahead the world pf 5G × OpenStack アジェンダ： - Current Challenge -- DOCOMO Cloud Platform -- BizDevOps - Challenge for the future -- DOCOMO 5G Open Cloud -- Next Challenge

1. NTT DOCOMO, INC. All Rights Reserved
NTT DOCOMO’s challenge
looking ahead
the world of 5G × OpenStack
NTT DOCOMO, Inc. Hiroaki Inoue
Yuki Urano
VirtualTech Japan Hiromichi Ito
DOCOMO Cloud Platform
NTT DOCOMO, INC. All Rights Reserved
NTT DOCOMO’s challenge
looking ahead
the world of 5G OpenStack
NTT DOCOMO, Inc. Hiroaki Inoue
Yuki Urano
VirtualTech Japan Hiromichi Ito
DOCOMO Cloud Platform

2. NTT DOCOMO, INC. All Rights Reserved
 Hiroaki Inoue
 Engineer, NTT DOCOMO, Inc.
 Technical consultant of DOCOMO Cloud Platform
 Yuki Urano
 Engineer, NTT DOCOMO, Inc.
 Resource manager of DOCOMO Cloud Platform
 Hiromichi Ito
 CTO, VirtualTech Japan
 One of the first members of proposing OpenStack Bare Metal
Provisioning (currently called "Ironic")
About us

3. NTT DOCOMO, INC. All Rights Reserved
Current Challenge
 DOCOMO Cloud Platform
 BizDevOps
Challenge for the future
 DOCOMO 5G Open CloudTM
 Next Challenge

4. NTT DOCOMO, INC. All Rights Reserved
Current Challenge
 DOCOMO Cloud Platform
 BizDevOps
Challenge for the future
 DOCOMO 5G Open CloudTM
 Next Challenge

5. NTT DOCOMO, INC. All Rights Reserved
 DOCOMO Cloud Platform(DCP) is a project that started in 2015 as a
private cloud.
 Build with OpenStack and using various open source software.
 Our target is user who demands low price even if few functions.
What is DOCOMO Cloud Platform (DCP)
Project Management
Development Cloud Resources Log Monitoring

6. NTT DOCOMO, INC. All Rights Reserved
 Migrated a large scale in-house system to DCP (2015)
 Started sales outside the company (2017)
 The number of vCPU has exceeded 200,000 cores!!
 The scale is expanding!!
2015 2016 2017 2018
region 1 2 4 11
core 1,500 10,000 35,000 200,000
Scale of DCP
C h e c k Yo u Tu b e →
o u r ex p a n d i n g s t r at e g y
( s u m m i t @ B a r c e l o n a )

7. NTT DOCOMO, INC. All Rights Reserved
 Consulting team
 One consultant for each project
 Suggest the optimal configuration according to user requirements
 Reference model
 Template for building a system with DCP recommended configuration
using Heat
 Template the basics of services, such as WebApp and DB HA, OS and
middleware tuning, and security measures using OSS
How we focus on our target users

8. NTT DOCOMO, INC. All Rights Reserved
Operates with TCO that is smaller than AWS under the same conditions
Operational Costs
DOCOMO Cloud Platform

9. NTT DOCOMO, INC. All Rights Reserved
DCP has succeeded as a cloud project!!

10. NTT DOCOMO, INC. All Rights Reserved
Current Challenge
 DOCOMO Cloud Platform
 BizDevOps
Challenge for the future
 DOCOMO 5G Open CloudTM
 Next Challenge

11. BizDevOps
-Resource Efficiency and Cost Reduction-

12. NTT DOCOMO, INC All Rights Reserved
Background to DCP Expansion Strategy
 DCP Expansion Strategy
 Main Strategy : Forest and Tree
 Firstly, making a “forest ” a large project and then, filling in “ trees ” with small
and medium sized projects.
Make a forest
Fill in trees
Target

13. NTT DOCOMO, INC All Rights Reserved
Steps to Resource Efficiency and Cost Reduction
 Resource Efficiency and Cost Reduction in four steps:
1. Prepare an OpenStack environment
2. Visualize VM resources and investigate how many they are being used
3. Feedback VM usage to our planning department
4. Work towards resource efficiency and cost reduction
Step3:
Feedback
Step1:
System stability first
Step2:
Investigate VM resources
Step4:
Cost reduction

14. NTT DOCOMO, INC All Rights Reserved
Step1: DCP Migration of a Large Scale in-house System
 The number of resources was not changed
 This system is our commercial large-scale operation support system and need
to make the system stable
 Migrate to DCP because the physical servers were EOL
several thousands of physical severs
migration to DCP
OpenStack
VM VM VM
PM PM PM

15. NTT DOCOMO, INC All Rights Reserved
Step2: VM Resource Usage
 VM usage resource is important
 Find redundant resources for allocated flavor resources
Hardware
OS
OpenStack
VM
VM VMallocated
flavor resource
VM …
usage resource
all servers resources
KNOWN
KNOWN
IMPORTANT!
unused resource
IMPORTANT!
VM
reserved
resources
KNOWN
reserved resourcesKNOWN

16. NTT DOCOMO, INC All Rights Reserved
Step2: VM Resource Visualization
 Distribute Exporter files to each Compute Node
 Use Prometheus and Grafana to get resource data
・・・
libvirt
Compute Node
VM
Exporter
VM ・・・
Monitoring Server Region B
Region C
Region A
gathering VM
resource data
shaping VM
resource data

17. NTT DOCOMO, INC All Rights Reserved
Step2: VM Resource Visualization Results 1/4
 CPU and memory allocation (all servers)
 Visualize resources for each VM allocated on OpenStack
allocated reserved

18. NTT DOCOMO, INC All Rights Reserved
Step2: VM Resource Visualization Results 2/4
 Maximum CPU and memory usage per VM
 Visualize resources for each VM allocated on OpenStack
VMallocated
reserved
allocated VM
resource visualization

19. NTT DOCOMO, INC All Rights Reserved
Step2: VM Resource Visualization Results 3/4
 A lot of VMs do not use 100% even with maximum usage
 There are redundant resources
Low efficient VMs
These VMs use less CPU and
Memory
Medium / Low efficient VMs
These VMs use Memory than
CPU
VM
Highly efficient VMs
These VMs use more CPU
and Memory

20. NTT DOCOMO, INC All Rights Reserved
Step2: VM Resource Visualization Results 4/4
 More than tens of percent of CPU allocated resources are redundant
 Not thinking about using reserved resource
allocated
reserved
usage
redundant

21. NTT DOCOMO, INC All Rights Reserved
Step3: Feed Back the Results
 Feed back to our planning department and work towards resource
efficiency
 Specific measures:
 Reviewing the allocated flavor resource
 Applying overcommit with appropriate values.
 Verify in another environment and operate without problems.
 Currently trying to make resources more efficient!!

22. NTT DOCOMO, INC All Rights Reserved
Our BizDevOps Challenge
 One of our goal is to reduce costs
 Allocate resources for new projects without building new hardware
 Our BizDevOps enables centralized resource efficiency
 OpenStack gives us the flexibility to make our resources more
efficient.
 Custom flavors are available depending on application requirements.
 Appropriate overcommit can be set.

23. NTT DOCOMO, INC All Rights Reserved
Current Challenge
 DOCOMO Cloud Platform
 BizDevOps
Challenge for the future
 DOCOMO 5G Open CloudTM
 Next Challenge

24. NTT DOCOMO, INC All Rights Reserved
About 5G (1/2)

25. NTT DOCOMO, INC All Rights Reserved
About 5G (2/2)

26. NTT DOCOMO, INC All Rights Reserved
To promote 5G
 5G test base stations
 By merely providing 5G networks, our users can’t test.
 They need platform directly connected to 5G.
 We created a new region directly connected to 5G for them!!

27. NTT DOCOMO, INC. All Rights Reserved
 Hardware based VXLAN gateway device
 High throughput(Over 100Gbit/sec)
 Low latency(same as L3 switch)
5G
base station
Compute Node
Instance(VM)
vNIC
Virtual Switch
vPort
VTEP
vPort
Hardware based VXLAN
gateway
Port
VLAN
Hardware
VTEP
UDP UDPIPv4
Port
5G
UE
UPF
RANVXLAN
L2GW

28. NTT DOCOMO, INC. All Rights Reserved
 Start of 5G Open Partner Program (2018/2)
 Joined by 1,600 companies and organizations (2018/7)
 Released 5G Open CloudTM (2018/7)
DOCOMO Cloud Platform
Directly
connected
Directly
connecte
Directly
connecte
DOCOMO
asset
partner
asset
Tokyo Osaka
DOCOMO 5G Open CloudTM
DOCOMO 5G Open LabTM DOCOMO 5G Open CloudTM

29. NTT DOCOMO, INC. All Rights Reserved
 DCP
 Cloud Resources (nova, neutron, swift, GPU, etc.)
 Mobile network (5G, 4G)
 DOCOMO core technologies
 AI Agent
 Multipurpose Dialogue Engine
 Prefetching Engine
 IoT Control Engine
 Image Recognition Engine
 Specific Object Recognition
 General Object Recognition
 Object Detection
 Similar Image Search
Services

30. NTT DOCOMO, INC. All Rights Reserved
 Remote video production
 Can edit and broadcast in real time
 Only the camera is needed in the site
 No need to go the site with repeater car
 Remote 3D Design
 People in remote share the same VR space
 Can design 3D model in VR space
 Feel as if we are working in the same room
Solutions examples of 5G Open CloudTM
shooting at the interview site
and editing immediately
Remote video production PF
Remote 3D Design PF
Tokyo Osaka
・ design between people in
remote
・ 3D modeling with VR

31. NTT DOCOMO, INC. All Rights Reserved
Future Prospects
In the world of 5G,
direct connection of the network and the system platform
is essential.

32. NTT DOCOMO, INC. All Rights Reserved
Who can provide its platform?
Future Prospects

33. NTT DOCOMO, INC. All Rights Reserved
That is NTT DOCOMO
and our mission towards the world of 5G.
Future Prospects

34. NTT DOCOMO, INC. All Rights Reserved
Current Challenge
 DOCOMO Cloud Platform
 BizDevOps
Challenge for the future
 DOCOMO 5G Open CloudTM
 Next Challenge

35. NTT DOCOMO, INC. All Rights Reserved
 Container
 GPU
 FPGA
 Network
 QoS
 Persistent memory
Next Challenge

36. NTT DOCOMO, INC. All Rights Reserved
 Kubernetes on OpenStack
 Openstack Magnum(PoC)
 Rancher Kubernetes Engine(PoC)
 Custom Ansible playbook(PoC)
 OpenStack on Kubernetes
 Airship
 Akraino
 Mobile Edge Computing
 Akraino
Container

37. NTT DOCOMO, INC. All Rights Reserved
 bare metal(Production)
 L2GW connect between dedicated GPU clusters and OpenStack
instances.
 Passthrough(Production)
 Only short-period workload uses passthrough type.
 vGPU(PoC)
 NVDIA GRID
 Mobile Edge Computing
 Desktop as a Service
 Cyborg
Compute Node with GPU
Instance with vGPU
GPU

38. NTT DOCOMO, INC. All Rights Reserved
OpenStack L2GW
FPGA
Nodes
40GbE x ?
GPU
Clusters
 bare metal(PoC)
 L2GW connect between dedicated FPGA nodes and OpenStack
instances.
 Cyborg
FPGA

39. NTT DOCOMO, INC. All Rights Reserved
 NIC
 Smart NIC（PoC）
 25/50/100GbE(Staging)
 Switch
 100GbE(Staging)
 IPv6
 Tenant(Overlay) network(PoC)
 Infrastructure(Underlay) network
 Network Device Management
 OpenDaylight(PoC)
 NETCONF/YANG
 Open Networking Automation Platform
Network

40. NTT DOCOMO, INC. All Rights Reserved
 Storage
 IOPS(production)
 Latency
 Network
 Bandwidth(production)
 Latency
QoS

41. NTT DOCOMO, INC. All Rights Reserved
 Hardware
 NVDIMM-N(PoC)
 DDR4
 DRAM + NAND Flash
 Optane DC
 DDR4
 3D Xpoint
 Software
 Persistent Memory Development Kit(PoC)
 DAX(PoC)
Persistent memory

42. NTT DOCOMO, INC. All Rights Reserved
Thank you for listening!