Weitere ähnliche Inhalte
Ähnlich wie 如何用k8s打造國產5G NFV平臺?剖析經濟部5G核網技術的關鍵 (20)
Kürzlich hochgeladen (20)
如何用k8s打造國產5G NFV平臺?剖析經濟部5G核網技術的關鍵
- 2. About Me
版權宣告©工業技術研究院 2
• 梁維恩 Jace Liang
• 工研院資通所/資料中心架構與雲端應用軟體組 Div. X.
• What our team does
▪ NFV Performance Lab (cooperate with intel®)
▪ NFVI Architect of MOEA 5G Project
e-mail: jace.liang@itri.org.tw
Github: mJace
- 5. Network Function Virtualization
版權宣告©工業技術研究院 5
CDN Session Border
Controller
WAN
Acceleration
DPI Carrier
Grade NAT
Tester / QoE
Monitor
SGSN / GGSN PE Router Firewall
Typical network appliances
General purpose
servers
Standard storage
NFV-based Approach
Standard switches
NFV is a process of decoupling the network function
from proprietary hardware appliances
- 6. The Impact of NFV
版權宣告©工業技術研究院 6
• The existing player need to invest and adopt new techs
▪ Help telecom increase service agility and efficiencies
• Easier to deploy service
• Able to reuse physical resources
• The new business potential
▪ Server providers
▪ VNF providers
▪ NFVI providers
▪ Solution providers
▪ Network operators
- 7. Evolving from VNFs to CNFs
版權宣告©工業技術研究院 7
Ref : https://events.linuxfoundation.org/wp-content/uploads/2017/11/2018-
OSSNA-CNF-Journey-in-Telecom-Seminar.pdf
- 8. Performance Issue in NFV
版權宣告©工業技術研究院 8
• Traffic hidden from monitoring
▪ Cisco estimates that about 73% of data-center traffic will come from within the data
center by 2019—most of this traffic is virtual machine to virtual machine (VM to VM)
communication
▪ The inability to isolate production from monitoring traffic
• No Solution to address performance uncertainty in NFV
▪ Where is the major bottleneck point of an NFV system
Container / POD
VNF Application
vCPUs vNICs
Container / POD
VNF Application
vCPUs vNICs
Host OS
(State transition overhead due to privileged operations and idleness)
Hardware Layer
(Resource exhaustion, buffer overflow or queue imbalance at NICs)
Virtual Switch Layer
(Less optimized packet processing engine)
Less optimized VNF implementations
Inter-Container communication overhead
Network I/O
Overhead
NFV Architecture with Major Bottleneck Points
- 10. Pros to run VNF on k8s
版權宣告©工業技術研究院
10
• K8S as container orchestrator
▪ Container grouping using pod
▪ Self-healing
▪ Auto-scalability
▪ DNS management
▪ Load balancing
▪ Rolling update or rollback
▪ Resource monitoring and logging
......
版權宣告©工業技術研究院
- 11. Challenge of Networking
版權宣告©工業技術研究院 11
• Kubernetes native CNI is not design for telco VNF
▪ Decoupling Control/Data plane is not easy in native K8S network.
• Only one NIC for each Pod.
▪ The network behavior is not suit for VNF.
• Whole traffic must go through Master or Load Balancer.
- 12. Challenge of Compute & Management
版權宣告©工業技術研究院 12
• Native K8S does not guarantee performance stability.
▪ It is very critical for VNF to require a stable and predictable performance.
• Native Kubernetes does not support NUMA aware and CPU Pinning.
• Native k8S is lack of detail hardware spec. info.
▪ When Deploy/Migrate/Scale VNF, must consider its required HW function.
• Native K8S don’t know if the machine supports certain CPU instruction and the NIC’s
spec.
- 14. DANM
版權宣告©工業技術研究院 14
• From Nokia
▪ Support VXLan, IPAM
▪ Allowing pods to attach multiple network interfaces. Calico, Flannel, SR-IOV, etc
▪ a Kubernetes controller capable of centrally managing both VxLAN and VLAN
interfaces of all Kubernetes hosts
▪ another Kubernetes controller extending Kubernetes' Service-based service
discovery concept to work over all network interfaces of a Pod
https://github.com/nokia/danm
- 15. SONA
版權宣告©工業技術研究院 15
• Used by SK
• Extension work from OpenStack SONA
• Pure SDN Based overlay solution
▪ OvS L3 load balancing
▪ OvS L2/L3 connectivity
• Support OVS-DPDK, Smart NIC
• Support Multi NIC
https://github.com/sonaproject/sona-cni
- 18. Akraino
版權宣告©工業技術研究院 18
• Create an open source edge software stack for different
scenario
▪ IOT Edge, Edge on prem, AI Edge
▪ Nokia, AT&T, SAMSUNG, etc.
▪ https://www.lfedge.org/projects/akraino/
- 20. How do we address these problems
版權宣告©工業技術研究院 20
• Kubernetes native CNI is not design for telco VNF
▪ Decoupling Control/Data plane is not easy in native K8S network.
• Only one NIC for each Pod.
▪ The network behavior is not suit for VNF.
• Whole traffic must go through Master or Load Balancer.
• Native K8S can not guarantee performance stability.
▪ It is very critical for VNF to require a stable and predictable performance.
• Native Kubernetes does not support NUMA aware and CPU Pinning.
• Native k8S does not own detail hardware spec. info.
▪ When Deploy/Migrate/Scale VNF, must consider its required HW function.
• Native K8S don’t know if the machine support certain CPU instruction or the NIC’s spec.
- 21. ITRI Open Source X-K8S solution
版權宣告©工業技術研究院 21
• Network
▪ Multus CNI
▪ SRIOV
▪ SmartNIC
▪ SRIOV Device Plugin
• Compute/Management
▪ CMK – CPU Management for
Kubernetes
▪ NFD – Node Feature Discovery
▪ VNF Monitor
▪ Performance Optimizer
- 22. Multus CNI
版權宣告©工業技術研究院 22
• Enables attaching multiple network interfaces to pods
▪ So that the VNF pod can separate the network by different usage, for example
• flannel as manage
• SRIOV VFs as Data in/out
▪ Support any CNI – SRIOV, Host-device, etc
• Most Popular
▪ Most used by community
▪ Already included in kubespray
- 23. SRIOV
版權宣告©工業技術研究院 23
• Single Root I/O Virtualization
▪ As a PCI-SIG Standard, SRIOV allowing PCIe I/O device to provide multiple virtual
function.
• SRIOV-CNI
▪ Allowing Pods to use SRIOV VF as it’s CNI
• SRIOV Device Plugin
▪ Allowing K8S scheduler to manage SRIOV NIC as device resource.
NIC
Pod Pod
vNIC vNIC
- 24. 資通所重要營運資料,禁止複製、轉載、外流 ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE -24-
SRIOV vs. OVS-DPDK
64 Bytes 128 Bytes 256 Bytes 512 Bytes 1024 Bytes 1280 Bytes
OVS-DPDK-1-Core 2.51 2.46 2.38 2.26 1.74 1.47
OVS-DPDK-2-Cores 6.31 6.1 5.64 4.98 3.71 3.04
OVS-DPDK-4-Cores 12.96 12.5 11.82 9.32 4.77 3.83
SRIOV 59.52 32.89 17.85 9.32 4.77 3.83
BM 59.52 32.89 17.85 9.32 4.77 3.83
0
10
20
30
40
50
60
70
Throughput(Mpps)
OVS-DPDK vs SRIOV vs Baremetal 4 Port - Throughput (L2-Fwd)
OVS-DPDK performance is similar to SRIOV
performance in large packet size scenario.
- 26. SmartNIC
版權宣告©工業技術研究院 26
• Smart NIC enables the functionality of OVS in a hardware
approach
▪ Improve network throughput
▪ Reduce CPU usage
NIC
Pod Pod
vNIC vNIC
OVS
- 28. The Mapping Between CPU and NIC
版權宣告©工業技術研究院 28
• The CPU core and NIC for packet generator should be in the same CPU socket, or
the QPI interface will be the performance bottleneck.
▪ 10GbE line rate (64bytes packet) = 10.00e9 bits / (8 bits * (64 + 20)bytes ) = 14.88e6 packets -> 67.2 nano-sec / pkt
▪ CPU and NIC in same socket, 10G port could generate 14.88 Mpps.
▪ CPU and NIC in different socket, 10G port could only generate 10 Mpps.
18 Core Intel(R) Xeon(R) CPU
E5-2695 v4 @ 2.10GHz
CPU 1
18 Core Intel(R) Xeon(R) CPU
E5-2695 v4 @ 2.10GHz
CPU 2
QPI
X710-DA4
adapter GEN
QPI interface will be the
performance bottleneck.
- 29. Result of NUMA misconfiguration
版權宣告©工業技術研究院 29
L2-Fwd Rx L2-Fwd Tx L3-Fwd Rx L3-Fwd Tx ACL Rx ACL Tx MPLS Rx MPLS Tx
BM 59.52 59.52 53.29 53.29 37.38 33.82 59.52 58.17
VM w/ NUMA 59.52 59.52 53.29 53.29 37.41 33.78 59.52 58.17
VM w/o NUMA 59.52 59.46 53.36 43.4 37.46 30.66 59.52 56.96
0
10
20
30
40
50
60
70
Throughput(Mpps) Baremetal vs SRIOV 4 Port - Throughput (64Bytes)
VM without NUMA-aware configuration
may decrease 10 – 20% performance.
VM performance with NUMA-aware configuration
is similar to Baremetal performance.
- 30. CMK & NFD
版權宣告©工業技術研究院 30
• CPU Manager for Kubernetes
▪ Pin Pods on specific CPU core.
▪ https://github.com/intel/CPU-Manager-for-Kubernetes
• Node Feature Discovery
▪ Detects hardware feature available on each node in k8s cluster.
▪ https://github.com/kubernetes-sigs/node-feature-discovery
*CMK is now a milestone for k8s v1.16 as a alpha version.
- 31. Compute / Management
版權宣告©工業技術研究院 31
• Compute / Management – node wise
▪ Support NUMA aware and CPU Pinning
▪ Enhanced Platform Awareness (EPA) VNF management
SRIOV NIC SmartNIC
- 32. Compute / Management
版權宣告©工業技術研究院 32
• Compute / Management – node wise
▪ Support NUMA aware and CPU Pinning
▪ Enhanced Platform Awareness (EPA) VNF management
SRIOV NIC SmartNIC
kubelet
api-server
CoreDns
Scheduler
kube-proxy controller
MME
SGW-C
SGW-U
PGW-C
PGW-U
- 33. Compute / Management
版權宣告©工業技術研究院 33
• Compute / Management – cluster wise
▪ Support NUMA aware and CPU Pinning
▪ Enhanced Platform Awareness (EPA) VNF management
kubelet
api-server
CoreDns
Scheduler
kube-proxy controller
MME
PGW-C
SGW-U
PGW-C
PGW-U
SRIOV
Network
SmartNIC
Network
SGW-C
kubelet
api-server
CoreDns
Scheduler
kube-proxy controller
SGW-C
PGW-USGW-U
SGW-U SGW-U
SGW-U
- 34. VNF Monitor
版權宣告©工業技術研究院 34
• How to get NUMA related info?
▪ Which CPU core that my pod runs on?
▪ NUMA node of NIC in my pod?
▪ Is core-affinity set?
- 35. Which CPU core that my container
runs on?
版權宣告©工業技術研究院 35
https://github.com/mJace/numacc
- 36. Developing feature.
版權宣告©工業技術研究院 36
▪ APM system for 5G VNF
• Customized APM system for 5G VNF
▪ Self performance optimization
▪ Data plane network slicing
• Network slicing based on the SRIOV data plane
▪ DNS / Service discovery for extra network interfaces
- 38. EPC
版權宣告©工業技術研究院
38
• 完成3GPP R14版本 Container Based C/U Split vEPC
雛型系統,支援快速彈性擴充。每秒20個UE連線的情
況下約可乘載50K個使用者終端。
• 於 MME 實作 eMBMS 功能,支援車載網路群播應用。
並與Content Provider和商用E-UTRAN整合測試群播
功能完成
• 支持OVS-DPDK and SRIOV技術
• 透過SmartNIC兩種不同的數據路由,藉以實現大吞吐
量(接近10Gbps),改善虛擬化後造成之網路傳輸延遲,
不需額外佔用伺服器計算資源,延遲時間可縮短至
300us以內。
• 完整利用SmartNIC效能進行GTP封包處理
• Future Work: SBA 5GC
X-Kubernetes
- 39. iMEC
版權宣告©工業技術研究院 39
發展 MEC 與 4G/5G 接取網路的整合,技術成果為 iMEC
iMEC 提供節省後端網路頻寬且低延遲雲端平台之解決方案
Enterprise
EPC Cloud
Wi-Fi AP
4G/5G RAN
FTTx
iMEC
企業私有網路終
端
設
備
網路&服務虛擬化 智慧路由 使用者辨識
多邊緣雲管理 即時服務啟動 系統效能水平擴展
一鍵自動部署
支援 OPNFV, Kubernetes及
DC/OS等多種 NFV 平台
支援網路本地分流功能(Local
breakout/Traffic Offload) ,
流量負載平衡
支援 UE Identify 功能,可辨
別垂直專網之企業用戶
一鍵自動部署iMEC系統與
ME APP
支援多邊緣雲互連架構,進
行服務佈建與管理
Just-in-Time 技術,可於0.2
秒快速啟動服務,並節省
edge cloud資源
可以動態增加User Plane
Function數量,使得效能線
性成長
- 41. 5G Network Architecture on X-K8S
版權宣告©工業技術研究院 41
Key Features & Specification
• High Throughput
10 / 25 / 40 / 100 Gbps
• CNF – Containerized Network Function / Services
• Performance Optimized
SmartNIC, SRIOV, CMK, NFD, Multus
• Network Isolation
CNF / Services on X-K8S
• CNF : iMEC
Data traffic offloading
• CNF : III-vEPC
Use P4 SmartNICs as an accelerator for GTPU
• CNF : SON (Self-Organizing Network)
• APP : VR Community (III)