1. closedloopautomation
Telemetry/Analytics-ml/Orchestration
Q3 2019
Emma Collins/Tong Zhang

2. 2
Legal Disclaimer
General Disclaimer:
© Copyright 2019 Intel Corporation. All rights reserved. Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel.
Experience What’s Inside are trademarks of Intel. Corporation in the U.S. and/or other countries. *Other names and
brands may be claimed as the property of others.
Technology Disclaimer:
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software
or service activation. Performance varies depending on system configuration. No computer system can be absolutely
secure. Check with your system manufacturer or retailer or learn more at [intel.com].
Performance Disclaimers:
Cost reduction scenarios described are intended as examples of how a given Intel- based product, in the specified
circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does
not guarantee any costs or cost reduction.
Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and
provided to you for informational purposes. Any differences in your system hardware, software or configuration may
affect your actual performance.

3. ScaleEfficiencywithData-Driven,ClosedLoopAutomation
Intel Platform Features are part of intelligent, closed loop solutions
that are reactive, proactive and predictive, delivering new levels of
efficiency for IT and network infrastructure.
Automated Action Telemetry Analysis
Software and Services
Telemetry
IA Platform Telemetry
Fine-grained Hardware and software insights feeding operational intelligence and automation

4. Intel's Ingredients for Closed Loop Automation
 Intel driven Analytics solutions using
IA feature data
OR
 Integrate with proprietary/commercial
network monitoring and analytics
solutionsOrchestration
IA Platform
Telemetry
Analytics
 Intel driven MANO solutions to
provide scale/heal/placement
actions that incorporate IA features
OR
 Integrate with
proprietary/commercial MANO
solutions
IA Feature Metrics and
Events Exposure
IA Feature Detection and
Provisioning
Power PMU RDT RAS Other…
The Closed Loop
Intel Platform

5. Inteltelemetrycollectionandpublication
Platform
Compute Networking Memory Storage Acceleration
Collectd South Bound Plugins
Collectd
Collectd North Bound Plugins
Open stack
MANO
Platform/
NFVI
Monitoring /Analytics
Systems
Telemetry Publication
Telemetry Consumption
Kubernetes
ONAP
Telemetry Collection
Application

6. 6
IntelTelemetryCoverage
Compute Network Storage
Hypervisor
NFVIVirtualised
Compute
Virtualised
Network
Virtualised
Storage
Collectd
PMU
counters
NIC counters
vSwitch
counters
Common / Standard Open APIs
VM Stall
Detection/
RT Stall Detection
Enterprise and Network Management Tools
RAS
Hypervisor/Container
Counters
Intel® Node Manager
Open Platform
Collector
Intel® Run Sure Technology
MCA* PCIe AER
Resilient System Technology
Resilient Memory Technology
SDDC DDDC+1 Mirroring
RAID
Intel® Rapid
Storage
Technology
Intel®
Management
Engine
IPMI
C
M
T
Intel® RDT
C
A
T
M
B
M
C
D
P
P
O
W
E
R
VIM
Intel® Infrastructure
Management
Technologies
Redfish SYSLOG KafkaSNMP API VES Plugin Prometheus
OpenStack Kubernetes

7. TyingtheIntelPlatformtotheCustomerBusinessCase
Customer Business Use Case
Platform
Slicing
(Part of 5G Network
Slicing)
Platform
Resiliency
Security
(side channel attacks)
Power
Optimisation
Intel Platform
Power PMU RDT RAS Other…
MANO
Analytics

8. MachineLearningWillCompletetheEvolution
Learn
Learn
Learn
Learn
Watch Decide
Collect
Act
App
Server/Storage/Network
Watch
Expose Platform and
App data through
APIs
Act
Mechanism for
Policy Activation
and Enforcement
Learn
Machine Learning
for continuous
improvement
Decide
Analytics to define
correct response
Moving From Automation to Self-OptimizationTaking the manpower out of networking, cloud…

9. DeliveringClosedLoopAutomationforNFV
Automated
Action
IA Platform
Telemetry
Telemetry
Analysis
1. Enable IA Service Assurance Telemetry
through instrumentation and exposure in
an industry standard manner
2. Enable IA Telemetry Analytics through
Telemetry Compaction, KPI identification
and prediction
3. Enable Closed Loop Automation through
Orchestration enabling and industry
proof points

10. Casestudy:MLforNFVserviceassurance
10
• Efficient dynamic network management is one major challenge for NFV
• Machine learning plays an important role in addressing this challenge by
analyzing gathered data for various purposes:
• dynamic resource allocation
• security threats alert
• performance degradation detection
• demand prediction
“Cognet: A Network Management Architecture Featuring Cognitive Capabilities,” Proc. Euro. Conf. Networks and Commun., June 2016
1. Data pre-processing
Feature Engineering/Reduction
2. KPI prediction/forecasting
Regression/Classification
3. Closed loop optimization
Reinforcement Learning

11. NFVtestsystems
11
vEPC – virtual Evolved Packet Core
vCMTS – virtual Cable Modem Termination System
• Telemetry data dumped through Collectd includes CPU, PMU, Memory, Load, etc.
• Total number of telemetry data hundreds to thousands sampled at configured interval (1s or 10s)
• Target KPI (Key Performance Indicator): packet drop rate

12. Datapre-processing
12
• Data filtering – remove irrelevant data (e.g. control plane data for this case)
 1065 features remaining
• Data alignment and interpolation
• Feature selection – remove features with no change over time
 726 features remaining
• Data normalization
• Data splitting to training, validation and testing sets
 Tens of thousands of samples split into 8:1:1

13. Telemetryfeaturecompaction
• Feature Selection
• The process of selecting a subset of relevant features for use in the model construction
• Filter methods – Select features based on scoring from statistical measures e.g. SelectKBest
• Wrapper methods – search for optimal feature combination that results in best predictive results
e.g. Recursive Feature Elimination, Boruta
• Unsupervised learning methods – group features that behave similarly e.g. FeatureAgglomeration
• Feature Transformation (dimension reduction)
• Convert the feature vector into lower dimension space with learned transformations
• Supervised: PLS (Partial Least Squares), CCA (Canonical Correlation Analysis), LDA (Linear
Discriminant Analysis)
• Unsupervised: PCA (Principal Component Analysis)
13

14. 10 20 40 Feature 10 20 40 Feature 10 20 40 Feature
• cpu_value_idle_18 • • intel_rdt_value_bytes_llc_20 • • • intel_rdt_value_memory_bandwidth_local_12
• cpu_value_idle_49 • • intel_rdt_value_bytes_llc_27 • intel_rdt_value_memory_bandwidth_local_3
• • cpu_value_interrupt_18 • intel_rdt_value_bytes_llc_41 • intel_rdt_value_memory_bandwidth_local_33
• cpu_value_interrupt_20 • • intel_rdt_value_bytes_llc_50 • intel_rdt_value_memory_bandwidth_local_53
• cpu_value_system_39 • intel_rdt_value_bytes_llc_53 • • intel_rdt_value_memory_bandwidth_local_9
• • cpu_value_user_13 • • intel_rdt_value_bytes_llc_6 • intel_rdt_value_memory_bandwidth_remote_2
• cpu_value_user_3 • intel_rdt_value_bytes_llc_8 • intel_rdt_value_memory_bandwidth_remote_6
• cpu_value_user_46 • intel_rdt_value_ipc_nan_22 • • ipmi_value_fanspeed_System Fan 1 fan_cooling (29.1)
• cpu_value_user_9 • • intel_rdt_value_ipc_nan_23 • ipmi_value_temperature_HSBP 1 Temp drive_backplane (15.1)
• df_value_free_etc-hosts • intel_rdt_value_ipc_nan_26 • ipmi_value_temperature_LAN NICTemp system_board (7.1)
• df_value_used_etc-hosts • intel_rdt_value_ipc_nan_5 • ipmi_value_temperature_P2 DTS Therm Mgn processor (3.2)
• disk_read_disk_time_sda • • intel_rdt_value_ipc_nan_50 • • • irq_value_TLB
• intel_pmu_value_branches_42 • intel_rdt_value_ipc_nan_53 • • • load_longterm
• intel_pmu_value_instructions_14 • intel_rdt_value_ipc_nan_54 • • • load_midterm
• intel_pmu_value_page-faults_51 • intel_rdt_value_ipc_nan_9 • load_shortterm
• intel_pmu_value_page-faults_all • numa_value_other_node_node1 • • • memory_value_cached
• • memory_value_slab_unrecl
Top TopTop
TopTelemetryFeaturesSelectedByML
14
Recursive feature elimination stepping down the target from 40  20  10

15. Featureselectionresults
15
• Applying various feature selection algorithms to reduce the number of telemetry data sampled without
compromising the prediction accuracy
• Smaller set of telemetry data saves training/inference time
• vEPC test data set with original feature dimension ~400
• GradientBoostRegressor algorithm used for packet loss rate prediction
• Select top number of features from feature importance output of GradientBoostRegressor

16. KPIprediction&forecasting–supervisedlearning
16
• Regression
• Train the model to predict target KPI using reduced telemetry data samples
• Example: Packet drop rate
• Accuracy Measured by MSE (Mean Squared Error) – the smaller the better
• Classification
• Detect packet drop from telemetry data
• Measurement
• Accuracy_score = correct_prediction / total_sample
• Precision/Recall/F1-score, etc.
• Time Series Forecasting
• Predict future KPI value based on historical telemetry data and observed KPI trend

17. KPIforecastingusingLSTM
17
• vCMTS downlink test data
• KPI – scheduling packet loss rate
• Input: previous 60 seconds of 20 selected
telemetry data + KPI
• Output: predicted future value (5 seconds
later) of target KPI
• Tensorflow BasicLSTMCell: Two layers,
each layer 150 neurons

18. Closed-loopAutomation Xeon
vCMTS -
0
Traffic Generator
vCMTS -
0VNFs
NIC
Machine Learning
Modules
collectd
InfluxDB
Traffic
forecasting
Reinforcement
learning
HW
Optimization
(RDT, DVFS, WL
consolidation, etc.)
Compacted
metrics
Using ML:
• Track/forecast workload,
performance
• Dynamically adjust
resource allocation
Benefits:
• Reduced TCO through
power saving and
increased HW utilization

19. Frequencytuning

20. ENIPoC–NetworkSliceLifecycleManagement
20
• For generating new scale
up/down and converting the
intent to suggested configuration
• LSTM is used for traffic prediction
AI-based predictor:
TNSM:
CNSM:
• Provides underlay network
control to satisfy the network
slice requests
• FlexE and a FlexE-based
optimization algorithm are used
for underlay network slice
creation and modification
• Provides core network control
to satisfy the network slice
requests
ETSI ISG ENI – Experiential Networked Intelligence

21. Trafficpredictionforresourceoptimization
21
Blue: actual traffic
Orange: predicted traffic

22. AIsystemarchitecture
22
Source:

23. Intel Confidential
Backup
23

24. Further Resources
Learn more from these helpful sites:
https://networkbuilders.intel.com/network-technologies/serviceassurance
https://wiki.opnfv.org/display/fastpath/Barometer+Home
https://wiki.openstack.org/wiki/Telemetry
https://01.org/openstack/blogs/2015/openstack-enhanced-platform-
awareness-white-paper

25. 25
Collectd101materials
• Collectd 101
• https://wiki.opnfv.org/display/fastpath/Collectd+101
• Write simple read plugin
• https://wiki.opnfv.org/display/fastpath/Collectd+how+to+implement+a+si
mple+plugin

26. 26
Barometer Strategy:
• Ensure platform metrics/events are accessible
through open industry standard interfaces.
• Demonstrate IA platform technologies can be
monitored, consumed and actioned in real time
Opnfvbarometer–Intelplatformfeatureplugins
One Click Install:
• Easy install/configuration
for customers
• One command to install
Collectd/Influxdb/Grafana
Three container approach for Collectd:
• Stable Container: latest stable branch
• Master Container: up to date with master
• Experimental Container: cherry pick
features of interest

27. BarometerLinks
Barometer Home: https://wiki.opnfv.org/display/fastpath/Barometer+Home
Metrics/Events through Barometer (not on Collectd site):
https://wiki.opnfv.org/display/fastpath/Collectd+Metrics+and+Events#CollectdM
etricsandEvents-Metrics
Barometer “One-click” install:
https://wiki.opnfv.org/display/fastpath/One+Click+Install+of+Barometer+Contain
ers
27