6. Controllability and Observability
Basic concept is simple, a stable feedback control system requires:
1. ability to influence all important system states (controllable)
2. ability to monitor all important system states (observable)
7. It’s hard to stay on the road if you can’t see the
road, or keep to the speed limit without a
speedometer
It’s hard to stay on the road or maintain
speed if your brakes, engine or steering fail
Controllability and Observability driving example
Observability
Controllability
States location, speed, direction, ...
8. Effect of delay on stability
Measurement delay Planning delay
Time
Configuration delayDisturbance Response delay
EffectLoop delay
DDoS launched Identify target, attacker Black hole, mark, re-route? Switch CLI commands Route propagation Traffic dropped
Components of loop delay
e.g. Slow reaction time causes
tired / drunk / distracted
driver to weave, very slow
reaction time and they leave
the road
9. What is sFlow?
“In God we trust. All others bring data.”
Dr. Edwards Deming
11. Open source agents for hosts, hypervisors and applications
Host sFlow project (http://host-sflow.sourceforge.net) is center
of an ecosystem of related open source projects embedding
sFlow in popular operating systems and applications
13. Simple
- standard structures - densely packed blocks of counters
- extensible (tag, length, value)
- RFC 1832: XDR encoded (big endian, quad-aligned, binary) - simple to encode/decode
- unicast UDP transport
Minimal configuration
- collector address
- polling interval
Cloud friendly
- flat, two tier architecture: many embedded agents → central “smart” collector
- sFlow agents automatically start sending metrics on startup, automatically discovered
- eliminates complexity of maintaining polling daemons (and associated configurations)
Scaleable push protocol
14. • Counters tell you there is a
problem, but not why.
• Counters summarize
performance by dropping high
cardinality attributes:
- IP addresses
- URLs
- Memcache keys
• Need to be able to efficiently
disaggregate counter by
attributes in order to
understand root cause of
performance problems.
• How do you get this data
when there are millions of
transactions per second?
Counters aren’t enough
Why the spike in traffic?
(100Gbit link carrying 14,000,000 packets/second)
15. • Random sampling is lightweight
• Critical path roughly cost of
maintaining one counter:
if(--skip == 0) sample();
• Sampling is easy to distribute
among modules, threads,
processes without any
synchronization
• Minimal resources required to
capture attributes of sampled
transactions
• Easily identify top keys,
connections, clients, servers,
URLs etc.
• Unbiased results with known
accuracy
Break out traffic by client, server and port
(graph based on samples from100Gbit link carrying 14,000,000 packets/second)
sFlow also exports random samples
16. Integrated data model
Packet HeaderPacket Header
Source Destination
TCP/UDP Socket TCP/UDP Socket
MAC Address MAC Address
Sampled Packet Headers
I/F Counters
Power, Temp.
NETWORK
HOST
CPU
Memory
I/O
Power, Temp.
Adapter MACs
APPLICATION
Sampled Transactions
Transaction Counters
TCP/UDP Socket
Independent agents sFlow analyzer joins data for integrated view
19. Monitor
Feedback control loop with sFlow and OpenFlow
low configuration delay
low measurement delay
Together, sFlow and OpenFlow provide the observability and
controllability to enable SDN applications targeting low latency
control problems like load balancing and DDoS mitigation
low planning delay
SDN application
20. packets
decode hash sendflow cache flushsample
NetFlow/IPFIX
send
polli/f counters
sample
• sFlow exports packet samples immediately
• sFlow also exports interface counters
• NetFlow exports flow data on end of flow, active-timeout or inactive-timeout
• NetFlow data generation requires significant resources on switch that can
be better applied to increase size of forwarding table(s)
• OpenFlow metering has similar architecture to NetFlow and similar
limitations
sFlow and NetFlow/IPFIX in a switch
21. InMon sFlow-RT
active timeout active timeout
NetFlow
Open
vSwitch
SolarWinds Real-Time NetFlow Analyzer
• sFlow does not use flow cache, so realtime charts more accurately reflect traffic trend
• NetFlow spikes caused by flow cache active-timeout for long running connections
Rapid detection of large flows
Flow cache active timeout delays large flow detection,
limits value of signal for real-time control applications
23. REST API
Metrics
Flow Definitions
Thresholds
InMonsFlow-RT
REST API
OpenFlowController
Load Balancer DDoS Protection
REST Applications
Open “Southbound” APIs
Data Plane
Control Plane
Hosts
Open “Northbound” APIs
SDN Applications
SDN feedback control applications
24. ovs-vsctl set-controller br0 tcp:10.0.0.1:6633
ovs-vsctl — –id=@sflow create sflow agent=eth0
target=”10.0.0.1:6343” sampling=1000 polling=20
— set bridge br0 sflow=@sflow
Connect switches to central control plane
e.g connect Open vSwitch to OpenFlow controller
e.g. connect Open vSwitch to sFlow analyzer
Minimal configuration to connect switches to
controllers, intelligence resides in external software
25. • DDoS mitigation
• Load balancing large flows
• Optimizing virtual networks
• Packet brokers
Performance aware SDN application examples
Emerging opportunity for SDN applications to leverage
embedded instrumentation and control capabilities and deliver
scaleable performance management solutions
Many more use cases, particularly if you broaden the
scope to the SDDC (software defined data center)
26. Components of a DDoS flood attack
1. Command to attack target sent over
control network
2. Large number of compromised hosts
start sending traffic to target
3.Traffic converges on access link,
overwhelming capacity and denying
access
27. threshold
attack starts
detected
control implemented attack eliminated
http://blog.sflow.com/2013/03/ddos.html
Before
After
Use Case 1: DDoS mitigation
packets/secondpackets/second
sustained 6M packets/second attack
(30 Gigabits/second)
http://packetpushers.net/openflow-1-0-actual-use-case-rtbh-of-ddos-traffic-while-keeping-the-target-online/Also:
28. ECMP/LAG multi-path traffic distribution
http://static.usenix.org/event/nsdi10/tech/full_papers/al-fares.pdf
index = hash(packet fields) % linkgroup.size
selected_link = linkgroup[index]
Hash collisions reduce effective cross sectional bandwidth
1:1 subscription ratio doesn’t eliminate blocking, collision
probabilities are high, even with large numbers of paths
29. Birthday Paradox
What is the chance that at least two people in a room will share a birthday?
50/50 chance with 23 people, virtual certainty with the 90 people in this room.
This is a “paradox” because the probability seems remarkably high considering
that there are 365 possible birthdays (366 if you include Feb 29) and 23 people
represents just over 6% of the theoretical maximum and 90 people is only 25%.
http://en.wikipedia.org/wiki/Birthday_problem
ECMP/LAG/MLAG collision probabilities are surprisingly high
32. VMTo
VM From
FW
LB
a
a b
b c
c
d
d
Virtual network
packet paths
Lack of topology
awareness results
in random
placement ofVMs
Traffic matrix on physical
network appears random
Random traffic patterns
appear to need a completely flat physical
network topology, i.e. non-blocking between
all node pairs (fat tree, CLOS)
- expensive (cost, power, space)
- limited scaleability
- limited flexibility
- not easily achieved in practice (large flows)
33. VMTo
VM From
Largesttenant
Largest tenant
Use Case 3: Network awareVM placement
VM2 VM1VM1 VM2
SDN provides
network topology
and load information
that allowsVMs to be
optimally placed
Resulting sparse, highly structured traffic
matrix efficiently maps into physical
resources, allows SDN controller to
deliver predictable performance and
workload isolation
http://blog.sflow.com/2013/04/multi-tenant-traffic-in-virtualized.html
Extension of OpenFlow to optical
circuit switches allows network to be
rewired for actual demand
34. Traffic is sparse for each tenant
Traffic within each tenant’s virtual network is similarly sparse, e.g.
Hadoop above, or scale out web, cache, storage clusters
http://research.microsoft.com/en-us/UM/people/srikanth/data/imc09_dcTraffic.pdf
35. Use Case 4: Packet broker
ONS 2013: DEMon Software Defined Distributed Ethernet Monitoring System, Rich Groves, Microsoft
http://blog.sflow.com/2013/04/sdn-packet-broker.html
• Offloading basic traffic monitoring to sFlow takes pressure off capture network
• Visibility into traffic volumes before triggering capture
• Trigger capture based on non OpenFlow 12 tuple fields (e.g. tenant IP, VNI etc)
• Trigger on very large match lists (lists of compromised hosts etc.)
38. 10.0.0.16 10.0.0.20 10.0.0.28
XenServer Pool
Demo data from small test lab
10.0.0.30
Hyper-V
VMs: 10.0.0.1,10.0.0.59,10.0.0.114,10.0.0.121,10.0.0.150 - 10.0.0.154,10.0.0.158,10.0.0.160,10.0.0.162
Applications: HTTP, Memcached, PHP, Java
vSwitches: Open vSwitch, Hyper-V extensible vSwitch
Other sFlow sources
10.0.0.253
50. import requests
eventurl = 'http://localhost:8008/events/json?maxEvents=10&timeout=60'
eventID = -1
while 1 == 1:
r = requests.get(eventurl + "&eventID=" + str(eventID))
if r.status_code != 200: break
events = r.json()
if len(events) == 0: continue
eventID = events[0]["eventID"]
events.reverse()
for e in events:
print str(e['eventID']) + ',' + str(e['timestamp']) + ',' +
e['thresholdID'] + ',' + e['metric'] + ',' + str(e['threshold']) + ','
+ str(e['value']) + ',' + e['agent'] + ',' + e['dataSource']
Tail events using HTTP “long” polling
extras/tail_log.py
51. Define flow keys
DDoS Protection
define address groups
define flows
define thresholds
while(running) {
receive threshold event
monitor flow
deploy control
monitor flow
release control
}
OpenFlow
Controller
REST API
sFlow-RT
REST API
1
2
3
4
6
5
8
7
REST operation flow chart
52. Large flow detection script (initialization)
import requests
import json
rt = 'http://localhost:8008'
groups = {'external':['0.0.0.0/0'],'internal':['10.0.0.0/8']}
flows = {
'keys':'ipsource,ipdestination',
'value':'frames',
'filter':'sourcegroup=external&destinationgroup=internal'}
threshold = {'metric':'ddos','value':400}
r = requests.put(rt + '/group/json',data=json.dumps(groups))
r = requests.put(rt + '/flow/ddos/json',data=json.dumps(flows))
r = requests.put(rt + '/threshold/ddos/
json',data=json.dumps(threshold))
...
extras/ddos_log.py
53. Large flow detection script (monitor events)
...
eventurl = rt + '/events/json?maxEvents=10&timeout=60'
eventID = -1
while 1 == 1:
r = requests.get(eventurl + "&eventID=" + str(eventID))
if r.status_code != 200: break
events = r.json()
if len(events) == 0: continue
eventID = events[0]["eventID"]
events.reverse()
for e in events:
thresholdID = e['thresholdID']
if "ddos" == thresholdID:
r = requests.get(rt + '/metric/' + e['agent'] + '/' + e['dataSource'] + '.'
+ e['metric'] + '/json')
metrics = r.json()
if len(metrics) > 0:
evtMetric = metrics[0]
evtKeys = evtMetric.get('topKeys',None)
if(evtKeys and len(evtKeys) > 0):
topKey = evtKeys[0]
key = topKey.get('key', None)
value = topKey.get('value',None)
print e['metric'] + "," + e['agent'] + ',' + key + ',' + str(value)
54. Next Steps
Build your own test bed:
1. sFlow-RT is already installed on your laptop, capable of monitoring thousands of
switches (remember to turn off demo.pcap and enable UDP port 6343 on your firewall)
2. Enable sFlow in your network (OVS, Hyper-V, physical switches, http://sflow.org/
products/network.php)
3. Install Host sFlow agents http://host-sflow.sourceforge.net/ + application agents:Apache,
NGINX,Apache, HAProxy etc. http://host-sflow.sourceforge.net/relatedlinks.php
Engage with the broader sFlow community:
https://lists.sourceforge.net/lists/listinfo/host-sflow-discuss
http://groups.google.com/group/sflow
4.You don’t have to have access to a physical test lab, build a Mininet / Open vSwitch virtual test
lab, e.g. http://blog.pythonicneteng.com/2013/05/pytapdemon-part-3-pro-active-monitoring.html
http://groups.google.com/group/sflow-rt
Find out more about sFlow:
http://sflow.org/
http://blog.sflow.com/