1
6TiSCH
(IPv6-based)
Deterministic WSN
for the Industrial Internet
Pascal Thubert
Cisco Systems
January 14th, 2015

2
Deterministic Networking
Industrial Internet
Technical choices for the Industrial Internet
6TiSCH in a nutshell
6TiSCH Architecture deep dive
6TiSCH Applications

3
Deterministic Networking

4
New level of (Deterministic?) guarantees
A differentiator for high-end forwarding engines
Sharing physical resources with classical networking
For traffic known a priori (control loops…)
Time Synchronization and global Schedule

5
End-to-End latency enforced by timed pause at station
Periodic trains along a same path and same schedule
Collision avoidance on the rails guaranteed by schedule
5

6
A bus every T. minutes, Stop A to Stop B in X. minutes
Worst end-to-end delivery time < (T + X)
Every packet in an Airbus 380 takes a bus across the fabric
6

7
Two classes of bleeding-edge customers, Industrial and Audio/Video.
Both have moved into the digital world, and some are using packets, but now they all
realize they must move to Ethernet, and most will move to the Internet Protocols.
1. Industrial: process control, machine control, and vehicles.
At Layer 2, this is IEEE 802.1 Time-Sensitive Networking (TSN).
Data rate per stream very low, but can be large numbers of streams.
Latency critical to meeting control loop frequency requirements.
2. Audio/video: streams in live production studios.
At Layer 2, this is IEEE 802.1 Audio Video Bridging (AVB).
Not so many flows, but one flow is 3 Gb/s now, 12 Gb/s tomorrow.
Latency and jitter are important, as buffers are scarce at these speeds.
3. Vehicule, SmartGrid: streams in live production studios
Norm Finn draft-finn-detnet-problem-statement-01 7

8
Back-of-the-envelope calculations:
1. Industrial:
Automotive factory floor: 1000 networks • 1000 packets/s/network •
100,000 s/day = 1011 packets/day.
Machine fails safe when 2 consecutive packets are lost.
At a random loss ratio of 10–5, 10–10 is chance of 2 consecutive losses.
1011 packets/day • 10–10 2-loss ratio = 10 production line halts/day.
In extreme cases, lost packets can damage equipment or kill people.
2. Audio video production: (not distribution)
1010 b/s • 10 processing steps • 1000 s/show = 1014 bits = 1010 packets.
Waiting for ACKs and retries = too many buffers, too much latency.
Lost packets result in a flawed master recording, which is the user’s end product.
Norm Finn draft-finn-detnet-problem-statement-01 8

9
1. Zero congestion loss.
This requires reserving resources along the path. (Think, “IntServ” and “RSVP”) You cannot guarantee
anything if you cannot say, “No.”
This requires hardware in the form of buffers, shapers, and schedulers. Overprovisioning not useful: its
packet loss curve has a tail.
Circuits only scale by aggregation in to larger circuits. ( MPLS? Others?)
0 congestion loss goes hand-in-hand with finite guaranteed latency, also of importance to the users.
2. Seamless redundancy.
1+1 redundancy: Serialize packets, send on 2 (or more) fixed paths, then combine and delete extras.
Paths are seldom automatically rerouted.
0 congestion loss means packet loss is failed equipment or cosmic rays.
Zero congestion loss satisfies some customers without seamless redundancy. The reverse is not true in a
converged network—if there is congestion on one path, congestion is likely on the other path, as well.
draft-finn-detnet-problem-statement-01Norm Finn 9

10
It’s all about
Real boxes and links
Real buffers and queues
Real packets
Real Time
10
It’s not about
Classical Layers
Standard bodies
QoS and stat mux

11
• Single nailed-up path
Sender
(Talker)
NIC
NIC
Flow ID
Receiver
(Listener)
Per-Flow State
11

12
Replication & Elimination of individual packets
Sender
(Talker)
NIC
NIC
Flow ID
Receiver
(Listener)
Per-Flow State
12

13
Control (southbound)
interface
Controller
Service (northbound)
interface
Topology,
resources
Sender
(Talker)
NIC
NIC
Receiver
(Listener)
13

14
Control (southbound)
interface
User/app/ser
vice
Per-Flow State
TSpec
User/app/ser
vice
Controller
Service (northbound)
interface
?
Reservation
request
Sender
(Talker)
NIC
NIC
Flow ID
Receiver
(Listener)
14

15
Sender
(Talker)
User/app/ser
vice
NIC
NIC
Per-Flow State
TSpec
User/app/ser
vice
Controller
Flow ID
?
Service (northbound)
interface
Receiver
(Listener)
?
?
UNI
interface
Control (southbound)
interface
UNI
interface 15

16
Same as normal networking, but with the following features for critical data streams:
1. Time synchronization for network nodes and hosts to better than 1 µs.
2. Software for resource reservation for critical data streams (buffers and
schedulers in network nodes and bandwidth on links), via configuration,
management, and/or protocol action.
3. Software and hardware to ensure extraordinarily low packet loss ratios, starting
at 10–6 and extending to 10–10 or better, and as a consequence, a guaranteed
end-to-end latency for a reserved flow.
4. Convergence of critical data streams and other QoS features (including ordinary
best-effort) on a single network, even when critical data streams are 75% of the
bandwidth.
Norm Finn draft-finn-detnet-problem-statement-01 16

17
• IEEE Std 802.1BA-2011 “Audio Video Bridging (AVB) Systems”
A profile of a number of standards, picking out required options, special initialization parameters, etc., required for AVB-compliant
bridges and end stations.
An “AVB Device” is a device conforming to 802.1BA.
• IEEE Std 802.1AS-2011 “Timing and Synchronization for Time-Sensitive Applications in Bridged Local
Area Networks”
A plug-and-play profile of IEEE 1588, including master clock selection, link discovery, and automatic creation of a tree to distribute
the clock signal.
• IEEE Std 802.1Qat-2010 “Stream Reservation Protocol (SRP)”
The software protocol for making stream reservations
Has been rolled into 802.1Q as Clauses 34 and 35 of IEEE Std 802.1Q-2011.
• IEEE Std 802.1Qav-2009 “Forwarding and Queuing Enhancements for Time-Sensitive Streams”
A special credit-based hardware shaper for bridges and end stations that gives better latency guarantees than the usual shapers.
Has been rolled into 802.1Q as Clause 34 of IEEE Std 802.1Q-2011.
• IEEE Std 1722-2011 “Layer 2 Transport Protocol for Time Sensitive Applications in a Bridged Local Area
Network”
A Layer 2 transport protocol carrying a time-to-display stamp on each packet.
• (All 802 standards (not 1722) are free 6 months after publication at http://standards.ieee.org/about/get.)
Norm Finn draft-finn-detnet-problem-statement-01 17

18
• P802.1AS-REV* “Timing and Synchronization”
Revision of 802.1AS making it clear that it can run on a router as easily as on a bridge.
• P802.1Qbu* “Frame Preemption”
Amends 802.1Q to support 802.3br
• P802.3br “Interspersed Express Traffic”
One level of transmission preemption – interrupts transmission of an ordinary frame to transmit an “express” frame, then resumes the
ordinary.
802.3 document, not an 802.1 document.
• P802.1Qbv* “Enhancements for Scheduled Traffic”
Runs the 8 port output queues of a bridge on a rotating schedule.
• P802.1Qca* “Path Control and Reservation”
Enhances 802.1 ISIS to create multiple paths through a network.
• P802.1CB* “Seamless Redundancy”
Defines the sequence-split-recombine method for reliability improvement.
Stand-alone document. NOT an amendment to 802.1Q.
• P802.1Qcc* “Stream Reservation Protocol (SRP) Enhancements and Performance Improvements”
For more streams, faster convergence, less chattiness, and maybe more.
* For the necessary password, Google “p802.1 username password”
Norm Finn draft-finn-detnet-problem-statement-01 18

19
Industrial Internet

20
Challenge: harness innovation
More efficient operations
New and/or improved experience
Beyond Control and Automation
Optimize processes (by 1%?)
Leveraging IT, Live big data and Analytics

21
Control Loops
global optimization of routes for jitter an latency – thus computed by a PCE -,
flow (over) provisioning, duocast for 1 hop (1Hz and 4Hz, no routing) with static multipath hard slot allocation.
Control Loop plan B
self-healing -thus distributed- routing, RPL
Supervisory Flows
no goal: requires determinism up into the backbone
Management
a separate topology – a RPL instance - that does not break.
OF?, root selection?
Alerts
bursty, unexpected, on-demand slot allocation, prioritization

22
Large Scale Monitoring / Live Big Data
stuff like corrosion? low cost scalability – this distributed routing, soft slots?
Device (Limited) Mobility
E.g. Cranes spanning area (ship)
with linear and circular motion that are mobile within a range
Mobile Worker
additionally provision slots ready for mobile workers) does not need deterministic since human
Wide Subnets
thousands of devices– thus a backbone, multiple BBRs for 1 LLN –
within a subnet – to avoid renumbering though not always the case, sometimes isolation is wanted
Non-Production Episodes
fast and autonomic behavior – again distributed routing
Coexistence with Legacy Devices
a common management above, coarse level makes sense (channels backlisting but no more…single admin is good)

23
• Not Process Control but “Missing Measurements”
Reliability and availability are important, which implies
Scheduling and admission control
• Scalability
10’s of thousands of new devices
• Deployment cost factor is key
For Emerson this is the second layer of automation:
Typically missing are the measurements you need to
monitor the condition of the equipment--temperature,
pressure, flow and vibration readings you can use to
improve site safety, prevent outages and product losses,
and reduce maintenance costs of equipment such as
pumps, heat exchangers, cooling towers, steam traps and
relief valves.

24
15M to 115M Analytics
related connections*
Classical Monitoring
only doubles
Analytics related M2M
connections surge
* Source:
ABI Research

25
* Source:
ABI Research
Big Data and
Analytics
“Without adequate analytics clout available, and the right practices to take advantage of it, the companies
rolling out M2M solutions will be destined to be stuck in the lower realms of applications: monitoring, reporting,
and simple rules-based actions.” *
Large Scale
monitoring
6TiSCH

26
Industrial
Internet
Control Data
Office Data
Business Data
Nb of Devices (order of magnitude)
ProcessCriticality
1 000250 10 000 1 000 000
ISA100.11a
WiFi
WirelessHART
6TISCH / PCE
LPWA
6TISCH / RPL
CG-Mesh

27
Maintenance and operation
represent 75% of the Total
equipment cost
Downtime
Maintenance & operation COST
Corrective maintenance
Preventive
Maintenance
Predictive
Maintenance
Actionable Prescriptions
 Deployment of Wireless sensors is seen as an efficient solution

28
Data: Capillary Wireless
Acquisition of large amounts of
live Big Data
Information: Fog DiM to add
descriptive meta-tags,
allowing for compression
and correlation
Knowledge: Prediction from
self-learned models and
Knowledge Diffusion
Wisdom: Machine Learnt
Expertise, auto-Generating
Prescriptions and Actionable
Recommendations
(Data)
ACQUISITION
(Knowledge)
DIFFUSION
(Wisdom)
OPEN LOOP
(Information)
AGGREGATION
6TiSCH Data in
Motion / Fog
Learning
Machines
/ Cloud

29
• IIC website http://www.iiconsortium.org
• New York Times http://bit.ly/IIC-03272014
• IEEE http://flip.it/zbfnU
• Huge tutorial http://bit.ly/iic-319
• McRock's Industrial Internet of Things Report 2014
• AT&T, CISCO, GE, IBM AND INTEL FORM INDUSTRIAL INTERNET CONSORTIUM TO
IMPROVE INTEGRATION OF THE PHYSICAL AND DIGITAL WORLDS March 27, 2014

30
Customers after next % point of operational optimization:
Requires collecting and processing of live “big data”, huge amounts of missing
measurements by widely distributed sensing and analytics capabilities.
Achievable by combination of the best of IT and OT technologies together, forming
the IT/OT convergence, aka Industrial Internet.
Architectural approach, standards, Industry adoption needed to embrace radical
changes happening in IT networking technologies. Products are following.
Secured-by-default model required throughout network lifecycle.
6TiSCH extends Deterministic Wireless Industrial Networking technologies to also
reach higher scales at lower costs (but then, guarantees as well).
Operational technology (OT) is hardware and software that detects or causes a change through the
direct monitoring and/or control of physical devices, processes and events in the enterprise.

31
Technical choices for the
Industrial Internet

32
Link State requires full & constant topology awareness
=> 6TiSCH / RPL chose Distance Vector
Wireless scales but power constraints and fuzzy topology
=> Different risks, value in diversity
TDMA requires timesync, CSMA-CA needs idle time
 Different usages, deterministic vs. best effort
Reactive mixes routing and forwarding (eg IPv6 ND)
Centralized optimizes, Distributed scales economically
=> Different usages, deterministic vs. best effort
Link State ?
Wired?
CSMA?
Reactive ?
Centralized
?
Single
Hop ?
Spatial reuse
=> Wired backhaul a good idea in any case

33
Reaching farther out
New usages / types of devices
Global Coverage from Near Field to Satellite via 3/4G
BUT
Lack of trust in Industrial vs. Wired
Multiple Interferers, ISM band crowded
Issues with IPv6 for Scalability and Mobility
New level of cost effectiveness
Deploying wire is slow and costly
Low incremental cost per device

34
Higher speed does not necessarily mean more energy
Energy budget: Power vs. Time
LP WiFi a contender to 802.15.4
802.11ah designing for 2 hops
Interesting developments in LowPower-WideArea
ultra narrow band (e.g. SigFox 100KHz)
802.15.4K (e.g. OnRamp Wireless)
LOng RAnge (Cycleo / Semtech, 2.4GHz

35
16channeloffsets
e.g. 31 time slots (310ms)
A
BC
D
E
F
G
H
I
J
• Schedule => direct trade-off between throughput, latency and power consumption.
• A collision-free communication schedule is typical in industrial applications.
• But requires network synchronization, and de-sync means long isolation

36
Channel Hopping :
retry around interference,
round robin strategy
Time Slotted (or Synchronized) :
Deterministic: through TDM, Synchronized + Time formatted in SlotFrame(s)
Tracks: below IP, can be orchestrated by a third party like virtual circuits
Slotted: benefits of slotted aloha vs. aloha => reduce chances of collisions
Battery operation: when traffic profile is known, devices only wake upon need

37
Ethernet
Complexity
DV
Single
Hop ?
CSMA-CACSMA-CD
Wired
Single
Hop ?
6TiSCH
Reactive
Distributed
?
CG-Mesh LTE
ISA100.11a
YES NO
WiFi

38
Direct Communication when possible yields:
• Simpler design
• Lower OPEX
• Multipath issues and uneven transmission quality
Low Power TSCH mesh is a complex technology adapted to:
• Range extension with Spatial reuse of the spectrum
• Centralized Reservation for deterministic critical data streams.
• Separation of resources for classical IP routing (RPL)
Norm Finn draft-finn-detnet-problem-statement-01 38

39
6TiSCH in a nutshell

40
IEC based on HART 7.1.
TDMA
fixed time slots (10ms)
Mesh only
Shipped YE-2008.
Vendor driven
Emerson, E&H, ABB,
Siemens
IEC based on 2011 revision
TDMA+CSMA
Var. time slots
Star, mesh and hybrid topology
IPv6, 6LoWPAN, TCP-friendly
Shipped mid-2010
Mostly user driven
Honeywell, Yokogawa, Invensys
IEC 62601
WIA-PA
IEC 62591 IEC 62734
ISA100.11a
Alternate from China
Star, mesh and hybrid
topology
Standardization work
started in 2006.

41
6TiSCH
WiHART WIA-PAISA100
Common Network
Management and Control
HART WIA-PAISA100
MGT /
CTRL.
APPLI.
HART WIA-PAISA100
Better Co-ExistenceTRIPLE OPEX + interferences
Wia-PAISA100NETW. Wireless
HART

42
• Industrial requires standard-based products
• Must support equivalent features as incumbent protocols
• Must provide added value to justify migration
• 6TiSCH value proposition
Design for same time-sensitive MAC / PHY (802.15.4e TSCH)
Direct IPv6 access to the device (common network mgt)
Distributed routing & scheduling for scalability (for monitoring)
Large scale IPv6 subnet for mobility (50K +)

43
Active IETF WG, 6 WG docs in or close to last call
Define an Architecture that links it all together
Align existing standards
(RPL, 6LoWPAN, PANA, RSVP, PCEP, MPLS) over 802.15.4e TSCH
Support Mix of centralized and distributed deterministic routing
Design 6top sublayer for L3 interactions
Open source implementations (openWSN…)
Multiple companies and universities participating

Cisco Confidential 44© 2013-2014 Cisco and/or its affiliates. All rights reserved.
6TiSCH Client stack
Centralized route and
track computation
and installation
Management and
Setup
Discovery
Pub/Sub
Authentication for
Network Access
Wireless ND
(NPD proxy)
Time Slot
scheduling and
track G-MPLS
forwarding
Distributed route and
track computation and
installation
Distributed route and
track computation and
installation
IEEE 802.15.4e TSCH
6LoWPAN HC
IPv6
RPL
6top
TCP UDP ICMP CCAMP
PCEP/PCC CoAP/DTLS AAA 6LoWPAN ND
}

45
Limit of IP direct connectivity
ISA100.11a ||
WirelessHART
ISA100.11a ||
WirelessHART
Sensor
Wireless Control
Loop
Actuator
PLC
Root AP
Management
Wireless Controller
NAC
/Firewall
- Control loop limited to subnet
- No end-to-end IP connectivity
- No single view management
- No Cisco presence in control loop
Mesh AP +
Manager +
App GW
VPN Concentrator

46
6TiSCHSensor
Actuator
Backbone
Router
PCE
Intelligent device
Management
Wireless Controller
Backbone
Router
Limit of IPv6 subnet(s)Limit of IPv6 subnet
End to end IPv6 routing
NAC
/Firewall
VPN Concentrator

47
6TiSCH
6TiSCHSensor
Actuator
Backbone
Router
Management
Wireless Controller
Backbone
Router
Single Multilink IPv6 Subnet with free inner mobility (same subnet == no renumbering)
Virtual Service Engine
(virtual PLC, PCE …)
+ IPv6 ND registrar
+ Network Function
Virtualization (NFV)
NAC
/Firewall
VPN Concentrator
IPv6 ND registration and proxy operation

48
So WHY ?
• From a technology point of view
Scheduled network + PCE
 Emulate Industrial WSNs including TSN aspects and limitations (mobility, scalability)
Peristaltic Scheduling + Distributed routing and scheduling (RPL) + Efficient IPv6 ND (WiND)
 Large scale monitoring for Industrial Internet in Co-Existence with Industrial WSNs
• From a user point of view
Common Network management (over IPv6)
Open standards (IETF, IEEE, IEC)
Reduced OPEX (though we suggest end-to-end pcpl & multiple apps as opposed to single protocol)
• From a market point of view
IPv6 end-to-end  enable control loop virtualization
Support dynamic scheduling and deterministic on a same network
Single administration, lower OPEX

49
Deterministic IPv6 over IEEE802.15.4e TimeSlotted Channel Hopping (6TiSCH)
The Working Group will focus on enabling IPv6 over the TSCH mode of the
IEEE802.15.4e standard. The scope of the WG includes one or more LLNs, each
one connected to a backbone through one or more LLN Border Routers (LBRs).
Active drafts
http://tools.ietf.org/html/draft-ietf-6tisch-terminology
http://tools.ietf.org/html/draft-ietf-6tisch-tsch
http://tools.ietf.org/html/draft-ietf-6tisch-architecture
http://tools.ietf.org/html/draft-ietf-6tisch-minimal
http://tools.ietf.org/html/draft-ietf-6tisch-6top-interface
http://tools.ietf.org/html/draft-ohba-6tisch-security
http://tools.ietf.org/html/draft-ietf-6tisch-coap

50
Converged Plant Network
High availability
Flow Isolation
Guaranteed Bandwidth
IP based Control Network
Autonomic, zero touch commissioning
Time Sensitive Networking for critical apps
Packet Reliability
IPv6-based Wireless Field Network
Deterministic, Autonomic, Secure
Large Scale for Monitoring (RPL)
Backward Compatible (with ISA100 or HART)
10’s
of K
10’s
100’s
Control
network
Converged plant net
Wireless Field network

51
6TiSCH
Architecture
deep dive

52

53
1. Scans all channels to detect
Extended Beacon
2. Synchronizes
3. Parses beacons for schedule
4. Performs Join Process (AAA)
5. Renumbers and DADs
6. Route setup (eg RPL DAO)
6TiSCH forms a single multilink
subnet, no loss of sync, no
renumbering
Discovery steps; new/moving device:

54
Neighbors are discovered at L2 from 15.4e MAC Extended Beacon
Time parent selection based on a join priority field in the EB
Join priority as a distance but no real loop avoidance
=> time loops and
=> node isolation
When a new node B joins the network
it needs to wait for an EB to align itself to the sequence of slots
EBs are sent using TSCH
Node cannot know in advance the channel to listen
it may take a long time for B before catching an EB
The problem exacerbates for bootstrapping networks with many hops
After a node joins
synchronization should be maintained using data/ack and keepalive handshakes

55
One RPL instance dedicated to time parenting
RPL Rank used as join priority, set once once joined that timing instance
0xFF (Infinite) as long as not joined that instance => not a suitable time parent
RPL provides a better loop avoidance yet may let temporary loops
If DAGMaxRankIncrease is non-zero
8.2.2.4. Rank and Movement within a DODAG Version
If multiple control messages are lost
Loops will be detected reactively on traffic
3.2.7. Data-Path Validation and Loop Detection
11.2. Loop Avoidance and Detection
Q: Proscribe DAGMaxRankIncrease > 0 ?

56
56
Multiple uncoordinated DODAGs with
independent roots (differing DODAGIDs)
DODAGs partition the network
No connectivity between DODAGs
to distribute time:
e.g. roots are GPS-based time sources
Nodes may jump to alternate DODAG
A single DODAG with a single root
May be used for time synchronization
Time may drift with distance if large mesh
A single DODAG with a virtual root that
coordinates LLN sinks (with the same
DODAGID) over a backbone network
May be used for time synchronization
Time can be synchronized over the backbone
e.g. Precision Time Protocol (IEC 61588, IEEE 1588v2)
6TiSCH
LLN
6TiSCH LLN
6TiSCH
LLN
Common time
source
Floating
root

57

58
Centralized
God’s view optimization
Multipath redundancy
Deterministic (optimized)
Virtualization
Distributed
Autonomic & Mobile
Highly available (DARPA)
Deterministic
Scalability

59
Distance Vector + stretch
Peer only with parents
DV + Non-storing mode
Lazy Update & datapath valid.
Constrained instances, TID
Req coupling with LISP/NEMO
Dynamic topologies
Peer selection
Constrained Objects
Fuzzy Links
Routing, local Mobility
Global Mobility
Low Power Lossy Nets Addressed in RPL ?

60
In the context of routing, a Directed
Acyclic Graph (DAG) is formed by a
collection of vertices (nodes) and edges
(links).
Each edge connecting one node to
another (directed) in such a way that it is
not possible to start at Node X and follow
a directed path that cycles back to Node
X (acyclic).
A Destination Oriented DAG (DODAG) is
a DAG that comprises a single root node.
Here a DAG that is partitioned in 2
DODAG
Clusterhead
5
4
4
0
1
3
1 1
2
2
2
2
2
3
3
3
3
3
3
2
4
4
5
0
6
5
4

61
In Green: A’s subDAG.
Impacted if A’s connectivity is
broken
Domain for routing recovery
In Red: B’s fanout DAG
(or reverse subDAG)
Potential SPAN on B’s DAO
Thus potential return paths
Fanout must be controlled to limit
intermediate states
Clusterhead
5
4
4
0
1
3
1 1
2
2
2
2
2
3
3
3
3
3
3
2
4
4
5
0
6
5
4
A
B

62
e.g. ARC of siblings
Allows more alternates
ARCs are walked with no routing
progress
Forwarding over DAG AND ARCs
Reduces congestions of upper links
of the DAG
Still LORA for P2P
IGP subarea (bidirectional)
Link selected and oriented by TD
Potential link
Clusterhead
5
Clusterhead
0
1
1
1
2
2
2
2
2
3
3
3
3
3
3
2
3
5
4
4
4
4

63
Either
routes in the RIB / VRF or
G-MPLS forwarding state
Indexed by tuple (IPv6, InstanceID)
Steps:
1. A needs a route to B
2. A selects a local InstanceID
3. A requests a computation from PCE
4. PCE returns route steps (and timeslots)
5. A install routing/forwarding states along
returned route or use source routing
Track
PCEP request Potential link
Clusterhead
5
Clusterhead
0
1
1
1
2
2
2
2
2
3
3
3
3
3
3
2
3
5
4
4
4
4
PCE
BA

64
The RPL instance ID allows different routing optimizations, constraints and policies.
D Local Instance ID 0..64
0 1 2 3 4 5 6 7
0 Global Instance ID 0..128
0 1 2 3 4 5 6 7
1
The RPL instance ID is encoded in 1 octet. The first bit indicates whether Global or Local.
“A local RPLInstanceID is autoconfigured by the node that owns the DODAGID and it MUST be unique for that
DODAGID. The DODAGID used to configure the local RPLInstanceID MUST be a reachable IPv6 address of the node,
and it MUST be used as an endpoint of all communications within that Local instance.”
Inside a packet: “If the 'D' flag is set to 1, then the destination address of the IPv6 packet MUST be the DODAGID. If the
'D' flag is cleared, then the source address of the IPv6 packet MUST be the DODAGID.”
6TiSCH extends RPL’s language of DODAGID to route (reservation) endpoint.

65
128 global instances per network
Indexed by tuple (IPv6, InstanceID)
Running as Ships-in-the-night
1 instance = 1 VRF = 1 « L3 vlan »
Serving different Objective Functions
Using different metrics
Can be shared between RPL and PCE
RPL along a DODAG
PCE adding orthogonal shortcuts
Clusterhead
0
1
1
1
2
2
2
2
2
3
3
3
3
3
3
2
3
5
4
4
4
4
Clusterhead
Constrained instance
Default instance
Potential link
A

66
Clusterhead
5
Clusterhead
0
1
1
1
2
2
2
2
2
3
3
3
3
3
3
2
3
5
4
4
4
4
64 local Instances
per IPv6 source address
Indexed by tuple (IPv6, InstanceID)
Used by RPL or PCE
RPL: for P2P applications
RPL: to index RSVP path
PCE: Serves as Track ID,
included in PCEP request
from the source device
Track –PCE)
RSVP reservation for RPL
Potential link

67
RPL is an extensible proactive IPv6 distance vector protocol
Supports MP2P, P2MP and P2P between devices (leaves) and a root (border router)
P2P reactive extension
RPL specifically designed for “Lossy” networks
Agnostic to underlying link layer technologies (802.15.4, PLC, Low Power Wireless)
RFC 6206: The Trickle Algorithm
RFC 6550: RPL: IPv6 Routing Protocol for LLNs
RFC 6551: Routing Metrics Used for Path Calculation in LLNs
RFC 6552: Objective Function Zero for the Routing Protocol for LLNs
RFC 6553: RPL Option for Carrying RPL Information in Data-Plane Datagrams
RFC 6554: An IPv6 Routing Header for Source Routes with RPL
RFC 6719: MRHOF Objective Function with hysteresis
RPL is pronounced
“Ripple”

68

69
A
B
X Y
U
V
1TX
1TX
1RX
1RX
shaping
QoS
RED
1+1 TX
Packets
are
serialized

70
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
UYA X

71
1TX
1TX
1RX
1RX
Dest MAC set to
broadcast 0xFFFF
DSCP Deterministic
Dest MAC restored by
6top at final destination
Dest MAC not changed
DSCP Deterministic
2TX 2RX
A
B
X Y
U
V

72
1TX
1TX
1RX
1RXAdditional retries
Using L3 bundle
Dest MAC set to Y
DSCP Deterministic
Transmission failure
Over track slots
Retries exhausted over
Track L2 bundle
2TX 2RX
A
B
X Y
U
V
If arrival in time
Dest MAC reset to
broadcast 0xFFFF
DSCP Deterministic
Placed back on track
Dest MAC set to
broadcast 0xFFFF
DSCP Deterministic
1TX 1RX

73
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
UYA X

74
ISA100.11a /
WIHART
15.4e TSCH 15.4e TSCH 15.4e TSCH15.4e TSCH
15.4 PHY 15.4 PHY 15.4 PHY15.4 PHY
CoAP
UDP
IPv6
6LoWPAN-HC
6top
CoAP
UDP
IPv6
6LoWPAN-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LoWPAN-HC
6top
15.4 PHY
15.4e TSCH
ISA100.11a /
WIHART
UDP
IPv6
6LoWPAN-HC
6top
CoAP
Dest MAC set to
broadcast
0xFFFF
Dest MAC not
changed
Dest MAC restored
by 6top

75
1TX
1TX
1RX
1RX
Dest MAC restored
by 6top
Dest MAC set to X
DSCP not Deterministic
2TX 2RX
P
B
X Y
U
V
A
Q
Packet to be routed via Y
Dest MAC set to Y
Placed of deterministic track
Packet extracted from track due
to dmac == Y and/or DSCP and
then routed to a next hop != V
No frame in slot
Associated to
deterministic

76
A
B
X Y
U
V
1TX
1RX
1RX
Next
fragment
follows
n TX
First
fragment
installs
states

77
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
15.4 PHY
15.4e TSCH
CoAP
UDP
IPv6
6LP-HC
6top
UYA X
1st1st
NextNext

78

79
Channeloffset
Upto16channel
slotOffset, 0 .. N
The matrix repeats itself over time. CDU is defined by 6TiSCH and represents the global characteristics of the
network, channel unused, duration of the timeslot and number of timeslots per iteration.
6TiSCH defines a new global concept that is called a Channel distribution/usage (CDU) matrix; a
Channel distribution/usage (CDU) matrix is a matrix of so-called “cells” with an height equal to the
number of available channels (indexed by ChannelOffsets), and a width in timeslots that is the period of
the network scheduling operation (indexed by slotOffsets).
Timeslot width, typically 10-15ms each in 802.15.4e
Cell (slotOffset,
channelOffset)

80
A Slotframe is a MAC-level abstraction that is internal but common to all nodes and contains a series of
timeslots of equal length and priority. It is characterized by a slotframe_ID, and a slotframe_size. Multiple
slotframes can coexist in a node’s schedule, i.e., a node can have multiple activities scheduled in different
slotframes, based on the priority of its packets/traffic flows. The timeslots in the Slotframe are indexed by
the SlotOffset; the first timeslot is at SlotOffset 0.
Channeloffset
Upto16channel
slotOffset, 0 .. N
TimeSlot
Slotframe
CDU
matrix
Cell (slotOffset,
channelOffset)
Timeslot width, typically 10-15ms each in 802.15.4e

81
A Chunk is a well-known list of cells, well-distributed in time and frequency, within the CDU matrix; a chunk
represents a portion of the CDU. The partition of the CDU in chunks is globally known by all the nodes in the
network to support the appropriation process, which is a negotiation between nodes within an interference
domain. A node that manages to appropriate a chunk gets to decide which transmissions will occur over the
cells in the chunk within its interference domain. Ultimately, a chunk represents some amount of bandwidth
and can be seen as the generalization of a transmission channel in the time/frequency domain.
Chunk 1
Chunk 2
...
Chunk 8
CDU matrix, 6 channels, 7 timeslots, 8 chunks, represented from ASN= 0

82
The chunks are defined at the epochal time but the 802.15.4e operation is an iterative process that repeats
over and over. Typically, the effective channel for a given transmission is incremented by a constant that is
prime with the number of channels, modulo the number of channels at each iteration. It results that the
channel of a given transmission changes at each iteration and the matrix virtually rotates.
To illustrate that rotation, we represent above where the transmission would happen in the second
iteration, for a CDU matrix of 8 channels if we increment the effective channel by 3 at each iteration:
Chunk 1
Chunk 2
...
Chunk 10
Effective channel offsets for chunks 1 .. 10 at the reference epoch and after one iteration

83
slotOffset, 0 .. N
High Priority
RCV XMIT RCVSlotframe 1
slotOffset, 0 .. M
Low Priority
RCV RCV RCV XMIT XMITSlotframe 2
Will receive
Will receive if nothing to xmit
Will xmit
Will receive
Will receive
Decision to transmit / rcv
made at the beginning of
time slot based on
existing frames in queues
and slotframe priorities

84
11 12 13
25242322
3534333231
464544434241

85

86
Parent schedule
Child 1 schedule
Child 2 schedule
Child 3 schedule
Xmit mcast rcv rcv xmit rcv xmit
rcv rcv xmit
rcv xmit rcv
rcv xmit
Slotframe
Say a parent
appropriates the
pink chunk from
the CDU matrix
above. Now the
parent gets to
decide when the
pink cells are
used and by
which node.
A device maintains a schedule that dictates its
transmissions and receptions inside the slotframe.
The RPL parent
allocates timeslots
between itself and
its children. The
parent takes the
timeslots off a
chunk that it has
appropriated.

87

88
6TiSCH finally defines the peer-wise concept of a bundle, that is needed for the communication between
adjacent nodes.
A Bundle is a group of equivalent scheduled cells, i.e. cells identified by different [slotOffset,
channelOffset], which are scheduled for a same purpose, with the same neighbor, with the same flags, and
the same slotframe.
The size of the bundle refers to the number of cells it contains. Given the length of the slotframe, the size
of the bundle translates directly into bandwidth, either logical, or physical. Ultimately a bundle represent a
half-duplex link between nodes, one transmitter and one or more receivers, with a bandwidth that amount
to the sum of the timeslots in the bundle. Bundles thus come in pairs, an incoming and an outgoing.
A bundle is globally identified by (source MAC, destination MAC, trackID)
timeslot
bundle

89
We use a well-known constant as trackId for a L3 link, that I’ll refer as NULLT.
So an IP link between adjacent nodes A and B comprises 2 bundles, (macA, macB, NULLT) and (macB,
macA, NULLT).
timeslot
bundle
timeslot
bundle
outgoingIncoming
IP layer
AA
B

90
Along a that has a segment …A->B->C… , there are 2 bundles in node B,
incoming = (macA, macB, trackId) and outgoing = (macB, macC, trackId).
timeslot
bundle
timeslot
bundle
outgoingIncoming
C
B
A

91

92
Multiple L3
Multicast
messages
RA
Protections: MLD snooping for SNMA (limited)
and RS. Cisco only: IPv6 FHS (SISF) in WLC
1. MAC address flooded over
spanning tree for L2
switching
2. Device sends multiple
multicasts
3. L2 fabric handles as
broadcast
4. Broadcast clogs the wireless
access at low access speed
VM, NFV,Wireless or IoT device moves:
Multiple L2
Broadcast
messages

93
Authoritative Registrar(s)
MIPv6 HA, 6LBR, interface to external services
Intermediate Registrars
6LR, NEAR, Optionally ND proxy
Backbone Routers
RPL root, ND proxy
Legacy IPv6 devices
Authoritative
Registrar
Authoritative
Registrar / 6LBR
Intermediate
Registrar / 6LR
Intermediate
Registrar /
opt ND proxy
Backbone router
(ND proxy)

94
6LR Wireless
Router
IPv6 registrar
(ND proxy)
6LR Wireless
Router
IPv6 registrar
(ND proxy)
6LBR/RPL Root 6LBR/RPL Root
6LR Wireless
Router
6LR Wireless
Router
Backbone
Gateway
Gateway

95
Multicast Avoidance
Registration for DAD
Binding (location, owner, MAC) to IPv6 address
Registrar hierarchy for lookups and policy enforcement
Scalable Backbone Operation
L2 routing (IS-IS) + proxy-ND in the backboneRegistration + L3 Routing in
attached networks (RPL RFC 6550)
Optional MAC address proxying to avoid MAC@ floods
New ND methods between registrars and other devices (e.g. LISP)
IETF drafts
draft-thubert-6lowpan-backbone-router
draft-chakrabarti-nordmark-6man-efficient-nd
draft-thubert-6man-wind-sail

96
6LR 6LBR 6BBR Router/ServerLP Node
Any
IPv6 Radio
Mesh Ethernet
NA (~O)
NS (ARO)
NS (ARO)
DAR(ARO), DAO
Router/Server
Router/Server
Ethernet
inside
Radio (1 Hop)
NS lookup
NS DAD
NS (ARO)
Create proxy state
Classical NDRPL, others…6LoWPAN ND Efficient ND
NA (ARO)
DAC(ARO)
NA (ARO)
ML Subnet

97
A new Efficient ND, aka Wireless ND
Multicast Avoidance
Registration for DAD
Binding (location, owner, MAC) to IPv6 address
Registrar hierarchy for lookups and policy enforcement
Operation
L2 routing (IS-IS) + proxy-ND in the backboneRegistration + L3
Routing in attached networks (RPL)
Optional MAC address proxying to avoid MAC@ floods
New ND methods between registrars and other devices (e.g. LISP)

98
Used to resolve conflicts
Need In ND: TID to detect movement ->eARO
Need In RPL: Object Unique ID if we use RPL for DAD
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type | Length = 2 | Status | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Res|P|N| IDS |T| TID | Registration Lifetime |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ OUID ( EUI-64 or equivalent ) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

99
Routers within subnet have a connected route
installed over the subnet backbone.
PCE probably has a static address in which
case it also has a connected route
Connected
Route to
subnet

100
Gateway to the outside participate to some IGP
with external network and attracts all extra-
subnet traffic via protocols over the backbone
Default
Route
In RIB

101
Directly upon NS(ARO) or indirectly upon DAR
message, the backbone router performs DAD on
behalf of the wireless device.
DAR
NS
(ARO)
DADNS DAD
(ARO)

102
NA(ARO) or DAC message carry succeful
completion if DAD times out.
NA(Override) is optional to clean up ND cache
stale states, e.g. if node moved.
DAC
NA
(ARO)
Optional
NA(O)

103
The BR maintains a route to the WSN node for
the DAO Lifetime over instance VRF. VFR may
be mapped onto a VLAN on the backbone.
RPL
DAO
Host
Route
Optional
NA(O)

104
The BR maintains a route to the WSN node for
the DAO Lifetime over instance VRF that is
continued with RPL over backbone.
RPL
DAO
RPL DAO
Host
Route

105
DAD option has:
Unique ID
TID (SeqNum)
Defend with NA if:
Different OUID
Newer TID
NS DAD
(ARO)
NA (ARO)
NS
(ARO)

106
DAD option has:
Unique ID
TID (SeqNum)
Defend with NA if:
Different OUID
Newer TID
DAR
NA
(ARO)
DAD

107
RPL
DAO
Optional
NA(ARO)
Host
Route
DAD option has:
Unique ID
TID (SeqNum)
Defend with NA if:
Different OUID
Newer TID
NA (ARO) with
older TID (loses)

108
Packet
NS
lookup
NA ARO option has:
Unique ID
TID (SeqNum)
NA (ARO)

109
NS
lookup
Mixed mode ND
BBR proxying over
the backbone
NA (ARO)
Packet

110
6TiSCH
Applications

111
Control loops
global optimization of routes for jitter an latency – thus computed by a PCE -,
flow (over) provisioning, duocast for 1 hop (1Hz and 4Hz, no routing) with static multipath hard slot allocation.
control loop plan B
self-healing -thus distributed- routing, RPL
supervisory flows
no goal: requires determinism up into the backbone
management
a separate topology – a RPL instance - that does not break.
OF?, root selection?
alerts
bursty, unexpected, on-demand slot allocation, prioritization

112
Monitoring of lots of lesser importance
stuff like corrosion? low cost scalability – this distributed routing, soft slots?
Cranes spanning area (ship)
with linear and circular motion that are mobile within a range
no goal some will need deterministic for coordination between cranes
Mobile worker
additionally provision slots ready for mobile workers) does not need deterministic since human
Large plants
thousands of devices– thus a backbone, multiple BBRs for 1 LLN –
within a subnet – to avoid renumbering though not always the case, sometimes isolation is wanted
Non-production episodes
fast and autonomic behavior – again distributed routing
Coexistence with legacy devices
a common management above, coarse level makes sense (channels backlisting but no more…single admin is good)

113
Deterministic
Wi-Fi or Ethernet
Enterprise
Plant Control
Network

114
(road, street) traffic control
increase of bandwidth allocated as more traffic in the road
Smart urban parking
redundancy of routes an over-provision as RF degrades with
cars on top of the sensors
Green zones. Monitor moisture in gardens, trigger
watering.
Toll, micro-paiment
Gargabe detection / collection path optimization
Citylights monitoring
large scale meshes

115
an entity operating an "umbrella" network on
which different types of traffic flow, possibly
coming from customers of this entity.
When a new customer asks to be granted
access to the network, the network operator
can predict with pretty good granularity the
impact this new traffic on the remaining
bandwidth of the network, and on the power
consumption of individual motes, which
allows the operator to grant/deny, and
probably charge differently

116
Resource management
Heating, Cooling and Lighting
Air quality, water leaks etc…
Food / Drink vending machines
Room occupancy
Security and Safety
(e.g. locate people in case of fire)
Exit signs monitoring
TSCH for improved availability
redundancy as RF degrades when
there are changes on the environment,
doors, moving metallic apparel, etc…

117
Less wire / copper
Lower cost of goods
Easier assembly => lower manufacturing cost
Lower weight => lower gas consumption
Less maintenance / repair
Easier addition of second mount devices
Connected vehicule
Remote maintenance
Fleet managements

118
asset tracking operations on the move (again mobility, and dynamics).
monitoring e.g. temperature of freezers, intrusion sensors, …

119
Domotics & Home Automation/ Monitoring & Security
Energy and water use:
energy and water supply consumption monitoring to obtain advice on how to save cost and resources.
Intrusion Detection Systems:
Detection of window and door openings and violations to prevent intruders
Brute force & scalability to defeat tempering

120
The Vision

121
Trillion devices in the Internet
The core technologies will not change
Leakless Autonomic Fringe
Leakless Route Projection
Million devices in a Subnet
New models for the subnet protocols
IPv6 ND, ARP, Service Discovery …
Fiber + Copper + Radio Backbone
802.11 + 802.15.4 + … Fringe
Unhindered Mobility
Location / ID Separation
10’s
of K

Thank you.

123
The Problems

124
10s of Ks devices with multihop LLN access
No broadcast in the LLNs
No renumbering (quasi permanent IPv6 address)
Continuous reachability as a LLN device moves in Subnet
Radio conditions change cause network reorganisation
Already use cases such as handheld, cranes, small vehicles
Backward Compatibility with classical IPv4 and v6 devices
Support of discovery protocols throughout the subnet

125
Cost Optimized in the backbone
Any to any traffic
Multipath and load balancing a bonus
Power Optimized in the LLNs
Most traffic flows to or from the LLN Backbone Router
Control traffic must be limited vs. Data traffic
Broadcast or scalable multicast in the backbone
Multicast or Controlled dissemination in the LLN

126
LLNs Time synchronization via the backbone
 to keep all LLNs in sync and
 allow movement from one LLN to the next
Slow deterministic LLNs (process control)
No need for Deterministic backbone
 Until scale and congestion loss becomes an issue
Upcoming faster deterministic wireless (factory automation)
 deterministic loops across networks and server OS

127
End-to-End time-synchronization
Synchronization via the backbone a plus
Standard time exchange, adapted precision.
Sync and schedule with the Deterministic backbone
MTU size a complex issue across networks
Ideally harmonize MTUs across multilink subnet or
at least detect discrepancies
Link Local lookups should be emulated
Proxy operation (per protocol)
Genralized hash based multicast