The document presents a reinforcement learning approach called Smart-DTN-CC for congestion control in delay-tolerant networks (DTNs). It describes DTNs and their characteristics. It also discusses existing congestion control mechanisms' limitations. Smart-DTN-CC uses Q-learning to allow nodes to autonomously learn and select actions to mitigate congestion. The paper evaluates Smart-DTN-CC's performance against other mechanisms in different scenarios using a simulator, finding it outperforms existing approaches.

1. 1/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Smart Congestion Control for Delay- and
Disruption Tolerant Networks
Aloizio P. Silva14 Katia Obraczka2 Scott Burleigh3
Celso M. Hirata1
1Instituto Tecnológico de Aeronáutica - ITA
Department of Electronic and Computer Engineering
2University of California Santa Cruz - UCSC
Department of Computer Engineering
3Jet Propulsion Laboratory - NASA
California Institute of Technology - Caltech
4Universidade Federal de Minas Gerais - UFMG
Department of Computer Science
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

2. 2/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Outline
1 Introduction
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
2 Smart-DTN-CC
Reinforcement learning
Q-learning
3 Experimental Analysis
4 Conclusion
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

3. 3/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
Delay and Disruption Tolerant Networks - DTNs
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

4. 4/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
Delay and Disruption Tolerant Networks - DTNs
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

5. 5/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
Delay and Disruption Tolerant Networks - DTNs
Extreme environments
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

6. 6/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
Delay and Disruption Tolerant Networks - DTNs
Original Application
Deep space communication
IPN - Interplanetary Networks
Characteristics
Long and variable delays
Intermittent connectivity
Asymmetric data rates
High error rates
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

7. 7/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
Congestion Problem
Incoming traffic rate exceeds the rate at which packets are
processed.
Network becomes saturated/ overloaded.
Figure: Congestion in networks
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

8. 8/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
DTN Congestion Problem
TCP congestion control mechanisms don’t work in DTN’s.
Figure: DTN congestion
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

9. 9/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
Existing DTN Congestion Control Mechanism
Main features:
DTN application specific
Network’s global information
Reactive approach
Dependence of the routing protocol
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

10. 10/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Delay and Disruption Tolerant Network - DTN
Congestion Problem
The State-of-the-Art DTN Congestion Control
A Novel DTN Congestion Control Framework
DTN congestion control requirements:
1 Local and autonomous
2 Proactive and reactive
3 Able to adapt automatically to network dynamics
Our solution: machine learning.
1 Supervised learning
2 Unsupervised learning
3 Reinforcement learning
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

11. 11/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Reinforcement learning
Q-learning
Reinforcement learning
Employ reinforcement learning
Objective: perform congestion control
Need to make a sequence of good decisions to mitigate
DTN congestion
Similar to training a pet dog
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

12. 12/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Reinforcement learning
Q-learning
Reinforcement learning
Figure: Reiforcement learning
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

13. 13/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Reinforcement learning
Q-learning
Smart-DTN-CC
Figure: DTN node state machine
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

14. 14/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Reinforcement learning
Q-learning
Smart-DTN-CC
Figure: DTN node actions
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

15. 15/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Reinforcement learning
Q-learning
Smart-DTN-CC
Action selection methods
1 Boltzmann distribution
Parameter: a temperature T controls the amount of
exploration.
2 WoLF: Win or Learn Fast
Parameter: two learning rates: γmin and γmax .
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

16. 16/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Reinforcement learning
Q-learning
Smart-DTN-CC
Figure: Reward function
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

17. 17/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Reinforcement learning
Q-learning
Q-learning Algorithm
Figure: Q-learning
algorithm Figure: Q-values update function
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

18. 18/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Experimental Methodology
We use the ONE Simulator
Evaluation Metrics:
Average delivery ratio
Average end-to-end latency
The scenario is a terrestrial mobile network
Mobility models: RW, RWP and SPMBM.
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

19. 19/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Experimental Methodology
We evaluate SMART-DNT-CC by itself
We compare SMART-DTN-CC’s performance against;
1 AFNER
2 CCC
3 RRCC
4 SR
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

20. 20/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Results: Average cumulative reward as a function of the
simulation time
Figure: Boltzmann action selection method
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

21. 21/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Results: Average delivery ratio as a function of buffer
threshold.
(a) Delivery Ratio
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

22. 22/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Results: Comparative analysis
Average delivery ratio as a function of message generation
period.
(b) Terrestrial Scenario
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

23. 23/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Results: Comparative analysis
Average delivery ratio for different mobility models and
routing protocol (Buffer size of 500kB, Message Generation
Period of 300 s, Buffer Threshold for Smart-DTN-CC of
60%)
(c) Random Walk (d) Random Way Point (e) Shortest Path Map
Based Movement
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

24. 24/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Results: Comparative analysis
Average latency for different mobility models and routing
protocol (Buffer size of 500kB, Message Generation Period
of 300 s, Buffer Threshold for Smart-DTN-CC of 60%)
(f) Random Walk (g) Random Way Point (h) Shortest Path Map
Based Movement
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs

25. 25/ 36
Introduction
Smart-DTN-CC
Experimental Analysis
Conclusion
Conclusion
We investigate the congestion control in DTN.
A novel congestion control framework was designed
(SMART-DTN-CC):
It is based on machine learning: reinforcement learning
it allows the DTN nodes autonomously mitigate congestion
It adopts an hybrid approach: proactive and reactive, open-
and closed loop.
it allows a DTN node to adapt to different DTN
environments
The results shown that it outperforms the existing
mechanisms.
IEEE SECON’2016 - London, UK - June 27-30, 2016 Smart Congestion Control for DTNs