Presentation for the Jisc event Tech 2 Tech Network performance on 24 May 2021

1. Janet Tech 2 Tech: Network Performance
Tim Chown, Duncan Rand, Raul Lopes (Jisc)
Online, 24 May 2021
Any questions? Need help? Email: netperf@jisc.ac.uk

2. Agenda
•Overview
•Performance troubleshooting
•Best practice: Science DMZ
•Janet-hosted test tools
•perfSONAR in detail (Duncan)
•Discussion
Janet Tech 2 Tech - Network Performance

3. Welcome to this Tech 2 Tech session
Making optimal use of your Janet IP network connection

4. Overview
The challenge
•Jisc is keen to see our members making optimal use of their
Janet IP connection service
•Researchers in particular want to transfer large volumes of data
•Implies our members need to follow good principles when
designing their networks and supporting their researchers
•But theoretical data rates are not always achieved in practice
•What are the reasons for this?
•What can we do to troubleshoot and improve performance?
•Let’s minimise the volume of data shipped by hard disk!
Janet Tech 2 Tech - Network Performance

5. Network expectations
The art of the possible
•In ideal conditions, with the network there for just you…
Janet Tech 2 Tech - Network Performance

6. Performance troubleshooting
Causes of problems and tools to troubleshoot them

7. Performance troubleshooting (1)
Some common causes of performance problems:
•Network congestion (from limited network capacity)
•Disk I/O or other transfer host limitations (hardware or lack of tuning)
•Poor choice of transfer tools (ftp or GridFTP?)
•Packet loss (from soft faults or underperforming network devices)
•Too small / too large buffers (cf. bursts and bufferbloat)
•Optics faults
•Asymmetric routing
•PMTUD not working (esp. for IPv6)
Janet Tech 2 Tech - Network Performance

8. Performance troubleshooting (2)
Your Janet connection capacity and usage
•How do you know if you are using all your site capacity?
•Netsight3 gives Janet sites a view of their traffic levels
•General view of up/down statuses
•View of specific site traffic over time with an approved login
•Jisc monitors usage levels for capacity planning
•Some sites now upgrading to 100G connectivity to Janet
Janet Tech 2 Tech - Network Performance

9. Netsight3 - https://netsight3.ja.net/
The high level view and more
specific data when logged in
Janet Tech 2 Tech - Network Performance

10. Performance troubleshooting (3)
Using external network views of your site
•One approach is through a looking glass
•Allows a limited set of commands to be run towards a
destination from a specific observation point
•Typically ping and traceroute, but often more
•e.g., GÉANT Looking Glass - https://lg.geant.org/
•A new Janet Looking Glass is under development
Janet Tech 2 Tech - Network Performance

11. GÉANT looking glass example
Janet Tech 2 Tech - Network Performance

12. Performance troubleshooting (4)
Approaches and tools
•The ESnet fasterdata.net guide is very good on methodology:
•https://fasterdata.es.net/performance-testing/troubleshooting/
•Many useful tools, for example:
•mtr – can show where on a path drops are happening
•iperf – raw throughput testing, can be used with fair queueing
•nuttcp – to test for small buffer impacts on bursts
•It’s ideal to have measurements over time
•Not just when a fault arises
•This is a major reason to consider deploying perfSONAR
Janet Tech 2 Tech - Network Performance

13. The Science DMZ
Following best practice

14. Science DMZ
General principles
•Treat science/research and business traffic differently
•Elements:
•Friction-free network path
•Optimise your local network architecture (a Science DMZ ‘off ramp’)
•Efficient application of security policy (avoid main campus firewall)
•Well tuned data transfer nodes (DTNs)
•With appropriate data transfer tools, e.g., Globus
•Persistent performance monitoring, e.g., perfSONAR
•With strong user engagement – know who your data movers are
Janet Tech 2 Tech - Network Performance

15. Example Science DMZ architecture
Janet Tech 2 Tech - Network Performance
Source: https://fasterdata.es.net

16. Science DMZ
Data transfer tools
•Usually desirable to avoid the ‘simpler’ tools like ftp, scp
•Look for tools designed with large scale transfers in mind
•GridFTP / Globus Connect
•XRootD
•FDT
•These will typically support parallel streams and transfers
•Minimises impact of packet loss on TCP
•Be wary of UDP tools that lack congestion control
Janet Tech 2 Tech - Network Performance

17. Janet-hosted test tools
Tools we make available on the Janet backbone

18. Testing connectivity
• Easiest are browser based speed tests, but these vary in accuracy
• What we provide, at 10G:
• Ad hoc throughput testing with iperf3 or ethr
• Collecting measurements of network characteristics over time with
perfSONAR; allows historic comparisons and analysis (for changes)
• Ad hoc disk to disk copying from a data transfer node (DTN)
• A RIPE Atlas anchor, for bespoke RIPE Atlas tests
• Note: our NOC have additional tools, such as hardware line testers
Janet Tech 2 Tech - Network Performance

19. Ad-hoc throughput testing: iperf3
Janet Tech 2 Tech - Network Performance
• There is an iperf3 server at the Jisc Slough data centre: iperf-slough-10g.ja.net
(accessible over both IPv4 and IPv6)
• iperf3 software installation:
• CentOS7: sudo yum install iperf3
• Using from Windows
• Microsoft guide to iperf usage
• Compiling iperf/iperf3 from source
• Example of Windows usage

20. Example of using iperf3 (from a MacBook)
Janet Tech 2 Tech - Network Performance
$ iperf3 -4 -t 4 -c iperf-slough-10g.ja.net
Connecting to host iperf-slough-10g.ja.net, port 5201
[ 6] local 192.168.1.128 port 54403 connected to 194.81.18.231 port 5201
[ ID] Interval Transfer Bitrate
[ 6] 0.00-1.00 sec 2.49 MBytes 20.9 Mbits/sec
[ 6] 1.00-2.00 sec 2.24 MBytes 18.8 Mbits/sec
[ 6] 2.00-3.00 sec 2.26 MBytes 19.0 Mbits/sec
[ 6] 3.00-4.00 sec 2.21 MBytes 18.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 6] 0.00-4.00 sec 9.20 MBytes 19.3 Mbits/sec sender
[ 6] 0.00-4.00 sec 8.98 MBytes 18.8 Mbits/sec receiver
iperf Done.
• Add –R to reverse the direction

21. Ad-hoc throughput testing: ethr
Janet Tech 2 Tech - Network Performance
• ethr is a “comprehensive network measurement tool for TCP, UDP &
ICMP” from Microsoft
• Natively cross platform thanks to golang
• Available for CentOS7 with ‘snap’ & installed on our server at Slough
$ ethr -d 4s -c iperf-slough-10g.ja.net
Connecting to host [2001:630:3c:f803::12], port 9999
[ 6] local 2001:630:3c:f803::6 port 51706 connected to 2001:630:3c:f803::12 port 9999
- - - - - - - - - - - - - - - - - - - - - - -
[ ID] Protocol Interval Bits/s
[ 6] TCP 000-001 sec 6.67G
[ 6] TCP 001-002 sec 9.77G
[ 6] TCP 002-003 sec 7.49G
[ 6] TCP 003-004 sec 9.45G
Ethr done, duration: 4s.

22. Persistent measurement over time: perfSONAR
Janet Tech 2 Tech - Network Performance
• Free, open source – https://www.perfsonar.net
• Easy to download and install on CentOS7 (and Debian)
• Very useful to have persistent testing: collect history of network
characteristics – throughput, loss, latency, path
• Test against our 10Gbps node in the Jisc Slough data centre
• https://ps-slough-10g.ja.net/ - ps-slough-10g.ja.net
• We also are testing 1Gbps small nodes (including RPi) and Docker versions
• We have a GÉANT PMP small node (Gigabyte Brix):
• https://pmp-central.geant.org/maddash-webui/

23. Data Transfer Node (DTN): Globus endpoint
Janet Tech 2 Tech - Network Performance
•We have a DTN in the Jisc Slough data centre: dtn-slough-10g.ja.net (dual-stack)
•A number of files are available: 1M.dat, 2M.dat, 10M.dat, 50M.dat, 1G.dat, 10G.dat,
20G.dat, 100G.dat, 1000G.dat accessible with globus-url-copy
•Can copy to /dev/null or to the file system, e.g. copying a 10GB file to /dev/null
$ globus-url-copy -vb ftp://dtn-slough-10g.ja.net:2811/space00/10G.dat /dev/null
•There is also a directory with 100 x 1GB files for more sustained testing:
$ globus-url-copy -r -vb ftp://dtn-slough-10g.ja.net:2811/space00/small/ file:///tmp/
•Other transfer tools can be made available on request to netperf@jisc.ac.uk

24. RIPE Atlas anchor
•See https://atlas.ripe.net/
•Supports measurements from RIPE Atlas nodes
•Hardware (available from RIPE) or software probes
•The RIPE Atlas ecosystem is mature
•Over 11,000 probes around the world
•Our anchor node is deployed at Slough
•See https://atlas.ripe.net/probes/6695/
•Useful for loss and latency, but can do more bespoke tests
Janet Tech 2 Tech - Network Performance

25. perfSONAR in detail
Over to Duncan…

26. Open questions
•What performance problems are you facing?
•What help might Jisc be able to provide?
•What other backbone-hosted tools would be useful for you?
•Any other suggestions for areas we should be covering?
•WiFi performance?
•Residential quality of experience (to campus and cloud)?
•Low latency (and jitter) applications?
•Experiments with (say) TCP BBR, or 9000 MTU?
•Disk to disk testing at 100G+
Janet Tech 2 Tech - Network Performance

27. Thank you
•Please feel free to get in touch
•Email: netperf@jiscmail.ac.uk
Janet Tech 2 Tech - Network Performance