This is BKNIX initiatives to support remote peering over existing internet infrastructure where last mile connectivity is limited. Resources can be used effectively as well as offering cost effective solution to small network in remote areas
2. BKNIX Peering Forum 2023 | May, 15-16
Agenda
BKNIX at a glance
Local peering and Remote peering
Challenge of connecting remote location to IX
Design goal and solution
BKNIX Case Study
• Community networks
Conclusion & Future works
2
3. BKNIX Peering Forum 2023 | May, 15-16
BKNIX at a glance
3
• BangKok Neutral Internet eXchange Point
• Layer-2 community-based exchange
• First neutral IXP in Thailand since 2015
• Non-for-profit IXP operator under THNIC Foundation
• Peak exchange traffic rate at 148G
• 50 ASN exchange members
4. BKNIX Peering Forum 2023 | May, 15-16
Traditional peering
• aka Local peering or physical peering
• Routers resides at the same facility as IX switch
• The connection is simply just using cross-connect inside facility
• Most simple and Most affordable
4
Datacenter IX switch
5. BKNIX Peering Forum 2023 | May, 15-16 5
Remote peering
Datacenter
A
IX switch
Datacenter
B
Transport
network
• Routers resides at different facility as IX switch (Remote location)
• Remote connection is required
• Different degrees of remote locations
• Same city but different facility
• Different city, province
• Different country, continent
• Can be implemented using a dedicated connection or a virtual private network (VPN)
Dedicated connection
6. BKNIX Peering Forum 2023 | May, 15-16
Challenge of connecting remote location to IX
6
Source: Public Infrastructure
• Readiness of public infrastructure
• underground, road, electrical system
• Cost of local loop circuit
• transmission equipment
• datacenter, building, land accessing
• Insufficient cable in that area
• Distance to IX facility
• IX may only be present in major city/province
7. BKNIX Peering Forum 2023 | May, 15-16
Design goal and solution
• Cost effective and resource sharing
• Seamlessly integration
• Same experience as connecting directly to BKNIX port
• High performance
• High reliability
• Secure routing and connection
• Easy to scale
7
8. BKNIX Peering Forum 2023 | May, 15-16
IPsec (IP Security)
• IPSEC provides secure communication
between 2 peers over IP networks
• Comprise of 2 main components:
• Authentication Header (AH) - Data Integrity
• Encapsulating Security Payload (ESP) - Data encryption
• Modes: Transport mode and Tunnel mode
• Security Associations (SA): establish and manage security parameters
• IPsec NAT-T support connection behind NAT network
8
Source : commons.wikimedia.org
9. BKNIX Peering Forum 2023 | May, 15-16
OpenVPN
• Open-source VPN protocol
• TCP and UDP; UDP being faster and more suitable for real-time
applications
• SSL/TLS for encryption
• supports multiple platforms, including Windows, Linux, macOS, and
mobile devices
• supports multiple authentication methods, including certificates,
usernames and passwords, and two-factor authentication
9
Source : openvpn.net
10. BKNIX Peering Forum 2023 | May, 15-16
Wireguard
• modern VPN protocol designed for simplicity and efficiency
• It is based on state-of-the-art cryptography, including Curve25519 for key
exchange, ChaCha20 for encryption and Poly1305 for message authentication
• WireGuard operates at the kernel level, making it faster and more lightweight
than traditional VPNs
• The protocol uses a single round-trip time (RTT) for key exchange, reducing
latency and improving performance
• WireGuard has a small codebase and is easy to audit, reducing the risk of
security vulnerabilities
• Work on UDP only (not support TCP)
10
Source : wireguard.com
11. BKNIX Peering Forum 2023 | May, 15-16
Internal testbed setup
• Set up 2 routers, first router is local site and second router is from
remote site
• 2 routers connect to the same lan ; lan network representing internet
• Try to set up different vpn tunnels, IPsec, OpenVPN, Wireguard over
this network
11
LAN
10G
10G
Remote
Router
IX
Router
VPN Connection
(IPsec, OpenVPN, Wireguard)
12. BKNIX Peering Forum 2023 | May, 15-16
Performance overview
Latency comparison on different packet size
12
ps=24 RTT (ms)
Min Avg Max
Raw 1.765 1.866 1.979
IPsec 1.711 1.880 2.144
OpenVPN 1.836 2.141 2.597
Wireguard 2.168 2.326 2.671
0
0.625
1.25
1.875
2.5
3.125
Raw IPsec OpenVPN Wireguard
ps=1300 RTT (ms)
Min Avg Max
Raw 1.846 1.935 2.121
IPsec 1.808 1.908 2.141
OpenVPN 1.960 2.171 2.522
Wireguard 2.169 2.448 2.989
Avg. RTT
Millisecond
ps=24
ps=1300
17. BKNIX Peering Forum 2023 | May, 15-16
Performance overview (5)
Throughput test using iperf [tcp mode] - Overall
17
10,200 Mbps
2,110 Mbps
1,890 Mbps
254 Mbps
0 3000 6000 9000 12000
Raw
Wireguar
d
IPsec
OpenVP
N
Mbps
18. BKNIX Peering Forum 2023 | May, 15-16
BKNIX Case Study :
Community networks
18
19. BKNIX Peering Forum 2023 | May, 15-16
Community network (1)
• Decentralized, community-owned and
managed communication infrastructure.
• Built using wired and wireless
technologies and open standards.
• Provides internet access, emergency
communications, and community-based
services.
• Promotes local ownership, decentralized
management, and resilience.
• Empowers underserved or remote areas
and supports community organizing.
19
20. BKNIX Peering Forum 2023 | May, 15-16
Community network (2)
20
• Faces challenges including funding, technical complexity, and regulatory barriers.
• Has the potential to empower communities and promote digital inclusion.
Internet Internet
( )
21. BKNIX Peering Forum 2023 | May, 15-16
Community network overview
21
ISP A
ISP B
Global
Internet
• Community set up local infrastructure or ad hoc networks
• Community network connect its gateway to ISPs to connect to Internet
• Connecting to multiple ISPs improve reliability of the network
Gateway
22. BKNIX Peering Forum 2023 | May, 15-16
Connecting community network to IX
22
BGP
ISP B
ISP A
• Remote networks create remote connection through existing internet infrastructure.
• Remote connection is encrypted and protected over VPN.
• Dynamically advertise/withdraw prefixes or change policy over BGP.
• Peering with dual stack v4/v6 even underlying infrastructure is v4 only
IX ISP Community network
VPN Connection
(IPsec, OpenVPN, Wireguard)
23. BKNIX Peering Forum 2023 | May, 15-16
Connecting community network to IX
23
BGP
VPN Connection
ISP B
ISP A
IX ISP Community network
• Redundant connection through different ISPs
• Traffic engineering over multiple links (load sharing, active-standby)
Traffic flow
24. BKNIX Peering Forum 2023 | May, 15-16
Mueang Tak
Mae Sot
Myanmar
Community Wireless
Mesh Networks
(CWMN)
BKNIX
Bangkok
Avg. Distance
450-500 KM
Research & THNIC
Academy Center
Testbed setup on CN (1)
Tak
24
25. BKNIX Peering Forum 2023 | May, 15-16
Testbed setup on CN (2)
• Set up 2 routers, first router is local site and second router is from CN ;
• 2 routers connect to over the Internet through ISPs
• Try to set up different VPN tunnels, IPsec, OpenVPN, Wireguard over this network
25
BGP
IX ISP Community network
ISP A
Average distance : 450 - 500 KM
VPN Connection
26. BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (1)
Latency comparison on different packet size on community network
26
ps=24 RTT (ms)
Min Avg Max
Raw 11.569 11.971 12.557
IPsec 11.203 11.536 12.321
OpenVPN 11.670 12.008 13.097
Wireguard 11.557 12.018 12.58
10.5
11
11.5
12
12.5
13
Raw IPsec OpenVPN Wireguard
ps=1300 RTT (ms)
Min Avg Max
Raw 12.105 12.613 13.331
IPsec 12.014 12.374 13.021
OpenVPN 12.247 12.887 13.693
Wireguard 11.924 12.416 13.336
Millisecond
ps=24
ps=1300
Avg. RTT
27. BKNIX Peering Forum 2023 | May, 15-16
BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (2)
Latency comparison on different packet size on community network
X
Avg
IPsec
OpenVPN
Wireguard
-4 -3 -2 -1 0 1 2 3 4
0.393
0.309
-3.634
Max
IPsec
OpenVPN
Wireguard
-4 -3 -2 -1 0 1 2 3 4
0.038
2.715
-2.325
Percent (%)
Percentage change of Raw (ps=24)
Percent (%)
Percentage change of Raw (ps=1300)
27
28. BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (2)
Throughput test using iperf [tcp mode] - Raw
28
[ ID] Interval Transfer Bandwidth
[ 6] 0.0000-10.0503 sec 537 MBytes 448 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-10.0635 sec 537 MBytes 447 Mbits/sec
29. BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (3)
Throughput test using iperf [tcp mode] - IPsec (aes128gcm)
29
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-30.0380 sec 107 MBytes 29.9 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 25] 0.0000-30.0270 sec 107 MBytes 29.9 Mbits/sec
30. BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (4)
Throughput test using iperf [tcp mode] - IPsec (chacha20poly1305)
30
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-60.0367 sec 318 MBytes 44.4 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-60.0236 sec 318 MBytes 44.4 Mbits/sec
31. BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (5)
Throughput test using iperf [tcp mode] - OpenVPN
31
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-60.9050 sec 126 MBytes 17.4 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 27] 0.0000-60.8954 sec 126 MBytes 17.4 Mbits/sec
32. BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (6)
Throughput test using iperf [tcp mode] - Wireguard
32
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-60.1884 sec 505 MBytes 70.3 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 28] 0.0000-60.1865 sec 505 MBytes 70.4 Mbits/sec
33. BKNIX Peering Forum 2023 | May, 15-16
Performance overview on CN (7)
Throughput test using iperf [tcp mode] - Overall
33
447 Mbps
70 Mbps
44 Mbps
30 Mbps
17 Mbps
0 125 250 375 500
Raw
Wireguard
IPsec (chacha20pol
y1305)
IPsec (aes128gcm)
OpenVPN
Mbps
34. BKNIX Peering Forum 2023 | May, 15-16
Conclusion
• Remote peering solution can connect remote network from remote
area to IX
• Remote connection can be done by VPN or IP Tunneling over
Internet
• Resource can be shared by using existing cables and devices
• Peer at IX dynamically via BGP using IPv4 and IPv6
• Although VPN has more computing and latency overhead but it is
more cost effective solution comparing to dedicated links
34
35. BKNIX Peering Forum 2023 | May, 15-16
Future works
• Support more tunneling protocol
• Testing connection from more different vendors and devices
35