1. Copyright © 2014 BIGLOBE Inc.1
High speed fiber services
and challenges to the core network
August 2014 @ MyNOG4
BIGLOBE Inc.
Seiichi Kawamura

2. Copyright © 2014 BIGLOBE Inc.2
Some figures about BIGLOBE (AS2518)
l 3 million consumer ISP users in Japan
l 250Gbps total internet traffic
p including enterprise, data center, transit
traffic
San Jose [vPOP]
-‐‑‒Equinix
IX(10G)
LA [vPOP]
-‐‑‒Coresite Any2(10G)
Japan [Core]
-‐‑‒ 3 million broadband customers
-‐‑‒ 110 colocation customers
-‐‑‒ 2 DCs fully owned by BIGLOBE
-‐‑‒ Great connectivity
-‐‑‒ Equinix(20G), JPIX(10G),
BBIX(30G), JPNAP(30G)
-‐‑‒ PNI with ALL major ISPs
Hong Kong[POP]
-‐‑‒HKIX(10G)
Singapore [POP]
-‐‑‒Equinix
IX(10G)
-‐‑‒ SGIX (1G)

3. Copyright © 2014 BIGLOBE Inc.3
Network Topology in Japan
Metro Network
Metro Network
Subscribers Peering
Subscribers Peering

4. Copyright © 2014 BIGLOBE Inc.4
Growth of Traffic
Metro Network
(Tokyo)
Metro Network
(Tokyo)
Subscribers Peering
Subscribers Peering
Peering traffic grows 1.37x every year
(Mobile Traffic 1.8x, broadband 1.2x)

5. Copyright © 2014 BIGLOBE Inc.5
1.6x growth!
The impact of high speed services
1.2x growth

6. Copyright © 2014 BIGLOBE Inc.6
Metro Network
Metro Network
Challenge1 : Keeping the metro growth down
28*10G
Metro Network
Metro Network
10*10G 5 years
Core growth 1.2x every year
Keeping growth down by
• Hot potato routing
• ECMP tuning (enhanced hash)
• Higher thresholds in capacity growth
• Studying flows and moving peers to a closer exit
No MPLS Pure IP

7. Copyright © 2014 BIGLOBE Inc.7
Challenge 2: Streaming
l 30% of FTTH traffic is streaming
l We better have a cost effective way of handling
streaming traffic (and all video traffic)
l Cost effective peering was our answer
p Does not mean we love caches
Tactics
• Use netflow and PeeringDB to find where
we can pick up the traffic
• Remote peering (unfortunately) to keep costs low
• Extensive talks with content providers
• Disclosed peering policy to pick up peering traffic
more effectively(more on this later)
http://www.biglobe.co.jp/en/peering_policy.pdf

8. Copyright © 2014 BIGLOBE Inc.8
backbone
Streaming traffic basic principle
Minimize total cost of delivery and
keep it from going to other links
IX ports, transit ports Keep traffic from overflowing
into ports with high utilization.
Can be used as backup links,
but should control traffic
Try to peer at router closest
to end user
Lower backhaul cost
Lower peering costsCosts are
fixed
But actual networks don’t look like this…

9. Copyright © 2014 BIGLOBE Inc.9
Actual Network
Access network
concentrator
Core routing
Peering
Ideally, we want video traffic to
be here, but can we peer where
the access concentrators are?
No.
Can we have caches here?
Yes, but GGC, AANP, Open
Connect, and what next? A
horrible cost model
More flexible DC interconnects may help

10. Copyright © 2014 BIGLOBE Inc.10
Challenge3 : Peering
l Domestic peering was REALLY an issue
p IX port costs
Ø 2 years ago, 10G ports cost 20,000 usd per month
p the burden of peering, mistrust, lack of
communication between peers
Ø No place to talk about peering
l We wanted a better relationship with
content providers

11. Copyright © 2014 BIGLOBE Inc.11
Changes coming from Mobile and Cloud area
l More mobile and cloud traffic
p Broadband at 1.3x growth while mobile at
1.8x growth
l Content providers have different
requirements than ISPs
p semi-full route
p simple route servers
p latency aware
p mtu9000
p fast detection using BFD
p DoS protection as a service

12. Copyright © 2014 BIGLOBE Inc.12
Evolution of peering community
l Japan did not have a place to talk about
peering until a few years ago
p PAST: People disliked talking about peering at
JANOG
l Peering relied heavily on ISPs and Telcos
• BoFs, study councils, talks at NOGs, and
joint efforts by the IXPs and the community
– more focus on content/dc and its requirements

13. Copyright © 2014 BIGLOBE Inc.13
community activities
l Google Groups : Peering in Japan
p local Japanese language only
p discussion on latest peering issues
p no IX personnel on list
p host Peering BoFs
l Tutorials
p IXP provides low cost tutorials (available regularly)
p free tutorials at JANOG (not always available)
l CloudIX Study Council
p Group of members(BGP operators) in BBIX doing
technical experiments
p sharing skills and helping out each other to peer

14. Copyright © 2014 BIGLOBE Inc.14
Challenge4 : Automation[ a work in progress ]
l Our data center is well automated
p in the past
dc config change >>>>>>>>>> peering
p dc requires orchestration with servers and other
virtual devices
l Since we started peering more (and more
globally) we are now needing better
automation tools
l Current implementation
p Simple html(form) +mysql+ jinja2 to generate
configs via Web UI
Ø Trying to shift to Django
p Exscript to push configs to router

15. Copyright © 2014 BIGLOBE Inc.15
Other misc challenges
l Unnecessary traffic avoidance
p P2P services that have security problems are
controlled at the edge
p BCP38 to stop sending unnecessary traffic
Ø still in initial deployment phase
l Higher density with better power efficiency
p Working to keep PUE around 1.2-1.3
p Switching to higher density line cards
l Measuring actual user experience
p “1Gbps service” is more marketing talk than actual
experience
p Measuring response to popular sites (alexa)
p Python script to crawl command and filter out noise
data “hping -p 80 -c 5 --syn [url]” works well with unix,
PSPing works well with Windows

16. Copyright © 2014 BIGLOBE Inc.16

17. Copyright © 2014 BIGLOBE Inc.17
Next steps for evolution
l MPLS, TE, auto-bandwidth
p Not necessary for domestic consumer service
backbone, but need TE to control global network
p We did not need this in the past since we did not
provide VPN services (we may in the future)
p we have a vxlan network running, so considering
EVPN as one of the solutions
l Better network management and security
p BMP and other tools for better peering management
p APIs to the router
p BGP Flow Spec
l Differentiation of services inside the backbone
p Last mile QoS and internet fast lane?

18. Copyright © 2014 BIGLOBE Inc.18
Summary
l Higher speed services to the consumers,
means new challenges to the core network
l Peering really helps, but it requires a good
ecosystem to be helpful
p diversity in IX services, fair pricing, carrier
neutrality, open talks about peering
l Handling streaming and video, is still a big
headache
p is there a better way than everybody has their
own cdn?
l Automation and tools are key to handling
bigger traffic in a cost effective way
l Measurements make a difference in user
experience. “1Gbps marketing talks” don’t