APNIC Infrastructure & Development Director Che-Hoo Cheng gives a presentation on ROA and ROV deployment and why routing security is becoming more important than ever at the 32nd TWNIC IP OPM in Taipei from 20 to 21 June 2019.
2. Security matters as your network is
connecting to Internet
⢠You do NOT want your own routes to be hijacked by anyone, maliciously or
accidentally
⢠You also do NOT want to receive bad routing information from any of your
BGP neighbors or propagate bad routing information to any of them
⢠Basic measures include:
â Bogons and martians filtering
â Max prefix count
â IRR (Internet Routing Registry) database checking
â So on and so forth
⢠Additional measure should include:
â ROA (Route Origin Authorization) / ROV (Route Origin Validation)
3. Routing security is becoming more
important than ever
⢠Route-hijacking cases (malicious and accidental) are more and more
common
â Big incentive for hackers
⢠Hijack DNS, hijack websites, steal passwords and so on
â Misconfiguration does happen from time to time
⢠And, it is extremely easy to do route-hijacking, if protection measure is
not implemented
⢠A lot of route objects on IRR-DB are not authenticated properly and so
cannot be fully trusted
⢠Need better authenticity for routing info, i.e. need to make sure that the
route originators are the true âownersâ of the relevant IP resources
4. Fat-finger / hijacks
⢠Quad101 related route (101.101.101/24) was hijacked by
AS268869 (FIBRA PLUS) for 3.5 mins on 8 May 2019
â https://blog.apnic.net/2019/05/30/public-dns-in-taiwan-the-latest-
victim-of-bgp-hijack/
â Origin AS should be AS131621 (TWNIC)
â Implication can be huge if anycast is not done well
5. Fat-finger / hijacks
⢠Amazon (AS16509) Route53 hijack â Apr 2018
â AS10279 (eNET) announced/originated more specifics (/24s) of
Amazon Route53âs prefix (205.251.192.0/21)
⢠205.251.192.0/24 âŚâŚ. 205.251.199.0/24
⢠https://ip-ranges.amazonaws.com/ip-ranges.json
â The motive?
⢠During the period, DNS servers in the hijacked range only responded to queries for
myetherwallet.com
⢠Responded with addresses associated with AS41995/AS48693
6. Fat-finger / hijacks
⢠Bharti (AS9498) originates 103.0.0.0/10
â Dec 2017 (~ 2 days)
â No noticeable damage done â more than 8K specific routes!
⢠YouTube (AS36561) Incident
â Feb 2008 (down for ~ 2 hours)
â PT (AS17557) announced 208.65.153.0/24 (208.65.152.0/22)
⢠Propagated by AS3491 (PCCW)
7. RPKI
⢠RPKI is a public key infrastructure (PKI) framework,
designed to secure BGP routing
â Based on X.509 PKI standards
⢠RPKI adds Internet number resources (INR) information to
X.509 certificates issued to resource holders
â Representing âownershipâ and other status
â Certification hierarchy follows INR delegation hierarchy
IANA â RIR (â NIR) â ISP â âŚ
8. RPKI hierarchy
NIR CA
ISP EE ISP EE ISP EE ISP EE
IANA
CA
APNIC
CA
LACNIC
CA
RIPE- NCC
CA
ARIN
CA
AFRINIC
CA
9. RPKI service models
⢠Hosted model
â APNIC performs CA functions on behalf of members
â Manage keys, repository and so forth
â Generate certificates for resource delegations
â This âMember CAâ is separate from the âAPNIC CAâ
⢠Provisioning model
â Member operates full RPKI system including CA
â Communication with APNIC via âup-downâ provisioning protocol
⢠Either rsync (to be deprecated) or RRDP (preferred)
â This is live at JPNIC, CNNIC and TWNIC (IDNIC in progress)
10. RPKI objects
⢠Resource certificates
â Extension of standard X.509 certificates
â Providing authority to use given IPv4/6 and ASN resources
â Signed by issuing registry (serving as CA)
⢠Route Origin Authorization (ROA)
â Giving an ASN authority to route specific IP blocks
â Signed by IP resource holder
⢠âAnysignâ, âghostbusterâ and moreâŚ
â Other useful objects proposed and coming later
12. RPKI application: ROA
⢠Route Origin Authorization
â List of prefixes with ASN authorized to announce
â Signed by the prefix holder
⢠RPKI validates the integrity of the ROA
â It is provably created by the holder of the prefix
â Can now be used to construct route filters for prefix-OriginAS pair in BGP
⢠Multiple ROAs can exist for the same prefix
Prefix 203.176.32.0/19
Max-length /24
Origin ASN AS17821
13. Use of ROA: ROV
⢠Route Origin Validation
AS17821
Announce:
203.176.32.0/19
Peer/Upstream
?
14. RPKI validator
⢠Gathers and validates ROAs from the distributed RPKI databases
â Using rsync or RRDP (preferable)
â Maintains a validated cache representing complete global state
⢠Can then perform ROV for routers using RPKI-Router (RTR) protocol
rpki.apnic.net
IANA
APNIC RIPE
ISP ISP
rsync /
RRDP
Validated
cache
Validator
16. Route validation states
⢠Not Found (Unknown)
â No ROA found, probably not created yet
â This will be âdefaultâ for some time.
⢠Valid
â ROA exists
â Prefix, Origin ASN and prefix-length match those found in validated cache
⢠Invalid
â ROA exists
â Prefix found, but Origin ASN is wrong, Prefix-length longer than Max-length, or
certificates are expired or otherwise invalid.
â Some action needed
18. Options when receiving invalid routes
⢠For End/Stub Networks:
â Drop them, OR
â Give them lower LOCAL_PREF, OR
â Do nothing (not recommended)
⢠For Transit Networks:
â For inbound routes from upstreams / peers:
⢠Give them lower LOCAL_PREF, OR
⢠Drop them, OR
⢠Do nothing (not recommended)
â For outbound routes to customers:
⢠Tag them before re-distributing them to customers and allow customers to make their
own choices
20. ROV at IXP â RS usage options
⢠Similar to the case of Transit Networks
⢠Lower LOCAL_PREF, OR
⢠Filtering
â Do not advertise Invalid routes
â Need to publish on RS policy
⢠Tagging
â Apply community tags based on the validation state
⢠let individual member ASNs act on the validation states
â Example:
⢠Valid (ASN:65XX1)
⢠Not Found (ASN:65XX2)
⢠Invalid (ASN:65XX3)
21. ROV at IXP â Examples in Asia Pacific
⢠Shared validator provided by:
â JPNAP & BKNIX
⢠Other IXPs?
â IXPs are good locations to place shared validator as they are
just one hop away from their participants and they are mostly
trustable
â You may push your IXPs to support it to ease your burden of
setting up your own Validator/Cache
â IXP Manager SW (https://www.ixpmanager.org) now supports
easy ROV deployment on RS at IXPs
22. ROA+ROV â Why do we do it?
⢠Contribute to Global Routing Security
â Help reduce the effect of route hijacking or
misconfiguration
â Protect your own networks and your customers better
⢠Collaborative effort among network operators is
key
23. ROA+ROV is NOT a bullet-proof solution
⢠It is just one small step for improving routing security
⢠But it helps improve the situation for routing security,
especially if everybody does it
⢠Coupled with more and more direct peering, the protection
for routing security should be more effective
⢠Highly recommend doing full MANRS as well
24. ROA stats â global snapshot
⢠Source: https://rpki-monitor.antd.nist.gov/?p=0&s=0
25. ROA stats â global trend
⢠Source: https://rpki-monitor.antd.nist.gov/?p=0&s=0
26. ROA stats â APNIC region snapshot
⢠Source: https://rpki-monitor.antd.nist.gov/?p=3&s=0
27. ROA stats â APNIC region trend
⢠Source: https://rpki-monitor.antd.nist.gov/?p=3&s=0
28. ROAs in APNIC region
⢠Source (IPv4 prefixes covered): https://lirportal.ripe.net/certification/content/static/statistics/world-roas.html
29. ROA+ROV deployment steps
⢠Create your own ROAs at relevant registries to better
protect your own routes
â And encourage your peers/customers to do the same
â For APNIC members, it is easy to do it on MyAPNIC
⢠We can help!
⢠Please contact APNIC Helpdesk
⢠Next step is to set up route origin validation (ROV) at your border
routers
â Using public or IXP validator, or your own
⢠Do care about backup and resilience
â And ask your IXP/upstream providers to implement ROV
⢠Simple click to test your provider: https://ripe.net/s/rpki-test
30. Industry development on ROA+ROV
⢠NTT â IRR improvement favoring Route Objects with valid ROAs
⢠Cloudflare â Public validator service & invalid routes filtering
⢠AT&T â Invalid routes filtering on peering connections
⢠Netnod â Invalid routes filtering and favouring of valid routes on IXP Route
Servers
⢠AWS â BYOIP requires customers to set up ROAs
⢠Google â Will start to apply stricter filters to BGP announcements on all peering
sessions by Sep 2019, using RPKI data (ROAs) where available to validate IRR
data
⢠Big players are getting more and more serious with ROVâŚ
31. Announcing invalid routes
⢠Will get to fewer and fewer networks on Internet
â Similar to being disconnected from bigger and bigger part of
Internet
⢠If it is just a mistake, updating the relevant ROA records
(supposedly with proper authority) will solve the problem
â Should always keep your ROA records updated
⢠All can be managed at one place so should be easy
â Can have ROA records for the same prefix under multiple Origin
ASes at one time to help the cases of network migration and so on
32. Incentives for creating ROAs
⢠To have basic protection of your own routes from being hijacked at
those networks which do ROV
⢠Industry push:
â NTT â IRR improvement favouring Route Objects with valid ROAs
â Netnod â Invalid routes filtering and favouring of valid routes on IXP Route
Servers
â AWS â BYOIP requires customers to set up ROAs
â Google â Will start to apply stricter filters to BGP announcements on all peering
sessions by Sep 2019, using RPKI data (ROAs) where available to validate
IRR data
⢠More will be comingâŚ
â As a requirement for peering???
33. More about RPKI benefits
⢠Improved in-band verification of resource custodianship
â Much safer than manually checking whois or IRR database
â Ease of automation
⢠Secure Origin is the first step to preventing many attacks on BGP
integrity
â BGP Path remains a problem which is under development
â Related information such as IRR Policy can now leverage strong proofs
of validity (end the maintainer-authority problem in RADB/IRR)
⢠Instruction/information from the resource custodian can be
cryptographically verified (e.g. LOA signing)
Now, there's another one. Again, weâre going to move on now to something called RPKI, the Resource Public Key Infrastructure. That is a standard PKI framework. You might be aware of the X509 PKI standards ,which provide you with public and private key certificates. They're used actually in HTTPS in web authentication, they're used in S9 for signing and encrypting emails. X509 is fairly well recognized as a PKI framework. Our PKI is an adaptation of that, that adds Internet number resource information to those standard X509 certificates. Under RPKI, if you receive an allocation of address space from a registry then under RPKI you will also receive a certificate, a digital certificate that actually gives you the keys to the use of that set of Internet resources. Then you can use that certificate to show you are, to show cryptographically you are the valid and authorized holder of those resources.
So, the certificates sort of represent your ownership, if you like of the resources, and the certification hierarchy, which is a standard part of PKI, also represents, also follows the delegation hierarchy. So from the IANA to the RIR to the NIR to the route NIC space, the the ISP. You've got an allocation hierarchy and you've got a certification hierarchy. So you can see it here. IANA is allocating to each of the RIR and in this space IANA is a certification authority as well as an allocation authority. So the [inaudible 00:18:56]. In the case of APNIC in the middle, we are also a CA certification authority, for allocation to TWNIC and for distribution of certificates or certification of TWNICs address space. TWNIC is also a CA, providing the same service to ISPs. In this case it's showing all these ISPs at the bottom not at CAs, but as end entities in the terminology of RPKI.
Any of those ISPs can also perform a CA function if it's providing allocations to downstream ISPs and it can also provide a CA function to it's certified downstream allocations, that's part of the [inaudible 00:19:37] framework as well.
EE â End-Entity
What are we doing on RPKI at APNIC? Well it's a complex system and as I've said before and you might have wondered, as an ISP you're expected not only to do your work, but also tomorrow to start running a certification authority to make your downstream delegations, that's probably not something that youâre going to earn any or much revenue from. It's just an extra administrative tasks that you would need to do. So APNIC is actually, we've got what we call a hosted model of RPKI and that means that centrally at APNIC we can perform the CA functions on behalf of members. We can manage the keys in the repository and the certificate generation on behalf of a member ISP so that when a member of ISP is simply making a sub-delegation, a sub-assignment for instance, to one of their customers, all of the CIE and RPKI stuff can happen more or less automatically. That requires some trust in APNIC because of course we're managing your keys for you. We do that in a separate, we've got two CAs set up so we've got a member CA separate from the APNIC CA as a mechanism to ensure or help ensure the security of that model. So that's the first model is the hosted model providing that service on behalf of members.
The other model is the provisioning model where a downstream recipient of the address space such as an NIR, and this is where the provision model is being used at the moment, that party will operate a full RPKI system including certificate authority function. They communicate automatically with APNIC via another one of those standardized protocols. The up down provisioning protocol, which has got a couple of options, either rSync or RRDP and rSync is the old way and RRDP is the third new way of interacting with AB net. What that allows you to do is run your own full CA RPKI infrastructure and do your own certification, validation and so forth. This is something actually live already at JPNIC, at CNNIC, at TWNIC. We may have some large APNIC members in future who are using the same system because they probably want to do it themselves according to their own systems. It's definitely a more complex solution, but it's the right one for it's intended IR.
90+% of the networks will adopt Hosted model
So, RPKI has got, as a system it's got some digital objects which are involved. They are these things called resource certificates and they standard X 509 certificates. The data structures that represent the holder's authority or your authority to use IPv4 [inaudible 00:20:00] resources and that's what you get, as I said, with your address block from the registry.
There's something else which is used in operation called the route origin authorization. That's another data structure, assigned data structure within the RPKI that actually represents an authority for an autonomous system to actually originate route for particular address blocks. So as an IP address holder, you can sign the authority, digitally sign the authority for a particular ASN to originate your address blocks. As you may know, a lot of route high-jacking happens from people who are not authorized taking someone else's addresses and announcing them into the routing system. The point of this, of the RPKI is to actually prevent that from happening, to say that we can only announce address blocks into the routing system if you have the authorization to do so.
There's quite a few extensions to RPKI as well. A bit like RDAP, it's something that can be standardized and can actually be used in multiple useful ways in the future. We don't know quite how it's going to be used in the future. There's plenty of potential for that.
Back to this familiar diagram. Now we've got the registry database being used as a source for whois data, whois services, for RDAP services and for RPKI services. From RPKI, what you receive is an X509 certificate or an RPKI certificate and you can also receive and generate your ROAs, your route origin authorizations. So, the RAO is actually a document which is signed by your certificate, and once you have those data structures available to you, then you can do all sorts of things with them. You can use them for verification, you can use them potentially for other functions in [inaudible 00:21:58]. For instance, validation of powers when that becomes [inaudible 00:22:03].
We've got a service at APNIC under development called âanysignâ, which means that you can take your certificate representing your Internet number resource and sign anything you like with it. I'll explain that as well. Naturally, there's other things that will come along.
In general they do some checking to make sure that they announcement they're receiving actually is a valid one from that Autonomous System. If it's not a valid one, then of course that autonomous System Number could be high-jacking that address space, could be attracting that traffic and doing something bad with it like generating false responses and so on. The point of the Route Origin Authorization is this signed data structure, as I said, which gives a prefix, it gives the ASN, it gives the maximum length of sub-prefix which that ASN is authorized to advertise.
For instance, in this case you've got a /19 being authorized under this ROA, but it also says that the ASN 17821 can announce the /19 or announce any longer prefix up to a /24 and that's to accommodate traffic engineering in the [inaudible 00:23:41], so address management that might happen at an autonomous system. The point of the ROA is it's a signed document or a signed data structure, so the RPKI as a system is able to validate that and allow a router to know that that's a valid, the given announcement from a given AS is provably valid. That's what can then be used automatically in route filtering, configuration and so on.
Route Origin Validation, it's this thing I mentioned before where an Autonomous System can announce a particular address block to a peer. But what does the peer do with that? Does the peer just accept that that autonomous System actually is the holder or is a valid destination for routes to that address block? That peer should not just accept that. That peer really should be doing some checking.
RPKI is a pretty complex thing actually. It's got a lot of moving parts, but one of those that's relevant here is the validator. So an RPKI validator is a machine that gathers ROAs, route origin authentications from numerous sources. It validates those ROAs to make sure they actually are correctly signed. That the PKI is all valid within it. It gathers a cache then of all of the valid ROAs that it's aware of and it can then perform validation for routers as clients are using something called the RPKI intra-router protocol.
RP â Relying Party
There's a few validation states which are defined under those circumstances. So the route server, having consulted the validator will then determine, the validation status may be unknown or not found. That will be very common these days because that actually means that for a given prefix there is not ROA existing and so for the vast majority of prefixes in that routing system at the moment, there is no RAO yet because this is still being rolled out. But where the ROA exists, it can be a valid ROA and that means that the prefix and origin AS and the length matching and again the validation has been correctly authorized through the RPKI system. If it's not valid, then it's invalid and that means that the ROA exists, but there's a problem. So it means that this, the address prefix that's being routed, if it does have a ROA then if it's to be routed it needs, that ROA needs to be valid. This is where either address high-jacking or errors can be detected and avoided.
A router can inquire of the validator using the RTR protocol to find out whether a given announcement the router has received is valid.
In the case that the ROA is invalid, then some action is needed. That action could be, that action is actually up to the operator. So the operator might decide to just [inaudible 00:27:21] it's invalid. That could be risky because the RPKI itself might have an error in it and it might be declaring something is invalid, particularly now in these early days it might erroneously declare a route to be valid where it in fact is not. Probably rather than dropping, what's recommended is to give that route a lower local preference. That means that it will not, for instance, as we're more specific, it won't replace or redirect traffic from another route that exists within the system.
For outcome routes to your customers, or as I said, for tagging across an IXP, then the other thing that might happen with an invalid route is a tag added to it by the route server or the router that allows the recipient to then determine what to do. The sort of tagging you might have would be a community tag with an AS of 65XX1, but not found, or 65XX2 is valid, or whatever. So these are sort of conventions that need to be set up. The point is, what this mechanism provides is an automatic and reliable means for you to actually understand much better what is the status of routing announcements that are going across the Internet exchange or in fact into your network from the rest of the Internet. So that's route origin validation.
It can also work at IXPs. Because an IXP is a more closed community of members than Internet routing at large globally, this is actually being used actively in quite a few exchanges now. In this case, the route server is the one that enters into the RTR protocol with the validator. Any time an exchange partner wants to advertise a route, that will as usual go to the route server for redistribution. The route server in this case will actually first validate that route to make sure that the source is authorized. Then, once validated it will propagate that route to the rest of the partners. It will do that with some extra tagging or filtering on those routes that actually show the rest of the partners what the status of that route is.
Now what's the status of RPKI? Well, this is a little bit like looking at the early days of IPv6. So this is kind of transition that needs to take place across the entire Internet and actually we're not doing too badly. The global routing system according to this view, in this view it's got 783000 unique IPv4 prefix origins hits and of those, 78000, that is just over 10% actually have valid ROAs associated with them. So if you want to run and RPKI validation system, you can actually see the validation data coming through to you for 10% of the routes that you use, that you might be seeing on the Internet. That's pretty good I'd say, we're in the early days. Of course the green slice needs to grow to encompass 100%, but what this shows again is that 78000 routes, this is in real operational, real time deployment these days and it's very good to see.
The other component there is that we are seeing invalid routes so there are 6222 routes in the global Internet, as of this shot that are potentially valid. They could be mistaken configurations, they could be address high-jacking attempts, whatever. They could be either of those things. The fact that they're invalid allows an ISP to actually see that and take some action. As I said, the action you take may be just to de-preference those routes so the traffic will still go through those routes, but only as a lower preference to what would probably be the existing legitimate route.
At the APNIC region we're not doing quite as well as the global region, but we've got 5% of all space, 5% of all announcements that originate from APNIC resources as valid and we've got about the same, just under 1% invalid. So that I hope will increase. That can be another race that we all try and track through statistics and data as time goes on, just like we do with IPv6 where we can get ourselves a pat on the back as our numbers increase.
So how do we all start? There is the ability for most of us here, if we are members of APNIC or TWNIC to be able to create ROAs, to be able to create route origin authorizations. Then to encourage our peers and our colleagues to do the same, and to encourage the IXPs to actually implement ROV and route servers. Down the track you can set up route validation on your onboard or routers, you can also use, in doing that you can use validators which are public. If you don't want to run your own, you can use a public validator or a validator that your IXP might offer. For APNIC members at least, you can use MyAPNIC and APNIC will provide whatever help you might need from the help desk to get that up and running.
The benefits of RPKI is this improved and automatic, automatable validation of resource holdings which is something that we don't have at the moment. We might have it better under RDAP because RDAP does actually give you a much easier way to consult and to find out address holder information that doesn't provide the cryptographic security, for instance, that you get out of the ROA which cryptographically links the autonomous system with the prefix being announced. The whole point of this secure origin validation and path validation are the steps that are required to prevent a lot of attacks are happening on the [inaudible 00:38:45] system at the moment. Origin validation right now is, path validation is still something that's coming up. As I mentioned already, origin validation can still be very effective and it can be implemented today and quite readily across the Internet storage ports.
So RPKI is, it actually as I said, has a lot of moving parts. It's a much more complex set of specifications than you saw with RDAP and that's a list of about 20 or so out for 42 RFCs that exist in the IETF specifying RDAP and related BGPSec standards. The [inaudible 00:31:11], that registry is an APNIC in particular have been involved in a lot of these ITF specification documents, a lot of these RFCs. So over the years, we've really done a lot behind the scenes in setting this up and making it all happen. It is actually very good to see that it's starting to be coming to real time operation use these days.