1. Logging for Incident Response in the
Post-IPv4 World
Carlos Martinez Cagnazzo
LACNIC
carlos @ lacnic.net
@carlosm3011

2. Agenda
• The Post-IPv4 Internet
– No IPv4, CGNs, some IPv6
• Logging for incident response
• Logging and incident response in the post-IPv4 Internet

3. The Post-IPv4 Internet
• The Internet is at a crossroads. IPv4 exhaustion means
that there will not be enough IPv4 addresses for every
one, much less for every device
• To an extent, this is already happening, but from now it
will the norm
120
100
80
60
40
20
0
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
…

4. The Current, Almost End-to-End, Internet
• Once upon a time there was something called the ‘End
to End Principle’
– … describing how packets should travel from origin to
destination untouched by the evil middle boxes
• The current Internet is _almost_, but not quite, end-toend
– Proxies, home routers, firewalls, traffic shapers, all of them do
something to packets
– But packets travel mostly unharmed

5. The Current End-to-End Internet
• Well, almost end to end**
D_Addr | O_Addr | Payload
D_Addr | O_Addr | Payload
• Packets remain (mostly) unchanged along their network
path
• A given source IP can be a marker of an individual, a
household or an employee of a certain company

6. What happens when there is no IPv4 for every device ?
• The post-IPv4 Internet:
Single public
IP address
Web server sees
thousands of
users coming
from the
*same* IP
• IPv4 will be provided, in many places, by employing
CGNs, or Carrier-Grade NAT boxes

7. The CGN-ized Internet
• The CGN Internet hides many users behind a small set of
IP addresses
• Our previous assumptions about what a source IP
address means are no longer valid
– Can represent thousands of users, of different households and
different companies
• Many abuse mitigation measures need to be reexamined
– Be careful of blankly filtering out a single /24, that could now
mean 10.000 users

8. Current practice for Incident Response
• Think for a minute about your usual IR workflow
– Phishing, Spam, DDoSing, you name it
• When your incident involves network traffic, you try to
find the following information:
–
–
–
–
Source IP addresses
Destination IP addresses and destination ports
Maybe a packet dump, if available
All of this decorated with nice timing information, preferable
with a common time zone
• You then look the sources in WHOIS or in your friendly
CSIRT contact list and send the appropriate notifies

9. The Post-IPv4 Incident Response Workflow
• Well, source IPv4 address may not be enough of an
identifier anymore
– The source network will not be able to identify the actual
offender(s) just based on the source IPv4 address
• ISPs will need source port data to actually track any
abusers
• Law enforcement also needs to realize what this means
– Judges now need to look at an additional number before jailing
a person

10. Jeez, what do we do now ?
• First of all, accept that now your life as an incident
response or site administrator will be harder
– Hopefully for a short time, until the world gets its IPv6 act
together
• Additional requirements for post-IPv4 logging
– Logging of source ports
– Using the highest possible timing resolution
– Time sync on distributed logging platforms becomes critical

11. Example configuration, Source Port Logging in Apache
• [Ref: http://draft.scyphus.co.jp/articles/20110815.html]
– Default logging in Apache only provides basic client data
– Apache uses a printf()-like format for including additional log
fields in custom log files
#
# The following directives define some format nicknames for use
# with a CustomLog directive (see below).
#
LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"" combined
LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b" common

12. Example configuration, Source Port Logging in Apache
• [Ref: http://draft.scyphus.co.jp/articles/20110815.html]
– Default logging in Apache only provides basic client data
– Apache uses a printf()-like format for including additional log
fields in custom log files
<VirtualHost [2001:13c7:7001:4000::10]:80>
ServerAdmin carlos@lacnic.net
DocumentRoot /var/www/html/
ServerName w6.labs.lacnic.net
LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"" combined
CustomLog logs/w6.labs.lacnic.net-access_log
"[%h]:%{remote}p %l %u %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i""
# LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b" common
ErrorLog logs/w6.labs.lacnic.net-error_log

13. Example configuration, Source Port Logging in Apache
• Must enable “mod_log_config” if not already enabled
[2001:13c7:7003:89:fcda:8bea:3e8a:cedd]:57366 - [31/Oct/2013:15:01:33 -0200] "GET
/site/modules/openid/openid.js?A HTTP/1.1" 304 "http://w6.labs.lacnic.net/site/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/30.0.1599.101 Safari/537.36"
[2001:13c7:7003:89:fcda:8bea:3e8a:cedd]:57365 - [31/Oct/2013:15:01:33 -0200] "GET
/site/themes/newlabs/print.css?A HTTP/1.1" 304 "http://w6.labs.lacnic.net/site/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/30.0.1599.101 Safari/537.36”

14. Example configuration, Exim4 logging
• Sample configuration:
– [ http://www.exim.org/exim-htmlcurrent/doc/html/spec_html/ch-log_files.html ]
# uncomment this for debugging
# MAIN_LOG_SELECTOR == MAIN_LOG_SELECTOR +all subject -arguments
.ifdef MAIN_LOG_SELECTOR
log_selector = MAIN_LOG_SELECTOR +incoming_port
.endif
2013-10-28 17:22:17 1VasOD-0005hG-KT <= carlos@lacnic.net
H=localhost (coco) [127.0.0.1]:47264 P=esmtp S=474
2013-10-28 17:22:17 1VasOD-0005hG-KT => marcelo
<marcelo@localhost> R=local_user T=maildir_home
2013-10-28 17:22:17 1VasOD-0005hG-KT Completed

15. Distributed logging
• Did I say ‘time sync’ before ? 
• Use NTP Luke, You Must.
– It was invented for a reason
• Look into fast data stores and mining tools
– Splunk
– ElasticSearch
– NoSQL databases (Redis, MongoDB)

16. Key Takeaways
• Yes, our sys/netadmin life will be harder, at least until
IPv6 is widely deployed
– Let’s embrace it with a smile
• Do not assume that a source attack IPv4 address
uniquely identifies an attacker anymore
– Or a victim, in some cases, like phishing sites
• Start logging source ports now. If you are a CSIRT, do not
forget to reach out to your constituency and let them
know this
• Send source ports when reporting incidents. Ask for
source ports when receiving incident reports

17. Key Takeaways (ii)
• Log with the highest timing resolution your equipment
allows
• And repeat with me…
– I will time sync my systems
– I will time sync my systems
– I will time sync my systems

18. Thank you very much! Questions?
@carlosm3011