Your website just went down. As you try to understand what has gone wrong, you quickly realize something is different this time. There’s no clear reason why your site should be down, but indeed it is.
This talk is about the story of our team’s first unprepared fight against a DDoS attack.
14. tcpdump of anomalous traffic
GET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1
User-Agent: 1.{RND 10}.{RND 10}
Referrer: http://my.opera.com/Ao-Trang-Oi/
Cache-Control: no-cache
Cookie: __utma=218314117.745395330 […]
__utmz=218314117.1286774593. […]
utmcsr=google|utmccn= […]
utmctr=cach%20de%20hoc%20mon […]
<... random high speed junk follows ...>
15. tcpdump of anomalous traffic
GET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US;) Gecko/20030624 Netscape/7.1 (ax)
Accept: Accept=text/html,application/xhtml+xml,...
Accept-Language: Accept-Language=en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: Accept-Charset=ISO-8859-1,...
Referer: http://my.opera.com/Ao-Trang-Oi/blog/
Pragma: no-cache
Keep-Alive: 300
ua-cpu: x86
Connection: close
16. #nginx, 14th October 2010
cosimo: we're seeing a pretty "interesting" problem within our
nginx
fronts
BLAH BLAH BLAH
cosimo: there's a few hosts sending a legitimate HTTP GET
request
BLAH BLAH BL
cosimo: followed by a binary stream of random bytes that never
ends
BLAH BLAH BLAH
cosimo: this is just 1 request going on and on
cosimo: is there some way to alter the nginx config to shut
down these
client connections? OMGWTFBBQ!!!!11111
cosimo: the client is sending something like:
cosimo: GET /blah HTTP/1.1 “this is nkiller2”
cosimo: Host: ...
cosimo: Etc: etc...
cosimo: and then random bullshit
vr: :)
vr: this is nkiller2
vr: haproxy can fight this
vr: you can set a timeout http-request
vr: don't know if nginx can do this
cosimo: cool
41. What can we, as Ops, do better?
● Embrace failures and learn from them
● Be fast (no panic/blame, think Mr. Wolf)
● Coordinate (#ops, war rooms, ...)
● Take notes
● Learn TCP/IP
● Know your tools
(tcpdump, tcpflow, strace, nc, iptraf, …)
43. Thanks to...
● ithilgore (sock-raw.org) for writing nkiller2
● @vr in #nginx for pointing us at nkiller2
● David Falloon for his great “untested” idea
● marc.info for correctly handling “@” in ml
● SANS Institute for the TCP/IP references
● My team at Opera