1. Post mortem.
Or how me managed to handle
72 DDoS's a day. Using Rails.

2. Hello there.

6. 200 OK
404 Not Found

14. Bash Objective-C
Python Java
Ruby C / C++
NodeJS C# (.NET)

15. !
Bash Objective-C
Python Java
Ruby C / C++
NodeJS C# (.NET)

17. The problem.

18. 1.5M+
tracked devices

19. 25M+
reports received

20. */20 * interval *
execution
**

21. 250K
requests per minute

22. 90%second
the very same

23. +

27. =

29. We did what
the book says.

30. Master/slave

31. Load distribution

32. Memcached

33. Static cache

35. Click click!

40. [error] upstream prematurely
closed connection while reading
response header from upstream,
client: 24.26.30.50, server:
*.preyproject.com

41. TCP: Possible SYN flooding on
port 443. Sending cookies.

42. nf_conntrack: table full,
dropping packet

43. nf_conntrack: table full,
dropping packet
(…)
net_ratelimit: 8130 callbacks
suppressed

44. !

45. Time to roll
up our sleeves.

48. An
hour
in
the
life

50. #%@!

51. So where's the
bottleneck?

52. Nginx Stub Status
./configure
--with-http_stub_status_module
GET /nginx_status
Active connections: 2076
server accepts handled requests
16630948 16630948 31070465
Reading: 720 Writing: 379 Waiting: 981

53. Raindrops for Unicorn
config.ru
use Raindrops::Middleware
GET /_raindrops
calling: 254
writing: 0
0.0.0.0:8080 active: 48
0.0.0.0:8080 queued: 0

54. Tuning backlogs
Unicorn, config.rb
listen 'shared/sockets/unicorn.sock',
:backlog => 4096
Nginx Vhost conf
server {
listen 80 backlog=16384;
...
}

55. Linux TCP/IP Stack Tuning
Connection count by status
$ netstat -an | awk '/tcp/ {print $6}'
| sort | uniq -c
30 CLOSE_WAIT
2234 ESTABLISHED
4 FIN_WAIT1
14 LISTEN
6 SYN_RECV
3222 TIME_WAIT

56. Linux TCP/IP Stack Tuning
$ sysctl -a
# max sockets, connections
net.core.somaxconn = 131072
net.core.netdev_max_backlog = 131072
net.ipv4.tcp_max_syn_backlog = 35536
# reuse & recycle TCP sockets
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

57. Linux TCP/IP Stack Tuning
# disable syncookies
net.ipv4.tcp_syncookies = 0
# timeouts & retries
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_fin_timeout = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# sysctl -p # reloads settings

58. nf_conntrack_max
- limited by kernel memory
- decreased tcp_timeout_time_wait
- decreased tcp_fin_timeout
- no workie
- we disabled connection tracking
altogether, and it worked!
iptables -A PREROUTING -p tcp --dport
80 -j NOTRACK

60. The workers!

61. Decrease
memory usage.

62. config/environment.rb
config.frameworks -= [:action_view]

63. Speed
them up.

64. Benchmark!

65. Imlib2 >
ImageMagick

66. Paperclip
Processor

67. Throttle (only)
when needed.

68. Angry Boss!

69. Angry Boss!
- 11 jobs pending. 6 workers running. On loop 75036!
- Spawning worker #7...
- [Worker #6] Report.process! completed after 2.1819
- [Worker #2] Report.process! completed after 2.6006
- [Worker #1] Notifier.deliver_report_notification
completed after 0.1067
- 8 jobs pending. 7 workers running. On loop 75036!
- Spawning worker #8...

70. Imlib2 + Angry Boss!

71. What we
learned.

72. Rails can scale.

73. But it needs
a bit of help.

74. All apps
are different.

75. Don't just
follow the book.

76. What's next.

82. That's it.

84. Drew, Tom, Yehuda, Benny
the guys that make it possible

85. <a href>
Git repos
github.com/prey
Angry Boss, Resizer, etc
github.com/tomas
Home
preyproject.com

86. Thanks!
Carlos Yaconi
@cyaconi
Tomás Pollak
@tomaspollak forkhq.com