Slides from my Planning to Fail talk given at PHP North East conference 2013. This is a slightly longer version of the same talk given at the PHP UK conference. The talk was on how you can build resilient systems by embracing failure.

1. Planning
to fail
@davegardnerisme
#phpne13

2. dave

3. the taxi app

4. Planning
to fail

5. Planning
for failure

6. Planning
to fail

7. Why?
http://en.wikipedia.org/wiki/High_availability

8. 99.9% (three nines)
Downtime:
43.8 minutes per month
8.76 hours per year

9. 99.99% (four nines)
Downtime:
4.32 minutes per month
52.56 minutes per year

10. 99.999% (five nines)
Downtime:
25.9 seconds per month
5.26 minutes per year

11. www.whoownsmyavailability.com
?

12. www.whoownsmyavailability.com
YOU

13. The beginning

14. <?php

15. My website: single VPS running PHP + MySQL

16. No growth, low volume, simple functionality, one engineer (me!)

17. Large growth, high volume, complex functionality, lots of engineers

18. • Launched in London
November 2011
• Now in 5 cities in 3 countries
(30%+ growth every month)
• A Hailo hail is accepted around
the world every 5 seconds

19. “.. Brooks [1] reveals that the complexity
of a software project grows as the square
of the number of engineers and Leveson
[17] cites evidence that most failures in
complex systems result from unexpected
inter-component interaction rather than
intra-component bugs, we conclude that
less machinery is (quadratically) better.”
http://lab.mscs.mu.edu/Dist2012/lectures/HarvestYield.pdf

20. • SOA (10+ services)
• AWS (3 regions, 9 AZs, lots of
instances)
• 10+ engineers building services
and you?
(hailo is hiring)

21. Our overall
reliability is in
danger

22. Embracing failure
(a coping strategy)

24. VPC
(running PHP+MySQL)
reliable?

25. Reliable
!==
Resilient

26. Choosing a stack

27. “Hailo”
(running PHP+MySQL)
reliable?

28. Service Service Service Service
each service does one job well
Service Oriented Architecture

29. • Fewer lines of code
• Fewer responsibilities
• Changes less frequently
• Can swap entire implementation
if needed

30. Service
(running PHP+MySQL)
reliable?

31. Service MySQL
MySQL running on different box

32. MySQL
Service
MySQL
MySQL running in Multi-Master mode

33. Going global

34. CRUD
Locking
MySQL Search
Analytics
ID generation
also queuing…
Separating concerns

35. At Hailo we look for technologies that are:
• Distributed
run on more than one machine
• Homogenous
all nodes look the same
• Resilient
can cope with the loss of node(s) with no
loss of data

36. “There is no such thing as standby
infrastructure: there is stuff you
always use and stuff that won’t
work when you need it.”
http://blog.b3k.us/2012/01/24/some-rules.html

37. • Highly performant, scalable and
resilient data store
• Underpins much of what we do
at Hailo
• Makes multi-DC easy!

38. ZooKeeper
• Highly reliable distributed
coordination
• We implement locking and
leadership election on top of ZK
and use sparingly

39. • Distributed, RESTful, Search
Engine built on top of Apache
Lucene
• Replaced basic foo LIKE ‘%bar%’
queries (so much better)

40. NSQ
• Realtime message processing
system designed to handle
billions of messages per day
• Fault tolerant, highly available
with reliable message delivery
guarantee

41. • Real time incremental analytics
platform, backed by Apache
Cassandra
• Powerful SQL-like interface
• Scalable and highly available

42. Cruftflake
• Distributed ID generation with
no coordination required
• Rock solid

43. • All these technologies have
similar properties of distribution
and resilience
• They are designed to cope with
failure
• They are not broken by design

44. Lessons learned

45. Minimise the
critical path

46. What is the minimum viable service?

47. class HailoMemcacheService {
private $mc = null;
public function __call() {
$mc = $this->getInstance();
// do stuff
}
private function getInstance() {
if ($this->instance === null) {
$this->mc = new Memcached;
$this->mc->addServers($s);
}
return $this->mc;
}
} Lazy-init instances; connect on use

48. Configure clients
carefully

49. $this->mc = new Memcached;
$this->mc->addServers($s);
$this->mc->setOption(
Memcached::OPT_CONNECT_TIMEOUT,
$connectTimeout);
$this->mc->setOption(
Memcached::OPT_SEND_TIMEOUT,
$sendRecvTimeout);
$this->mc->setOption(
Memcached::OPT_RECV_TIMEOUT,
$sendRecvTimeout);
$this->mc->setOption(
Memcached::OPT_POLL_TIMEOUT,
$connectionPollTimeout);
Make sure timeouts are configured

50. here?
Choose timeouts based on data

51. “Fail Fast: Set aggressive timeouts
such that failing components
don’t make the entire system
crawl to a halt.”
http://techblog.netflix.com/2011/04/lessons-
netflix-learned-from-aws-outage.html

52. here?
95th percentile

53. Test

54. • Kill memcache on box A,
measure impact on application
• Kill memcache on box B,
measure impact on application
All fine.. we’ve got this covered!

55. FAIL

56. • Box A, running in AWS, locks up
• Any parts of application that
touch Memcache stop working

57. Things fail in
exotic ways

58. $ iptables -A INPUT -i eth0
-p tcp --dport 11211 -j REJECT
$ php test-memcache.php
Working OK!
Packets rejected and source notified by ICMP. Expect fast fails.

59. $ iptables -A INPUT -i eth0
-p tcp --dport 11211 -j DROP
$ php test-memcache.php
Working OK!
Packets silently dropped. Expect long time outs.

60. $ iptables -A INPUT -i eth0
-p tcp --dport 11211
-m state --state ESTABLISHED
-j DROP
$ php test-memcache.php
Hangs! Uh oh.

61. • When AWS instances hang they
appear to accept connections
but drop packets
• Bug!
https://bugs.launchpad.net/libmemcached/
+bug/583031

62. Fix, rinse, repeat

63. RabbitMQ RabbitMQ RabbitMQ
HA cluster
AMQP (port 5672)
Service

64. $ iptables -A INPUT -i eth0
-p tcp --dport 5672
-m state --state ESTABLISHED
-j DROP
$ php test-rabbitmq.php
Fantastic! Block AMQP port, client times out

65. FAIL

66. “RabbitMQ clusters do not
tolerate network partitions
well.”
http://www.rabbitmq.com/partitions.html

67. $ epmd –names
epmd: up and running on port
4369 with data:
name rabbit at port 60278
Each node listens on a port assigned by EPMD

69. $ iptables -A INPUT -i eth0
-p tcp --dport 60278
-m state --state ESTABLISHED
-j DROP
$ php test-rabbitmq.php
Hangs! Uh oh.

70. Mnesia('rabbit@dmzutilities03-global01-
test'): ** ERROR ** mnesia_event got
{inconsistent_database,
running_partitioned_network,
'rabbit@dmzutilities01-global01-test'}
application: rabbitmq_management
exited: shutdown
type: temporary
RabbitMQ logs show partitioned network error; nodes shutdown

72. while ($read < $n
&& !feof($this->sock->real_sock())
&& (false !== ($buf = fread(
$this->sock->real_sock(),
$n - $read)))) {
$read += strlen($buf);
$res .= $buf;
}
PHP library didn’t have any time limit on reading a frame

73. Fix, rinse, repeat

74. It would be
nice if we could
automate this

75. Automate!

76. • Hailo run a dedicated automated
test environment
• Powered by bash, JMeter and
Graphite
• Continuous automated testing
with failure simulations

77. Fix attempt 1: bad timeouts configured

78. Fix attempt 2: better timeouts

79. Simulate in
system tests

80. Simulate failure
Assert monitoring endpoint
picks this up
Assert features still work

81. In conclusion

82. “the best way to avoid
failure is to fail constantly.”
http://www.codinghorror.com/blog/2011/04/worki
ng-with-the-chaos-monkey.html

83. You should test for
failure
How does the software react?
How does the PHP client react?

84. Automation makes
continuous failure
testing feasible

85. Systems that cope well
with failure are easier
to operate

86. TIMED BLOCK ALL
THE THINGS

87. Thanks
Software used at Hailo
http://cassandra.apache.org/
http://zookeeper.apache.org/
http://www.elasticsearch.org/
http://www.acunu.com/acunu-analytics.html
https://github.com/bitly/nsq
https://github.com/davegardnerisme/cruftflake
https://github.com/davegardnerisme/nsqphp
Plus a load of other things I’ve not mentioned.

88. Further reading
Hystrix: Latency and Fault Tolerance for Distributed Systems
https://github.com/Netflix/Hystrix
Timelike: a network simulator
http://aphyr.com/posts/277-timelike-a-network-simulator
Notes on distributed systems for young bloods
http://www.somethingsimilar.com/2013/01/14/notes-on-distributed-
systems-for-young-bloods/
Stream de-duplication (relevant to NSQ)
http://www.davegardner.me.uk/blog/2012/11/06/stream-de-
duplication/
ID generation in distributed systems
http://www.slideshare.net/davegardnerisme/unique-id-generation-in-
distributed-systems