How to build resilient and reliable services by embracing failure.

1. Planning
to fail
@davegardnerisme
#phpuk2013

2. dave

3. the taxi app

4. Planning
to fail

5. Planning
for failure

6. Planning
to fail

7. The beginning

8. <?php

9. My website: single VPS running PHP + MySQL

10. No growth, low volume, simple functionality, one engineer (me!)

11. Large growth, high volume, complex functionality, lots of engineers

12. • Launched in London
November 2011
• Now in 5 cities in 3 countries
(30%+ growth every month)
• A Hailo hail is accepted around
the world every 5 seconds

13. “.. Brooks [1] reveals that the complexity
of a software project grows as the square
of the number of engineers and Leveson
[17] cites evidence that most failures in
complex systems result from unexpected
inter-component interaction rather than
intra-component bugs, we conclude that
less machinery is (quadratically) better.”
http://lab.mscs.mu.edu/Dist2012/lectures/HarvestYield.pdf

14. • SOA (10+ services)
• AWS (3 regions, 9 AZs, lots of
instances)
• 10+ engineers building services
and you?
(hailo is hiring)

15. Our overall
reliability is in
danger

16. Embracing failure
(a coping strategy)

18. VPC
(running PHP+MySQL)
reliable?

19. Reliable
!==
Resilient

21. Choosing a stack

22. “Hailo”
(running PHP+MySQL)
reliable?

23. Service Service Service Service
each service does one job well
Service Oriented Architecture

24. • Fewer lines of code
• Fewer responsibilities
• Changes less frequently
• Can swap entire implementation
if needed

25. Service
(running PHP+MySQL)
reliable?

26. Service MySQL
MySQL running on different box

27. MySQL
Service
MySQL
MySQL running in Multi-Master mode

28. Going global

29. CRUD
Locking
MySQL Search
Analytics
ID generation
also queuing…
Separating concerns

30. At Hailo we look for technologies that are:
• Distributed
run on more than one machine
• Homogenous
all nodes look the same
• Resilient
can cope with the loss of node(s) with no
loss of data

31. “There is no such thing as standby
infrastructure: there is stuff you
always use and stuff that won’t
work when you need it.”
http://blog.b3k.us/2012/01/24/some-rules.html

32. • Highly performant, scalable and
resilient data store
• Underpins much of what we do
at Hailo
• Makes multi-DC easy!

33. ZooKeeper
• Highly reliable distributed
coordination
• We implement locking and
leadership election on top of ZK
and use sparingly

34. • Distributed, RESTful, Search
Engine built on top of Apache
Lucene
• Replaced basic foo LIKE ‘%bar%’
queries (so much better)

35. NSQ
• Realtime message processing
system designed to handle
billions of messages per day
• Fault tolerant, highly available
with reliable message delivery
guarantee

36. Cruftflake
• Distributed ID generation with
no coordination required
• Rock solid

37. • All these technologies have
similar properties of distribution
and resilience
• They are designed to cope with
failure
• They are not broken by design

38. Lessons learned

39. Minimise the
critical path

40. What is the minimum viable service?

41. class HailoMemcacheService {
private $mc = null;
public function __call() {
$mc = $this->getInstance();
// do stuff
}
private function getInstance() {
if ($this->instance === null) {
$this->mc = new Memcached;
$this->mc->addServers($s);
}
return $this->mc;
}
} Lazy-init instances; connect on use

42. Configure clients
carefully

43. $this->mc = new Memcached;
$this->mc->addServers($s);
$this->mc->setOption(
Memcached::OPT_CONNECT_TIMEOUT,
$connectTimeout);
$this->mc->setOption(
Memcached::OPT_SEND_TIMEOUT,
$sendRecvTimeout);
$this->mc->setOption(
Memcached::OPT_RECV_TIMEOUT,
$sendRecvTimeout);
$this->mc->setOption(
Memcached::OPT_POLL_TIMEOUT,
$connectionPollTimeout);
Make sure timeouts are configured

44. here?
Choose timeouts based on data

45. “Fail Fast: Set aggressive timeouts
such that failing components
don’t make the entire system
crawl to a halt.”
http://techblog.netflix.com/2011/04/lessons-
netflix-learned-from-aws-outage.html

46. here?
95th percentile

47. Test

48. • Kill memcache on box A,
measure impact on application
• Kill memcache on box B,
measure impact on application
All fine.. we’ve got this covered!

49. FAIL

50. • Box A, running in AWS, locks up
• Any parts of application that
touch Memcache stop working

51. Things fail in
exotic ways

52. $ iptables -A INPUT -i eth0
-p tcp --dport 11211 -j REJECT
$ php test-memcache.php
Working OK!
Packets rejected and source notified by ICMP. Expect fast fails.

53. $ iptables -A INPUT -i eth0
-p tcp --dport 11211 -j DROP
$ php test-memcache.php
Working OK!
Packets silently dropped. Expect long time outs.

54. $ iptables -A INPUT -i eth0
-p tcp --dport 11211
-m state --state ESTABLISHED
-j DROP
$ php test-memcache.php
Hangs! Uh oh.

55. • When AWS instances hang they
appear to accept connections
but drop packets
• Bug!
https://bugs.launchpad.net/libmemcached/
+bug/583031

56. Fix, rinse, repeat

57. It would be
nice if we could
automate this

58. Automate!

59. • Hailo run a dedicated automated
test environment
• Powered by bash, JMeter and
Graphite
• Continuous automated testing
with failure simulations

60. Fix attempt 1: bad timeouts configured

61. Fix attempt 2: better timeouts

62. Simulate in
system tests

63. Simulate failure
Assert monitoring endpoint
picks this up
Assert features still work

64. In conclusion

65. “the best way to avoid
failure is to fail constantly.”
http://www.codinghorror.com/blog/2011/04/worki
ng-with-the-chaos-monkey.html

66. TIMED BLOCK ALL
THE THINGS

67. Thanks
Software used at Hailo
http://cassandra.apache.org/
http://zookeeper.apache.org/
http://www.elasticsearch.org/
http://www.acunu.com/acunu-analytics.html
https://github.com/bitly/nsq
https://github.com/davegardnerisme/cruftflake
https://github.com/davegardnerisme/nsqphp
Plus a load of other things I’ve not mentioned.

68. Further reading
Hystrix: Latency and Fault Tolerance for Distributed Systems
https://github.com/Netflix/Hystrix
Timelike: a network simulator
http://aphyr.com/posts/277-timelike-a-network-simulator
Notes on distributed systems for young bloods
http://www.somethingsimilar.com/2013/01/14/notes-on-distributed-
systems-for-young-bloods/
Stream de-duplication (relevant to NSQ)
http://www.davegardner.me.uk/blog/2012/11/06/stream-de-
duplication/
ID generation in distributed systems
http://www.slideshare.net/davegardnerisme/unique-id-generation-in-
distributed-systems