This is the talk given at Highload++ 2014 in Moscow, Russia. The topic was partition tolerance testing of Galera in a noisy high load environment with NetEm and Docker.

1. Corpus collapsum
Partition tolerance of Galera in a noisy high load
environment
Highload++ 2014
Raghavendra Prabhu
 raghavendra.d.prabhu@gmail.com
Percona  raghavendra.prabhu@percona.com
 randomsurfer  wnohang.net  rdprabhu  ronin13

2. The Title?

3. Our Cluster

4. Split brain

5. Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter
Deutsch
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
5 / 92

6. Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter
Deutsch
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
6 / 92

7. Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter
Deutsch
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
7 / 92

8. Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter
Deutsch
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
8 / 92

9. The fallacies
▶ The network is reliable
▶ Latency is zero
▶ Bandwidth is infinite
▶ The network is secure
9 / 92

10. The fallacies
▶ Topology doesn’t change
▶ There is one administrator
▶ Transport cost is zero
▶ The network is homogeneous
10 / 92

11. 20000 feet view

12. Actors
▶ Database - WSREP/PXC
▶ Plugin - Galera
▶ Traffic control
♦ Traffic Control - tc
♦ NetEm
12 / 92

13. Actors
▶ Database - WSREP/PXC
▶ Plugin - Galera
▶ Traffic control
♦ Traffic Control - tc
♦ NetEm
13 / 92

14. Actors
▶ Database - WSREP/PXC
▶ Plugin - Galera
▶ Traffic control
♦ Traffic Control - tc
♦ NetEm
14 / 92

15. Actors
▶ Containers - Docker
▶ Load
♦ Generators - Sysbench, RQG
▶ Network
♦ Dnsmasq
♦ nsenter
15 / 92

16. Actors
▶ Containers - Docker
▶ Load
♦ Generators - Sysbench, RQG
▶ Network
♦ Dnsmasq
♦ nsenter
16 / 92

17. Actors
▶ Jenkins
♦ Build flow and CI
▶ Storage
♦ Why
17 / 92

18. But why
▶ The ‘P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
18 / 92

19. But why
▶ The ‘P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
19 / 92

20. But why
▶ The ‘P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
20 / 92

21. But why
▶ The ‘P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
21 / 92

22. But why
▶ Failures in warehouses.
▶ Not quorum, but consensus.
▶ Real world networks and synchronous replication
- Delay
- Partition
22 / 92

23. Galera

24. Galera
▶ Data-centric approach
▶ EVS
▶ Causality and Synchronous
▶ Latency
24 / 92

28. Where did it start

29. Where did it start
▶ Bug! https://bugs.launchpad.net/galera/+bug/1274192
▶ Loss of PC
▶ Crash
▶ HA goal
29 / 92

30. One can bring the whole down

31. The Flow

32. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
32 / 92

33. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
33 / 92

34. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
34 / 92

35. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
35 / 92

36. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
36 / 92

37. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
37 / 92

38. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
38 / 92

39. Basic Flow
Jenkins Build image(s?) Start Dnsmasq Bootstrap
nsenter/netem Pre-sanity SST/Others Load/Sysbench
39 / 92

40. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
40 / 92

41. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
41 / 92

42. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
42 / 92

43. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
43 / 92

44. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
44 / 92

45. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
45 / 92

46. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
46 / 92

47. Basic Flow
RR sysbench
Detach/Keep
Post sanity Core trace
Sanity check Reconciliation
Examine!
Cleanup Collect logs
47 / 92

48. Cluster Resilience
48 / 92

49. Parameters
▶ Sysbench
▶ Segments
▶ Reconciliation period
▶ Number of nodes with loss
49 / 92

50. Parameters
▶ Sysbench
▶ Segments
▶ Reconciliation period
▶ Number of nodes with loss
50 / 92

51. Parameters
▶ Sysbench
▶ Segments
▶ Reconciliation period
▶ Number of nodes with loss
51 / 92

52. Parameters
▶ Sysbench
▶ Segments
▶ Reconciliation period
▶ Number of nodes with loss
52 / 92

53. Parameters
▶ NetEm
▶ Detach qdisc or keep it?
▶ Fsync
▶ Shutdown
53 / 92

54. Parameters
▶ NetEm
▶ Detach qdisc or keep it?
▶ Fsync
▶ Shutdown
54 / 92

55. Parameters
▶ NetEm
▶ Detach qdisc or keep it?
▶ Fsync
▶ Shutdown
55 / 92

56. Parameters
▶ NetEm
▶ Detach qdisc or keep it?
▶ Fsync
▶ Shutdown
56 / 92

57. Parameters

58. Containers!

59. Docker
▶ Why not virtualize
♦ Occam
♦ Namespaces
▶ Simplicity
♦ Network
♦ One application per node
59 / 92

60. Docker
▶ Portability
- See same qualitative behavior that I do.
▶ Reproducibility
- Makes it determinstic
▶ Configurable and CI
- Byproducts
60 / 92

61. Docker
▶ QEMU and Docker
▶ Scalability
♦ Performance
♦ Feature
▶ Abstraction of channels
61 / 92

62. Container Networking
▶ Linking didn’t help
▶ Dnsmasq to rescue!
♦ Hosts file and volumes
♦ SIGHUP and refresh
▶ More elegant methods
- Swarm
62 / 92

63. Noise
▶ Initial setup
- Bridge
- Egress only
- IFB
▶ Present state
▶ NetEm
- tc qdisc buckets
- packet loss, delay, corruption, duplication, reordering
- nsenter
▶ Future
- Docker exec
63 / 92

64. Testing methods

65. Method I
▶ Qdisc is detached after load
▶ Objective
- Time to recover of full cluster
▶ Done with a larger subset
65 / 92

66. Method II
▶ Qdisc is kept till the end
▶ Objective
- Formation of primary component
▶ Comparatively smaller subset
66 / 92

67. Observations
▶ Post sanity types
- Why
- Whom
▶ Which method is more pertinent
▶ State transfer issues
- Beginning
- During re-emergence
67 / 92

68. Observations
▶ Direct load to affected nodes
▶ Logs
- journalctl
- Streaming?
68 / 92

69. Other noises
▶ Aim
▶ Fsync
- libeatmydata
- Variance
▶ Correlation with network
▶ How with Docker
- LD_PRELOAD
69 / 92

70. System Load

71. Load generation
▶ Sysbench
- Generation
- Reconnect on partition
▶ Sockets chosen
- Load on affected nodes
▶ Distribution of Load
- RR with socat
- Native sysbench support
- HAProxy?
71 / 92

72. Load generation
▶ Nature of data/load
- DDL
▶ RQG in future
- Fuzz testing
72 / 92

73. The Fix

74. Strike Out!

75. Eviction
▶ STONITH
▶ Permanent eviction
▶ ’N’ strikes & out!
- Timers - evs parameters
- wsrep_evs_delayed and wsrep_evs_evict_list
75 / 92

76. Eviction
▶ Aim
▶ Quorum required
- Keep majority of the group live or to avoid eviction.
- Why? - Not shoot each other
- Non-PC nodes also.
76 / 92

77. Eviction
▶ Aim
▶ Quorum required
- Keep majority of the group live or to avoid eviction.
- Why? - Not shoot each other
- Non-PC nodes also.
77 / 92

78. Eviction
▶ EVS version and upgrade
▶ TODO!
- Ingress only
- Follow here.
▶ Credits to Teemu Ollakka, Yan Zhang and Alex Yurchenko from codership.
78 / 92

79. Coredumps with Docker
▶ Breakdown of abstraction
▶ Lack of isolation
▶ What was done
- Volumes
- core_pattern & sysctl
- suid and ulimit
79 / 92

80. WAN Segments
▶ How they work
▶ Random allocation
▶ Joiner starvation
▶ Simulates data center
▶ Donor selection
80 / 92

81. The code
▶ Github: https://github.com/percona/pxc-docker
▶ Jenkins: http://jenkins.percona.com/job/PXC-5.6-netem/
▶ Contributions/testing welcome!
▶ Dependencies
- Sysbench
81 / 92

82. Code: todo
▶ Docker automated builds
▶ Orchestration
- Kubernetes, Mesos et.al.
▶ Docker
♦ Injection
♦ Signal proxying
82 / 92

83. Code: todo
▶ Poll and exit during reconciliation
- Instrument v/s number of nodes
▶ Use actual channels
▶ Run it bare - CoreOS etc.
▶ Overlay with etcd/fleet/libswarm
83 / 92

84. Future work

85. Future work
▶ Fault injection
♦ Memory
- Poisoned memory/madvise
♦ Disk
- libeatmydata
- Opposite
- ENOSPC
85 / 92

86. Fault injection
▶ CPU
- NUMA?
- Hotplug
▶ More network
- corruption, duplication, reordering, rate-limit
- Better distribution
- Other shaping
86 / 92

87. More Chaos

88. Future work
▶ Disturb cluster more!
- Membership changes
* Manual eviction
* Pull the cord!
- Corrupt nodes
▶ Consistency voting
88 / 92

89. Further Reading
▶ Jepsen
▶ Byzantine fault tolerance
- Reaching agreement in presence of faults
▶ The Network is Reliable
▶ NetEm
▶ Latency: The New Web Performance Bottleneck
▶ Galera Cluster Documentation
▶ Auto eviction code
▶ Don’t Settle for Eventual Consistency
▶ Extended Virtual Synchrony
89 / 92

90. About
▶ /me: Raghavendra Prabhu, Product Lead, Percona XtraDB Cluster, Percona.
▶ Slides will be at slideshare.
▶ About.me: raghavendra.prabhu
▶ Keybase.io: rdprabhu
▶ Presentation under CC BY-SA 4.0
90 / 92

91. Image Credits
▶ http://galeracluster.com/documentation-webpages/
▶ http://www.thelastdragontribute.com/40th-anniversary-death-of-bruce-lee/
▶ https://upload.wikimedia.org/wikipedia/commons/6/60/Corpus_callosum.png
▶ http://www.thebarrow.org/Neurological_Services/Epilepsy/204354
▶ https://flic.kr/p/9J6GNu
▶ https://secure.flickr.com/photos/brewbooks/7780990192
▶ https://www.flickr.com/photos/kwerfeldein/2649294869
▶ https://secure.flickr.com/photos/mindmob/51951632
▶ https://secure.flickr.com/photos/arenamontanus/2227769907
▶ https://www.flickr.com/photos/markop/477199204
▶ http://galeracluster.com/wp-content/uploads/2013/10/galera_replication1.png
▶ https://www.flickr.com/photos/gcwest/281385801
▶ https://www.flickr.com/photos/opethdamna/360934079
▶ http://digital-amphetamine.deviantart.com/art/Sky-82555664
▶ http://highload.co/i/logo.png
▶ https://flic.kr/p/xTT8n
▶ https://www.flickr.com/photos/29233640@N07/13466208953
▶ https://www.flickr.com/photos/bob_in_thailand/9782777742/
91 / 92

92. Spaseeba & Da sveedaneeya!