1. Building a Resilient Cloud
Infrastructure. From Scratch.
by Jeremy Jarvis
Co-founder

2. “The Brightbox story.”

3. Overview
@jeremyjarvis

4. Overview
• Preface: Who, what etc.
@jeremyjarvis

5. Overview
• Preface: Who, what etc.
• Chapter 1: Feeling the pain
@jeremyjarvis

6. Overview
• Preface: Who, what etc.
• Chapter 1: Feeling the pain
• Chapter 2: Realising an opportunity
@jeremyjarvis

7. Overview
• Preface: Who, what etc.
• Chapter 1: Feeling the pain
• Chapter 2: Realising an opportunity
• Chapter 3: Design phase
@jeremyjarvis

8. Overview
• Preface: Who, what etc.
• Chapter 1: Feeling the pain
• Chapter 2: Realising an opportunity
• Chapter 3: Design phase
• Chapter 4: Let’s build this thing!
@jeremyjarvis

9. Overview
• Preface: Who, what etc.
• Chapter 1: Feeling the pain
• Chapter 2: Realising an opportunity
• Chapter 3: Design phase
• Chapter 4: Let’s build this thing!
• Chapter 5: It’s alive!
@jeremyjarvis

10. Overview
• Preface: Who, what etc.
• Chapter 1: Feeling the pain
• Chapter 2: Realising an opportunity
• Chapter 3: Design phase
• Chapter 4: Let’s build this thing!
• Chapter 5: It’s alive!
• Epilogue: Some conclusions/lessons
@jeremyjarvis

11. Preface.
@jeremyjarvis

12. Preface.
• Agile cloud infrastructure service
@jeremyjarvis

13. Preface.
• Agile cloud infrastructure service
• “Multi-zone” (datacentre) architecture
@jeremyjarvis

14. Preface.
• Agile cloud infrastructure service
• “Multi-zone” (datacentre) architecture
• UK-based (HQ in Leeds, DCs in Manchester)
@jeremyjarvis

15. Preface.
• Agile cloud infrastructure service
• “Multi-zone” (datacentre) architecture
• UK-based (HQ in Leeds, DCs in Manchester)
• Small team (still only 10 in total)
@jeremyjarvis

16. Preface.
• Agile cloud infrastructure service
• “Multi-zone” (datacentre) architecture
• UK-based (HQ in Leeds, DCs in Manchester)
• Small team (still only 10 in total)
• Distributed team (Mainly around Leeds)
@jeremyjarvis

17. Preface.
• Agile cloud infrastructure service
• “Multi-zone” (datacentre) architecture
• UK-based (HQ in Leeds, DCs in Manchester)
• Small team (still only 10 in total)
• Distributed team (Mainly around Leeds)
• Developer/DevOps focused (it’s who we are)
@jeremyjarvis

18. Preface.
• Agile cloud infrastructure service
• “Multi-zone” (datacentre) architecture
• UK-based (HQ in Leeds, DCs in Manchester)
• Small team (still only 10 in total)
• Distributed team (Mainly around Leeds)
• Developer/DevOps focused (it’s who we are)
• Profitable (for 4 yrs, built from revenue)
@jeremyjarvis

19. Chapter 1: Feeling the Pain
@jeremyjarvis

20. Chapter 1: Feeling the Pain
@jeremyjarvis

21. Chapter 1: Feeling the Pain
• Launched Ruby-specific hosting service (Sep 2007)
@jeremyjarvis

22. Chapter 1: Feeling the Pain
• Launched Ruby-specific hosting service (Sep 2007)
• Acquisition interest (no thanks!)
@jeremyjarvis

23. Chapter 1: Feeling the Pain
• Launched Ruby-specific hosting service (Sep 2007)
• Acquisition interest (no thanks!)
• Built a good reputation, hosted apps for some large
customers
@jeremyjarvis

24. Chapter 1: Feeling the Pain
• Launched Ruby-specific hosting service (Sep 2007)
• Acquisition interest (no thanks!)
• Built a good reputation, hosted apps for some large
customers
• Systems stable, but became unwieldy and hard for us
to manage + we wanted to expand offering
@jeremyjarvis

25. Chapter 1: Feeling the Pain
• Launched Ruby-specific hosting service (Sep 2007)
• Acquisition interest (no thanks!)
• Built a good reputation, hosted apps for some large
customers
• Systems stable, but became unwieldy and hard for us
to manage + we wanted to expand offering
• Began looking at existing options, nothing fitted our
needs (e.g Eucalyptus)
@jeremyjarvis

26. Chapter 1: Feeling the Pain
• Launched Ruby-specific hosting service (Sep 2007)
• Acquisition interest (no thanks!)
• Built a good reputation, hosted apps for some large
customers
• Systems stable, but became unwieldy and hard for us
to manage + we wanted to expand offering
• Began looking at existing options, nothing fitted our
needs (e.g Eucalyptus)
• Decided to build our own “cloud” (Brightbox NG)
@jeremyjarvis

27. Chapter 2: Realising an opportunity
@jeremyjarvis

28. Chapter 2: Realising an opportunity
@jeremyjarvis

29. Chapter 2: Realising an opportunity
• Limited options in EU/UK
@jeremyjarvis

30. Chapter 2: Realising an opportunity
• Limited options in EU/UK
• If we’re building this for ourselves why not sell it?
@jeremyjarvis

31. Chapter 2: Realising an opportunity
• Limited options in EU/UK
• If we’re building this for ourselves why not sell it?
• Generated lots of internal debate
@jeremyjarvis

32. Chapter 2: Realising an opportunity
• Limited options in EU/UK
• If we’re building this for ourselves why not sell it?
• Generated lots of internal debate
• Lack of flexibility with PaaS services
@jeremyjarvis

33. Chapter 2: Realising an opportunity
• Limited options in EU/UK
• If we’re building this for ourselves why not sell it?
• Generated lots of internal debate
• Lack of flexibility with PaaS services
• We’re good at this stuff + we have awesome team
@jeremyjarvis

34. Chapter 2: Realising an opportunity
• Limited options in EU/UK
• If we’re building this for ourselves why not sell it?
• Generated lots of internal debate
• Lack of flexibility with PaaS services
• We’re good at this stuff + we have awesome team
• Decided to shift development focus
@jeremyjarvis

35. Chapter 3: Design phase
@jeremyjarvis

36. Chapter 3: Design phase
@jeremyjarvis

37. Chapter 3: Design phase
• Requirements:
@jeremyjarvis

38. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
@jeremyjarvis

39. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
– Resilient network (multiple connections)
@jeremyjarvis

40. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
– Resilient network (multiple connections)
– Agile network layer (Load Balancing, Cloud IP, Firewall)
@jeremyjarvis

41. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
– Resilient network (multiple connections)
– Agile network layer (Load Balancing, Cloud IP, Firewall)
– Distributed (no SPOF)
@jeremyjarvis

42. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
– Resilient network (multiple connections)
– Agile network layer (Load Balancing, Cloud IP, Firewall)
– Distributed (no SPOF)
– Modular (easy to grow, JIT)
@jeremyjarvis

43. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
– Resilient network (multiple connections)
– Agile network layer (Load Balancing, Cloud IP, Firewall)
– Distributed (no SPOF)
– Modular (easy to grow, JIT)
– Future-proof (IPv6)
@jeremyjarvis

44. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
– Resilient network (multiple connections)
– Agile network layer (Load Balancing, Cloud IP, Firewall)
– Distributed (no SPOF)
– Modular (easy to grow, JIT)
– Future-proof (IPv6)
– Programmable (API, CLI)
@jeremyjarvis

45. Chapter 3: Design phase
• Requirements:
– Geographic redundancy (separate datacentres)
– Resilient network (multiple connections)
– Agile network layer (Load Balancing, Cloud IP, Firewall)
– Distributed (no SPOF)
– Modular (easy to grow, JIT)
– Future-proof (IPv6)
– Programmable (API, CLI)
– Open (easy to get stuff in and out)
@jeremyjarvis

46. Chapter 3: Design phase
@jeremyjarvis

47. Chapter 3: Design phase
• Process:
@jeremyjarvis

48. Chapter 3: Design phase
• Process:
– Consulted, at first. Network infrastructure gurus.
@jeremyjarvis

49. Chapter 3: Design phase
• Process:
– Consulted, at first. Network infrastructure gurus.
– We’re on our own!
@jeremyjarvis

50. Chapter 3: Design phase
• Process:
– Consulted, at first. Network infrastructure gurus.
– We’re on our own!
– Hands-on, R&D style, lots of testing, experimentation, iteration
@jeremyjarvis

51. Chapter 3: Design phase
• Process:
– Consulted, at first. Network infrastructure gurus.
– We’re on our own!
– Hands-on, R&D style, lots of testing, experimentation, iteration
– Investigated competition (what’s good/bad)
@jeremyjarvis

52. Chapter 3: Design phase
• Process:
– Consulted, at first. Network infrastructure gurus.
– We’re on our own!
– Hands-on, R&D style, lots of testing, experimentation, iteration
– Investigated competition (what’s good/bad)
– Access to plenty of kit -set up mini-clouds to hack
@jeremyjarvis

53. Chapter 3: Design phase
• Process:
– Consulted, at first. Network infrastructure gurus.
– We’re on our own!
– Hands-on, R&D style, lots of testing, experimentation, iteration
– Investigated competition (what’s good/bad)
– Access to plenty of kit -set up mini-clouds to hack
– John and Neil worked very closely on architecture (daily calls)
@jeremyjarvis

54. Chapter 3: Design phase
• Process:
– Consulted, at first. Network infrastructure gurus.
– We’re on our own!
– Hands-on, R&D style, lots of testing, experimentation, iteration
– Investigated competition (what’s good/bad)
– Access to plenty of kit -set up mini-clouds to hack
– John and Neil worked very closely on architecture (daily calls)
– Don’t reinvent the wheel (Rubyists!)
@jeremyjarvis

55. Chapter 3: Design phase
@jeremyjarvis

56. Chapter 3: Design phase
• Elements:
@jeremyjarvis

57. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
@jeremyjarvis

58. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
– Application/software architecture (evolving, iterative)
@jeremyjarvis

59. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
– Application/software architecture (evolving, iterative)
– Hardware selection:
@jeremyjarvis

60. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
– Application/software architecture (evolving, iterative)
– Hardware selection:
• Border routers (Cisco)
@jeremyjarvis

61. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
– Application/software architecture (evolving, iterative)
– Hardware selection:
• Border routers (Cisco)
• Switches (Cisco +)
@jeremyjarvis

62. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
– Application/software architecture (evolving, iterative)
– Hardware selection:
• Border routers (Cisco)
• Switches (Cisco +)
• Host servers (Dell)
@jeremyjarvis

63. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
– Application/software architecture (evolving, iterative)
– Hardware selection:
• Border routers (Cisco)
• Switches (Cisco +)
• Host servers (Dell)
• Standardised rack design (modular)
@jeremyjarvis

64. Chapter 3: Design phase
• Elements:
– Network architecture (proof-of-concept)
– Application/software architecture (evolving, iterative)
– Hardware selection:
• Border routers (Cisco)
• Switches (Cisco +)
• Host servers (Dell)
• Standardised rack design (modular)
– Datacentre selection (Proximity, Independent, Competing)
@jeremyjarvis

65. Chapter 4: Let’s build this thing!
@jeremyjarvis

66. Chapter 4: Let’s build this thing!
@jeremyjarvis

67. Chapter 4: Let’s build this thing!
• Network provisioning (Transit, Metro links, RIPE
membership)
@jeremyjarvis

68. Chapter 4: Let’s build this thing!
• Network provisioning (Transit, Metro links, RIPE
membership)
• Buying (negotiating)
@jeremyjarvis

69. Chapter 4: Let’s build this thing!
• Network provisioning (Transit, Metro links, RIPE
membership)
• Buying (negotiating)
• Installing kit (Routers, Switches, Servers)
@jeremyjarvis

70. Chapter 4: Let’s build this thing!
• Network provisioning (Transit, Metro links, RIPE
membership)
• Buying (negotiating)
• Installing kit (Routers, Switches, Servers)
• Use datacentre staff where possible (racking/stacking)
@jeremyjarvis

71. Chapter 4: Let’s build this thing!
• Network provisioning (Transit, Metro links, RIPE
membership)
• Buying (negotiating)
• Installing kit (Routers, Switches, Servers)
• Use datacentre staff where possible (racking/stacking)
• Software development (several applications, iterative)
@jeremyjarvis

72. Chapter 4: Let’s build this thing!
• Network provisioning (Transit, Metro links, RIPE
membership)
• Buying (negotiating)
• Installing kit (Routers, Switches, Servers)
• Use datacentre staff where possible (racking/stacking)
• Software development (several applications, iterative)
• Writing configs (again, iterative - infra as code)
@jeremyjarvis

73. Chapter 4: Let’s build this thing!
• Network provisioning (Transit, Metro links, RIPE
membership)
• Buying (negotiating)
• Installing kit (Routers, Switches, Servers)
• Use datacentre staff where possible (racking/stacking)
• Software development (several applications, iterative)
• Writing configs (again, iterative - infra as code)
• Documentation (wiki, changelogs, code comments)
@jeremyjarvis

74. Chapter 5: It’s alive!
@jeremyjarvis

75. Chapter 5: It’s alive!
@jeremyjarvis

76. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
@jeremyjarvis

77. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
• Iteration (fix bottlenecks, improve resilience)
@jeremyjarvis

78. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
• Iteration (fix bottlenecks, improve resilience)
• New features (Load balancing, Firewall etc)
@jeremyjarvis

79. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
• Iteration (fix bottlenecks, improve resilience)
• New features (Load balancing, Firewall etc)
• Billing data (distributed stats collection)
@jeremyjarvis

80. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
• Iteration (fix bottlenecks, improve resilience)
• New features (Load balancing, Firewall etc)
• Billing data (distributed stats collection)
• “Enterprise” billing software *sad face*
@jeremyjarvis

81. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
• Iteration (fix bottlenecks, improve resilience)
• New features (Load balancing, Firewall etc)
• Billing data (distributed stats collection)
• “Enterprise” billing software *sad face*
• Oct 2011 - General availability (no change really, just £
£)
@jeremyjarvis

82. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
• Iteration (fix bottlenecks, improve resilience)
• New features (Load balancing, Firewall etc)
• Billing data (distributed stats collection)
• “Enterprise” billing software *sad face*
• Oct 2011 - General availability (no change really, just £
£)
• Product development (more and better features)
@jeremyjarvis

83. Chapter 5: It’s alive!
• Nov 2010 - Launched private beta (700 users)
• Iteration (fix bottlenecks, improve resilience)
• New features (Load balancing, Firewall etc)
• Billing data (distributed stats collection)
• “Enterprise” billing software *sad face*
• Oct 2011 - General availability (no change really, just £
£)
• Product development (more and better features)
• Marketing (getting the word out, communication)
@jeremyjarvis

84. Epilogue: What did we learn?
@jeremyjarvis

85. Epilogue: What did we learn?
@jeremyjarvis

86. Epilogue: What did we learn?
• Building stuff is hard (but can lead to competitive
advantage)
@jeremyjarvis

87. Epilogue: What did we learn?
• Building stuff is hard (but can lead to competitive
advantage)
• Be your own customers (understand market, use
products)
@jeremyjarvis

88. Epilogue: What did we learn?
• Building stuff is hard (but can lead to competitive
advantage)
• Be your own customers (understand market, use
products)
• Don’t *over*estimate competition (look behind the
mask)
@jeremyjarvis

89. Epilogue: What did we learn?
• Building stuff is hard (but can lead to competitive
advantage)
• Be your own customers (understand market, use
products)
• Don’t *over*estimate competition (look behind the
mask)
• Learn good negotiation (clue: it’s not a battle)
@jeremyjarvis

90. Epilogue: What did we learn?
• Building stuff is hard (but can lead to competitive
advantage)
• Be your own customers (understand market, use
products)
• Don’t *over*estimate competition (look behind the
mask)
• Learn good negotiation (clue: it’s not a battle)
• All about the launch (could have timed/co-ordinated
things better for more “oomph”)
@jeremyjarvis

91. Epilogue: What did we learn?
• Building stuff is hard (but can lead to competitive
advantage)
• Be your own customers (understand market, use
products)
• Don’t *over*estimate competition (look behind the
mask)
• Learn good negotiation (clue: it’s not a battle)
• All about the launch (could have timed/co-ordinated
things better for more “oomph”)
• Momentum is important (PR, morale)
@jeremyjarvis

92. Thanks.
Any questions?