It’s a story of how we evolved the cloud-based IoT backend platform behind Gigaset elements to meet the tough business requirements. We needed to make it highly reliable while processing high volumes of events and to enable development teams to deliver new features to customers at rapid pace. What we started with was a monolith application that we struggled to operate, to make it reliable and to deliver updates. And it caused different kinds of bottlenecks for our development teams. We moved from this to a microservices architecture. The interesting parts of it are: decisions on how to cut monolith into small pieces, technology choices, automation used. And how we made this change while serving the customers and delivering new features all the time. Also the benefits of change are interesting: much higher deployment rate, shorter lead time to deliver new code and much shorter MTTR. Now, two years after that change we already see a need for another one. We need even higher velocity and more dynamism in the platform. And we realize that it’s hard to get it using our current tools. And that’s why we already work on moving to a cloud native infrastructure. This talk is a kind of case study on how the benefits of different DevOps practices help us to iterate rapidly on building our product. Presented at Atmosphere Conference on 2018-06-18: https://atmosphere-conference.com/lecture.html#id=44851

1. Evolution of
the IoT backend platform
Tomasz Tarczyński, Gigaset

2. Smart Home
@ttarczynski

3. Gigaset elements
@ttarczynski

4. Gigaset elements
@ttarczynski

5. Gigaset elements
@ttarczynski

6. Requirements of IoT Backend
● API for devices
● API for frontend applications
● Fast internal event processing
● Storage for events
@ttarczynski

7. Requirements of IoT Backend
1. Reliability
@ttarczynski

8. @ttarczynski
Requirements of IoT Backend
2. Velocity

9. First Generation IoT Backend

10. A monolith app
● All backend
functionality in 1 app
● Deployed in the cloud
● Infrastructure as a
Service

11. A monolith app
● All backend
functionality in 1 app
● Deployed in the cloud
● Infrastructure as a
Service
Rev.
Proxy
DB
App

12. A monolith app
● High Availability
● Resistant to
individual
component failure
Load Balancer
Rev.
Proxy
DB
App
Rev.
Proxy
Rev.
Proxy
DB
AppApp
DB
Load Balancer

13. Teams building it
@ttarczynski
Dev
Deliver new
features fast

14. Teams building it
Dev
Deliver new
features fast
@ttarczynski
Ops
Reliability

15. Limitations of the Monolith
Big releases
(Long tests)
@ttarczynski

16. Limitations of the Monolith
Painful deployments
(Ops manual work)
@ttarczynski

17. Limitations of the Monolith
Failures
@ttarczynski

18. Limitations of the Monolith
Infrequent releases
(monthly)
@ttarczynski

19. Limitations of the Monolith
Developers work tightly coupled
With: Front-End Dev, Ops, TestLab
@ttarczynski

20. Limitations of the Monolith
Developers work tightly coupled
Within the team
@ttarczynski

21. Second Generation IoT Backend

22. Microservices – Why
Decoupling of features
development
@ttarczynski

23. Microservices – Why
Frequency of deployments
@ttarczynski

24. Microservices – Why
Lower risk of deployment
@ttarczynski

25. Microservices – Why
Increase feature delivery peace
@ttarczynski

26. Microservices – How
● The monolith split into
~70 small single-purpose apps
@ttarczynski

27. Microservices – How
● The monolith split into
~70 small single-purpose apps
● Individual app developed by 1–2 devs
@ttarczynski

28. Automation
● Lower cost of single deployment
@ttarczynski

29. Automation
● Lower cost of single deployment
● Consistency
@ttarczynski

30. Automation
● Lower cost of single deployment
● Consistency
● Self-service
@ttarczynski

31. Automation
● Puppet code in version control
@ttarczynski

32. Automation
● Puppet code in version control
● Control all config of servers and apps
@ttarczynski

33. Automation
● Fully automated deployments of apps
○ With no human intervention required
@ttarczynski

34. Automation
● Fully automated deployments of apps
○ With no human intervention required
● Short time to deploy
○ 30 minutes from request to done
@ttarczynski

35. Monitoring
Rapid feedback loop
@ttarczynski

36. Monitoring
● Logs and metrics are key
@ttarczynski

37. Monitoring
● Logs and metrics are key
● Fully automated
@ttarczynski

38. Monitoring
● Logs and metrics are key
● Fully automated
● Focus on user experience
@ttarczynski

39. Monitoring
When to alert?
@ttarczynski

40. Monitoring – What
● When to alert?
@ttarczynski

41. Sharing – Why
Information flow
@ttarczynski

42. Sharing – How
● Tools
● Dashboards
● Knowledge and learnings
@ttarczynski

43. Culture
Collaboration
@ttarczynski

44. Culture
Work together every day
@ttarczynski

45. Culture
Fail Fast – Learn Fast
@ttarczynski

46. Architecture
@ttarczynski

47. Microservices – example
user
notif.
notif.
config
repo
Postgre
SQL
GCM
APNS

48. Building Blocks
APNS
analyzer
heart
beat
Elastic
search
Postgre
SQL
UI/UX
events
config
repo
GCM
analyzer
intrus.
event
log
user
AuthZ
user
notif.
Rabbit
MQ
adapter
base
station
redis
cache
reverse
channel
adapter
gobox
adapter
camera
rule
engine
UI/UX
config
state
repo
postfix

49. Migration
One bite at a time
@ttarczynski

50. Migration
Built new platform
next to existing one
@ttarczynski

51. Benefits of Second Generation IoT Backend
1. Faster iteration time
● 1 day (instead of 1 month)
@ttarczynski

52. Benefits of Second Generation IoT Backend
1. Faster iteration time
● 1 day (instead of 1 month)
2. Frequent releases
● Up to 10 production releases a day
@ttarczynski

53. Benefits of Second Generation IoT Backend
3. Fully automated deployments
@ttarczynski

54. Benefits of Second Generation IoT Backend
3. Fully automated deployments
4. Failures: improved MTTR
● (Mean Time To Recover)
● 30 minutes (instead of 4 hours)
@ttarczynski

55. Lessons Learned
Microservices require DevOps
@ttarczynski

56. Lessons Learned
Invest in observability
@ttarczynski

57. Lessons Learned
Look for balance:
Velocity vs Reliability
@ttarczynski

58. Lessons Learned
How fast can you learn?
@ttarczynski

59. Lessons Learned
Ability to adapt
@ttarczynski

60. Lessons Learned: Issues
Velocity is king
@ttarczynski

61. Lessons Learned: Issues
Complexity is hard
@ttarczynski

62. Lessons Learned: Issues
Host-centric model not optimal
@ttarczynski

63. Lessons Learned: Issues
Can one tool fix everything?
@ttarczynski

64. Future:
Third Generation IoT Backend

65. Cloud Native Platform – Why
Speed: Time to Value
@ttarczynski

66. Cloud Native Platform – Why
Scale + Reliability
@ttarczynski

67. Cloud Native Platform – Why
Efficient usage of resources
@ttarczynski

68. Cloud Native Platform – Why
VMs are the wrong abstraction
@ttarczynski

69. Cloud Native Platform – How
Tools?
@ttarczynski

70. Cloud Native Platform – How
Experiments
@ttarczynski

71. Cloud Native Platform – How
Orchestrator changes everything
@ttarczynski

72. Cloud Native Platform – How
Loads of learning
@ttarczynski

73. Cloud Native Platform – How
CNCF Trail Map
@ttarczynski

74. Cloud Native Platform – How

75. Cloud Native Platform
@ttarczynski

76. Cloud Native Platform
1. Use Infrastructure-as-a-Service (IaaS)
@ttarczynski

77. Cloud Native Platform
1. Use Infrastructure-as-a-Service (IaaS)
2. Microservices architecture
@ttarczynski

78. Cloud Native Platform
1. Use Infrastructure-as-a-Service (IaaS)
2. Microservices architecture
3. Containerize
@ttarczynski

79. Cloud Native Platform
1. Use Infrastructure-as-a-Service (IaaS)
2. Microservices architecture
3. Containerize
4. Automate
@ttarczynski

80. Cloud Native Platform
1. Use Infrastructure-as-a-Service (IaaS)
2. Microservices architecture
3. Containerize
4. Automate
5. Orchestrate
@ttarczynski

81. Summary
● Monolith makes you slow
● Microservices:
great benefits and big challenges
● Cloud Native:
to fully benefit from Microservices
@ttarczynski

82. Thanks!
Tomasz Tarczynski
@ttarczynski