How to Build and Operate a Global Behavioral Change Platform (Neil Adamson, Vitality and Jesse Leresche, Entelect / Discovery) Kafka Summit London 2019
This talk will focus on the move from a monolithic solution to an event driven microservices architecture to allow each of our partners to offer their clients a customizable and localized Vitality offering that is based on a consistent global experience. It will include details on how we manage client demands and legal and regulatory issues specific to each market, how we can rapidly implement Vitality into a country within the space of a few months and then how we manage and support each market. It will describe wow we moved off proprietary technologies to a cloud agnostic open source, horizontally scalable hosted solution that takes advantage of technologies such as Kafka and Kubernetes etc. and the challenges faced in doing this.
In addition it will provide detail as to how Vitality streams exercise activity data real time from a multitude of device manufacturers (such as Fitbit, Garmin, Suunto, Apple) and routes it to the correct Vitality instance to analyze and allocate points to the member. Globally we process on average 50 -60 million member workouts a week. As well as how we integrate with a number of rewards partners to ensure members can access and utilize their rewards through local partners (gyms, airlines and cinemas) as well as global partners such as Starbucks, Hotels.com and Amazon.
Similar to How to Build and Operate a Global Behavioral Change Platform (Neil Adamson, Vitality and Jesse Leresche, Entelect / Discovery) Kafka Summit London 2019
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...confluent
Similar to How to Build and Operate a Global Behavioral Change Platform (Neil Adamson, Vitality and Jesse Leresche, Entelect / Discovery) Kafka Summit London 2019 (20)
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
How to Build and Operate a Global Behavioral Change Platform (Neil Adamson, Vitality and Jesse Leresche, Entelect / Discovery) Kafka Summit London 2019
1. 11
2 0 1 9 – K a f k a S u m m i t – L o n d o n
Vitality ONE
2. 22
”Provide a global behavioural change platform that
is quick to implement, easy to configure and efficient
to operate - serving over 50 million global users by
2023”
Vision
3. 33
Vitality ONE Platform Objective
Create a platform that can adapt to the needs of different markets
Make the Vitality One Platform the preferred platform for the development of all
Behavioral Change Functionality by:
• Allowing for rapid development and deployment cycles
• Unblocking delivery bottlenecks and enabling empowered development / federated development
• Creating technical tools that facilitate rapid development into the platform
• Providing optimal support, consulting, training to all development teams
• Providing rational and practical collaborative governance
• Continuing to secure, enhance and improve the platform
• Ensuring the Integrity and Capability of the Platform – so development work is consistent, of a high
quality and predictable
V1 Accreditation is the process of training partners on how to develop in the V1 style / pattern
and to educate them on the governance structures and processes that need to be adhered to
when developing features for the V1 solution.
4. 44
V1 Platform supports over 1m registered users in 6 months
1st Quarter 2019
KOREA
1 000 000+
200 000+
P L A T F O R M
>1000 100 000
next 6 months
5. 55
Q2 2019 Q3 2019 Q4 2019
Group Benefits
Q1 2020
USA
Vitality ONE Market Rollouts planned and in progress
Q2 2020
GERMANY
New Markets
AUSTRIA
New Markets
P L A T F O R M
Ecuador
Argentina
Netherlands
Chile
Saudi Arabia
Argentina -2
6. 66
Ingres
Nodes
Master Nodes
Worker Nodes
Container Management Environment
Scale worker
nodes to
increase
throughput
WS02
API Manager & ESB
Database
WS02
IDM Manager
Elastic
Search
Kafka
Messaging
Zookeeper -
Distributed
Co-ordination
ETCD
Nodes
Liferay
Portal Server
Kibana
CF Hub
Worker Nodes contain:
Vitality One Microservices – approx. 60
Business Rules
Zuul - load balancing
Eureka – service registry
Config Server
“Standalone” Environment
Environment
Management
Auditing and
logging
SFTP
Production Environment Topology
Client
7. 77
Developed centrally and deployed / run in various
jurisdictions
UAT Prod
Central Asset
Dev
Central
Asset Test
Central
Asset QA
UAT Prod
UAT Prod
UAT Prod
UAT Prod
BUILD A WORLD CLASS TECHNOLOGY PLATFORM TO CATER FOR THE REQUIREMENTS OF THE BUSINESS
8. 88
So what’s inside the platform?
V 1 P L A T F O R M
Generic Business Capabilities
Client Management Product Management Contract Management Finance Management
Behavioural Change
Assessment
Partner Integration
Activities
Goals
Incentivisation
Rewards
Purchases
Gamification
Vitality-Specific CapabilitiesPoints Status
Gateway
IP
VDX
9. 99
Kafka topics used extensively
V 1 P L A T F O R M
Generic Business Capabilities
Client Management Product Management Contract Management Finance Management
Behavioural Change
Assessment
Partner Integration
Activities
Goals
Incentivisation
Rewards
Purchases
Gamification
Vitality-Specific CapabilitiesPoints Status
Gateway
IP
VDX
Event Driven through Kafka topics
Event Driven through Kafka topics
Event Driven through Kafka topics
Event Driven through Kafka topics
10. 1010
Vitality ONE
Back-end
Vitality ONE
Front-ends
Vitality ONE
Back-end
Partner
Front-ends
Vitality ONE APIs
Vitality ONE
Back-end
Partner
Front-ends
Vitality ONE APIs
Partner developed functions
As partners build into the platform it will increase the functionality available and accelerate delivery
Vitality ONE APIs
1 2 3
Provided Federated
Vitality ONE Development Options
Create a platform that can adapt to the needs of different markets
Service Portal
MCMP
Service Portal
MCMP
Service Portal
MCMP
13. 1313
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Architectural Capabilities – How does Kafka Fit?
14. 1414
Why is Vitality ONE so different and a leader in its class
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
1. Built from the ground up to cater for international requirements – multi language / multi
tenancy / multi language / time zones etc.
2. Built using new architecture and technologies to enable a more flexible delivery structure to
allow for scale. It leverages the technologies used by companies like Google, Netflix, Linked in
where scale and performance is critical
3. Built using industry standard technologies and models to ensure that we leveraged off proven
assets
4. Built to enable configuration over development
5. Built to allow third party developers and partners to contribute to the platform
6. Can be as open as we would like it to be
7. Built on a lightweight technology stack to avoid the overheads of software enterprise licenses
8. Designed to allow the parallel implementation of multiple countries
9. Built to provide a standard Vitality experience globally to ensure brand awareness, consistency
and value
15. 1515
Our Journey to Kafka as a 3 Act Story
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Act 1: The Setup
The Problem with our
Current Architecture
Choosing Kafka as
our Future
Act 2: The Confrontation Act 3: The Resolution
Our Kafka success story
16. 1616
Act 1: The Problem with our Current Architecture
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Before V1 - Vitality was built using a JMS implementation as it’s
message broker
JMS had a few drawbacks & complexities:
• Complicated error handling strategies
• Slower MTTR from outages
• Limited horizontal scalability
• The need to consume messages (on average) faster than or at the
same rate as they are published
17. 1717
Act 1: The Problem with our Current Architecture
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
“Considering we are building a health and
wellness platform for the future, we need to
look for a messaging system from the future,
rather than relying on the systems from the
past”
- Jesse Leresche
18. 1818
Act 1: The Problem with our Current Architecture
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Needed a system that could:
• Provide infinite horizontally scalability
• Be fault tolerant & resilient
• Be able to handle large amounts of data
• Be fast
• Scale as we grew (opposed to requiring large
infrastructures from the outset)
19. 1919
Act 2: Choosing Kafka as our Future
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
2 major concerns:
• Lack of transactionality / 2-phase commit
• Guaranteed single message delivery mechanisms
20. 2020
Act 2: Choosing Kafka as our Future
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
21. 2121
Act 2: Choosing Kafka as our Future – Educate!
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Take time to educate & change the organizational culture
• Kafka is not simply a JMS replacement
• Give people time to adjust to the new concepts - Kafka
can be difficult to explain & understand!
• Be patient
22. 2222
Act 2: Choosing Kafka as our Future – RTFM
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
23. 2323
Act 2: Choosing Kafka as our Future – Keep up-to-date
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Keep as up-to-date as possible.
• Falling behind can make for complex upgrades later on
• Every new version has got more bug fixes and is more
stable – why wouldn’t you upgrade
• Not fun explaining to people that the production
outage you just had was avoidable had you upgraded
24. 2424
Act 2: Choosing Kafka as our Future – Be cautious when creating
internal topics
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
25. 2525
Act 2: Choosing Kafka as our Future – Get your monitoring in place
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Originally – used open-source Kafka viewer
• Was good for point-in-time monitoring / spot checks
• Good tool for allowing dev’s to troubleshoot and
investigate issues
• Lacked ability to compare data-over-time
• Link: //TODO//
26. 2626
Act 2: Choosing Kafka as our Future – Get your monitoring in place
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
At launch we had a large load coming in - realized we
needed more in-depth monitoring
• Ad-hoc commands describing consumers not
sustainable
• Developed Bash script & PowerBi Dashboard
• Allowed comparison of throughput, alerting on lag etc.
• Process is slow – spins up new JVM per consumer group
• Migrating to Java-based service using the Kafka Admin Client – faster
responses, more fault tolerant, less bash, easier to deploy via our
pipeline
• Has RBAC and mobile view (via PowerBi) - can share this over the
internet securely
• Link: //TODO//
27. 2727
Act 2: Choosing Kafka as our Future – Get your monitoring in place
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Dynatrace is used throughout our system for
monitoring & alerting
• Can monitor application health, uptime, disk space
etc.
• Use JMX metrics & alert & monitor on those – ISR’s
etc.
28. 2828
Act 2: Choosing Kafka as our Future – Get your monitoring in place
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Upgraded our clusters to Confluent Platform
Enterprise, we were able to make use of Confluent
Control Centre
• Allowed monitoring of consumer groups, inspecting
of messages, schemas etc.
• Allowed monitoring of over & under consumption &
historical view of data processing/throughput
• Allowed monitoring of application health, ISR’s,
broker health etc.
29. 2929
Act 3: Our Kafka success story
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
1. Total number of Messages across clusters
2. Number messages per day
3. Number of Topics per cluster
4. Number of points awarded per hour
30. 3030
Act 3: Our Kafka success story
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION
Roadmap for Kafka in V1:
• Move logs to ELK via Kafka
• Data Warehousing – moving data from the DB to Data Warehouse
using CDC & Kafka
• Migration of legacy components to Kafka
• Implement streaming – real-time fraud detection, points
calculations, system analytics
• Publishing DevOps pipeline events – services scaling up/down,
deployments, alerts etc.
31. 3131
Act 3: Our Kafka success story
DEVELOP BEST IN INDUSTRY ASSETS TO DELIVER ON OUR VISION