Facing Open Banking regulation, rapidly increasing transaction volumes and increasing customer expectations, Nationwide took the decision to take load off their back-end systems through real-time streaming of data changes into Kafka. Hear about how Nationwide started their journey with Kafka, from their initial use case of creating a real-time data cache using Change Data Capture, Kafka and Microservices to how Kafka allowed them to build a stream processing backbone used to reengineer the entire banking experience including online banking, payment processing and mortgage applications. See a working demo of the system and what happens to the system when the underlying infrastructure breaks. Technologies covered include: Change Data Capture, Kafka (Avro, partitioning and replication) and using KSQL and Kafka Streams Framework to join topics and process data.

1. Introducing Streaming at
Nationwide Building Society
May 2019
Rob Jackson and Pete Cracknell
Nationwide Building Society

2. 2
Contents
1. Nationwide Building Society
2. What is the business challenge we’re responding to?
3. What is the Speed Layer?
4. Typical current state architecture
5. Target state architecture
6. How does data flow through the Speed Layer?
7. How we consume data from the Speed Layer
8. How the Speed Layer is deployed
9. Progress
10. Streaming assessment
11. Value achieved
12. Demo

3. 3
Nationwide Building Society
• Formed in 1884 and renamed to
become the Nationwide Building
Society in 1970
• We’re the largest building society
in the world.
• A major provider of mortgages,
loans, savings and current
accounts in the UK and launched
the first (or 2nd) Internet Banking
Service in 1997
• We recently announced an
investment of an additional £1.4
billion (total £4.1bn) over 5 years
to simplify, digitise and transform
our IT estate.
• Confluent and Kafka form the
heart of an important part of that
investment.

4. 4
• Regulation such as Open Banking
• Business growth
• 24 x 7 availability expectations from customers
and regulators
• Cloud adoption
• Capitalising on our data
• A need for agility and innovation
… and our existing platforms were making this
harder than we’d like.

5. The Speed Layer will be the preferred source of data for high-volume read-only data requests and event sourcing. It will
deliver secure, near real data from back end systems with speed and resilience. It will use the latest technologies built for cloud
and designed to be highly available. It will provide NBS with the first event based real-time data platform ready for digital.DEFINITION
FOUR KEY CHARACTERISTICS
SCALABILITY:
The Speed Layer platform will be
built using internet scale
technologies hosted on cloud ready
PaaS architecture.
FAST AND AGILE:
The Speed Layer will unlock data
from our Systems of Record
enabling digital and agile
development teams to rapidly
deliver new features and services
RICH DATA SET:
Provide a rich data set enhanced
with 3rd party data and analytics
RESILIENT:
Reduce the load on core systems
and isolate them from the demands
of the digital platforms. Designed to
be highly resilient and tolerate
multiple infrastructure failures
5
What is the Speed Layer?

6. 6
As is logical E2E Architecture
API Gateway
Channel Web Services
Enterprise Web Services
Back-end Services
Mainframes
Fairly normal, is there a problem?

7. 7
API Gateway
Stream Processing
Mainframes + other sources of data
Kafka Topics
Target System Architecture
CDC
Kafka Topics
Microservices Channel Services
Enterprise Services
Protocol adapters
WritesReads

8. System of Record(s)
CDC
Replication
Engine
Source
DB
Kafka Raw Topic – raw data
Stream
processingA
Microservice
Kafka Published Topic – processed data
Materialisation Microservice
NoSQL tables
{REST APIs}
1. Change Data Capture (CDC) is deployed to the System of Record (SoR) and
pushes changes from source database to Kafka Topic
2. Kafka topics contain data in the format of the source system. There will be one
raw topic per table replicated. Data is typically held here for c7 days.
3. Streams processing (Kafka Streams framework) is used to transform data into
processed data made available to consumers through “Published Topics”
4. Kafka Published Topics retain data long term (in line with retention policies
and GDPR) and can be used by many Speed Layer Microservices.
5. Speed Layer Microservices are consumers of Kafka Published Topics and push
the data they need into their persistence store (NoSQL, in-memory, etc.)
6. APIs expose data to consumers
7. Channel applications call Speed Layer Microservices to request data
8. Note, applications can subscribe to events and respond to events without
materialising them in a database, e.g., push notification to device.
124
5
3
6
Consuming Applications
7 8
Data Flow Diagram

9. 9
There are three main approaches for consumption of data from Speed Layer. 1. Immediate real time message consumption in the Event Driven pattern, 2. Usage specific
data sets are materialised and exposed through APIs in the Request Driven pattern. 3. Functionally aligned enterprise level data stores are materialised.
Enterprise/
Functional
microservices
Consumption
Ms & apps
Consumers are microservices that subscribe to
topics and materialise data to their requirements.
A set of functional microservices are created. For
example an “account” microservice from which all
consuming microservices and applications read
account data when needed.
Kafka consumers listen and respond to
messages that are arriving in near real time and
take immediate action on receipt of the message.
In this pattern there is no need to materialise the
data.
SL
subs
Producer Producer Producer Producer
SL
subs
Producer Producer Producer Producer
SL
Producer Producer Producer Producer
FUNCTIONAL SERVICEEVENT DRIVEN REQUEST DRIVEN
Applications and/or services can be re-written to consume data from the Speed Layer to improve performance and reduce demand on back-end systems.
Consumption Patterns Overview

10. 10
Multi-site deployment and resilience
Primary DC for SORs Standby DC for SORs Cloud hosting Deployment
1. CDC writes to a local Kafka Cluster, i.e., in the same
DC as the mainframe
2. Kafka topics are replicated to a separate Kafka cluster
in our 2nd
DC
3. Independent database clusters in each datacentre.
4. When required, Kafka topics are replicated using
Confluent Replicator to cloud providers
DB

11. Progress so far…
• Architectural PoC completed:
1. Initial logical proving
2. Functional and non functional proving
3. Load testing/benchmarking in Azure and IBM labs, > 80k TPS through a single broker
• Project launched to deliver the production capability and first use cases
1. Split into 3 use cases, with the first one code complete, 2 & 3 progressing well
• Adopting Confluent Kafka across multiple lines of business
1. Speed Layer
2. Event Based designs for originations journeys
3. High volume messaging in Payments
• Working on Streaming Maturity Assessment with Confluent

12. Adopting an Enterprise Event-Streaming Platform is a Journey
Nationwide nearly here - with Speed Layer + platforms
for Mortgages & Payments - but more potential to
share common ways of working and utilise a common
platform for more use cases
VALUE
1
Early interest
2
Identify a project /
start to set up
pipeline
3
Mission-critical, but
disparate LOBs
4
Mission-critical,
connected LOBs
5
Central Nervous
System
Projects Platform
Developer downloads
Kafka & experiments,
Pilot(s).
LOB(s); Small teams
experimenting;
→ 1-3 basic pipeline use
cases - moved into
Production - but fragmented.
Multiple mission critical use
cases in production with
scale, DR & SLAs.
→ Streaming clearly
delivering business value,
with C-suite visibility but
fragmented across LOBs.
Streaming Platform managing
majority of mission critical
data processes, globally, with
multi-datacenter replication
across on-prem and hybrid
clouds.
All data in the
organization managed
through a single
Streaming Platform.
Typically → Digital
natives / digital pure
players - probably using
Machine Learning & AI.

13. Expected value (this time next year)
Ø Enables agility and autonomy in digital development teams
Ø The first use case alone will remove c7bn requests / year from the mainframes.
Ø Will help us maintain our service availability despite unprecedented demand
Ø Kafka and streaming being adopted across multiple lines of business
Ø The move to micro services and Kafka enables Nationwide to onboard new use cases
quickly and easily
Ø Speed Layer, Streaming and Kafka will help Nationwide head-off the threat from agile
challenger banks
The Speed Layer will help Nationwide provide customers with a better customer experience leading to
better customer retention and new revenue streams.

14. 14
Demo of Speed Layer
Ø Why we did the Proof of Concept
Ø Functional walk through
Ø Non-functional view