Apache Kafka and Kubernetes in the Telco Industry - Cloud-Native 5G, MEC and OSS/BSS/OTT Real-World Use Cases.
This session explores architectures and use cases for event streaming with the open-source framework Apache Kafka in the Telco industry. Telcos modernize their edge and hybrid cloud infrastructure with Kafka and Kubernetes to provide an elastic scalable real-time infrastructure for high volumes of data. Demonstrated use cases include building 5G networks, proactive OSS network monitoring, integration with hybrid and multi-cloud BSS and OTT services.
Video recording of the presentation:
https://youtu.be/yoRUC0sFbJo
More details about the event streaming and cloud-native architectures and use cases in the telecom sector:
https://www.kai-waehner.de/blog/2020/03/06/event-streaming-apache-kafka-telecommunications-industry-telco-business/
https://www.kai-waehner.de/blog/2020/07/03/telco-ott-applications-with-apache-kafka-telecom-sector-oss-bss-hybrid-cloud/
https://www.kai-waehner.de/blog/2021/05/23/apache-kafka-cloud-native-telco-infrastructure-low-latency-data-streaming-5g-aws-wavelength/
https://www.kai-waehner.de/blog/2020/10/14/use-cases-architectures-apache-kafka-edge-computing-industrial-iot-retail-store-cell-tower-train-factory/
https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
Cloud-Native 5G, MEC and OSS/BSS/OTT Telco with Apache Kafka and Kubernetes
1. Apache Kafka and Kubernetes in the Telco Industry
Cloud-Native 5G, MEC and OSS/BSS/OTT Real-World Use Cases
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
3. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
4. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Telco need to transform everything
“The biggest hurdle to
overcome with digital
transformation is to
understand that it isn’t just a
simple technology challenge
– it covers every part of the
telco business.”
https://www.huawei.com/en/about-huawei/publications/winwin-magazine/plus-intelligence/telco-digital-transformation
5. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
5G enables new business models
https://www.youtube.com/watch?v=A9DDe0alvGo
6. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
5G Network Slicing Use Cases
https://www.youtube.com/watch?v=A9DDe0alvGo
7. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
5G Success Criteria
https://medium.com/open-5g-hypercore/open-5g-hypercore-episode-i-166a89805c08
8. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Dish builds a Cloud-Based 5G Network
Greenfield standalone network with pure 5G and cloud native from the start, rather than retrofitting a 4G core
Combination of Dish’ OSS and BSS with AWS Local Zones / Outposts, Kubernetes, ML and 3rd party services
Free from the security limitations of traditional 2G, 3G and 4G technologies.
“As DISH deploys its network, the company is
partnering exclusively with vendors offering
cloud-native technology, bringing them
together on AWS to provide DISH customers
greater flexibility and control of their 5G-
enabled solutions.
As a new carrier, leveraging AWS and its
extensive network of partners enables us to
differentiate ourselves by operating our 5G
network with a high degree of automation.”
https://awsinsider.net/articles/2021/04/26/dish-taps-aws-for-its-5g-network.aspx
9. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
https://www2.deloitte.com/content/dam/Deloitte/it/Documents/technology-media-telecommunications/Next%20Gen%20Telco%20Architecture_2017_final.pdf
10. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
11. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
11
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud-native
Event
Streaming
12. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Data in Motion in the Telco Industry
Your Business as Streams of Events, powered by Kafka
Continuous
Proactive Network
Monitoring
Incident
Management
Reporting
Data Normalization
Middleware
Telemetry
Data
OSS
Infrastructure
Example
13. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time Alerting
System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
13
14. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Truly CLOUD-NATIVE experience
at the edge, in the data center, and in the cloud
Confluent Cloud
A fully managed, cloud-native service for Apache Kafka
Confluent Platform
A complete, enterprise-grade distribution of Apache Kafka
Confluent for
Kubernetes
Ansible
Playbooks
Packages:
Docker, RPMs,
Tarball
Public Cloud Workloads Edge and On-Premise Workloads
On Kubernetes On VMs / Bare Metal
15. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design (DDD) and decoupled Telco services
Legacy Integration
Proactive
Monitoring
Mainframe
Connector
Java / C++ /
Go / ksqlDB
Schema Registry
Event Streaming Platform
BSS Domain Legacy Customer Domain OSS Domain
Audit Logs,
RBAC, etc.
16. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
From purpose-built hardware to generic hardware and elastic scale
From monoliths to decoupled, independent services
https://www2.deloitte.com/content/dam/Deloitte/it/Documents/technology-media-telecommunications/Next%20Gen%20Telco%20Architecture_2017_final.pdf
(Network Function Virtualization)
17. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Event Streaming for
traditional and new innovative use cases
in the telecommunications industry
Real Time Processing Digital Transformation Strategic Goals
Move data from OSS fixed,
mobile, cable into
various systems
Innovative OTT Services
Proactive network
monitoring
CDR
(Call Detail Record)
processing
Fleet management and
connected car solutions
Customer service
(e.g. chat bots) and
social Integration
Cloud-native network
simulation, planning
and deployment
Personalized
messages and
recommendations
Fraud detection
Predictive maintenance
across networks
and infrastructure
Nationwide
security
infrastructure
(SIEM / SOAR)
Augmented
reality
Spanning different
business units
(TV, analytics, video
platform, sales and
services)
OSS-BSS integration
and middleware
modernization
Regulatory
reporting
Customer churn and
cross- / up-selling
Merger after
acquisitions
Omni-channel
customer 360
Converged
billing solution
18. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
19. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Digital service at the edge
Single
Kafka Broker
or Cluster
Digital Service
Self-managed or
certified OEM Hardware
Kafka Cluster
in DC / Cloud
Replicator
SNMP
Monitoring
Syslog
Troubleshooting
TCP
Command & Control
20. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Hybrid Architectures in the Telco Industry
Digital Service
21. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint Edge
Deployments with Replication
(Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
21
22. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Low Latency 5G Use Cases for MEC
with AWS Wavelength (based on AWS Outposts) and Confluent
23. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Retail Architecture
24. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Global Inventory
Management
Event Streaming at the Edge
in the Smart Retail Store
Item Availability
25. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)
26. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
27. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
OSM - Open Source MANO
(NFV Management and Orchestration)
Addressing challenges with orchestration, interoperability and performance optimization
Based on ETSI (European Telecommunication Standards Institute) NFV standards and Information Model
Interoperability among different service provider NFV infrastructures or OSS systems
Apache Kafka asynchronous communication between OSM components and decoupling introduction of new pluggable modules
https://osm.etsi.org/news-events/blog/30-open-source-mano-addressing-interoperability-challenge-in-nfv
28. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Global IP Network @ NTT Com
https://imply.io/post/kappa-architecture-at-ntt
Challenges
• Blackbox
• Not scalable
• Limited ad-hoc analysis capabilities
Real-time visibility into global traffic
• Technical: Capacity planning, traffic engineering, etc.
• Economical: Resource optimization, revenue leak detection, etc.
• Security
Traffic matrix
Representation of all traffic among all possible pairs of sources and destinations in a given
internet domain
29. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
30. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
compax – BSS and CRM
AAX3 - A flexible, microservice-based BSS product
Powered by a cloud-native Kafka backbone
31. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
8x8 – Customer 360
Voice over IP (VoIP) provider
Cloud-based voice, contact center, video, mobile and
unified communications for >1 million business users
• Confluent to build a multi-tenant, scalable, elastic SaaS
contact center solution
• Strict data separation and secure, GDPR compliant data environment to
handle sensitive caller information
• Collection and processing of billions of metrics in real-time
• Replication between data centers for disaster recovery
• Real-time analytics for Dashboards and Machine
Learning
• Key customer experience metrics such as customer wait time and first-call
resolution rates
• Proactive alerts (for example, an issue with a local ISP)
• Prediction of staffing needs
• Predict contact center call volumes for the next 12 hours based on current
and historical data
https://www.confluent.io/customers/8x8/
32. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Omnichannel Retail
Time
P
App1 App2
App3
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago
33. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Omnichannel Retail
Time
P
App1 App2
App3
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions
34. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
35. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Intrado
Tech-enabled real-time communication solutions
Resilient and reliable service for customers
Enterprise collaboration, digital media, healthcare, public safety
Personalized experiences through market intelligence and social sensing
https://www.confluent.io/online-talks/how-intrado-transformed-conferencing/
36. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Hotstar – OTT for millions of cricket fans in India
https://www.confluent.io/kafka-summit-san-francisco-2019/scaling-for-indias-cricket-hungry-population/
https://www.confluent.io/customers/disneyhotstar/
@
37. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
38. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Replace old mainframe with Confluent
(instead of buying a new mainframe)
https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/destination-z1/2019/12/23/capacity-is-king-how-the-ibm-z14-addresses-telecom-volume-concerns
39. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Strangler design pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important
reason to consider a
strangler fig application
over a cut-over rewrite
is reduced risk.”
Martin Fowler
40. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 0: Direct communication between mainframe and app
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Customer Service System
‘1980’ (Mainframe)
41. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for decoupling between mainframe and app
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party Tool
Customer Service System
‘1980’ (Mainframe)
42. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New projects and applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party Tool
Customer Service System
‘1980’ (Mainframe)
43. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Customer Service System
‘2020’ (Modern Technology)
44. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Why Confluent?
45. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
52
46. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
53
47. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.