Rise of Data Streaming in Retail Industry

The Rise of Data in Motion in the Retail Industry
Use Cases, Architectures and Real-World Examples powered by Data Streaming with Apache Kafka
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
confluent.io
kai-waehner.de
@KaiWaehner

Live Commerce in the Retail Industry with Data Streaming – @KaiWaehner - www.kai-waehner.de
The world is changing.

Stores become Software

Payments become Software

Delivery become Software

Live Commerce with Social Interaction

The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data

This transformation is
happening everywhere

Retail

Software is Eating Retail
10
https://www.cbinsights.com/reports/CB-Insights_Retail-Tech-100-Webinar.pdf

What enables this
transformation?

Real-time Data beats Slow Data.
Logistics
Real-time sensor
diagnostics
Delivery planning
ETA updates
Payment
Fraud detection
Risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Sales
Real-time
recommendations
Personalized
coupon feed
Pay by walking out

This is a fundamental paradigm shift...
13
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Data
Streaming

What is Data Streaming?

‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Payment
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

Data in Motion in the Retail Industry
Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example

17
Increase Revenue
→ Customer Experience, Loyalty
Decrease Costs
→ Increase Operational Efficiency
Mitigate Risks
→ Regulatory Compliance
Reporting & Analytics
Stock inventory management
Legacy IT Replacement
(e.g. Middleware replacement)
Cyber Security
(incl. SIEM)
Fraud Prevention
(Anomalous credit card transactions)
Legacy IT Modernization
(e.g. Mainframe off-load / augmentation)
Example Retail Solutions for Event Streaming
Buyer journey - personalized offers
(Contextualized Next Best Action /
Targeted Offers)
Real-time Inventory Management
(across web, store, …)
Customer 360
(Call Center - Know Your Customer - KYC)
RT mobile applications
(clothing, food and beverage ordering, …)
Migration to the Cloud
(Hybrid on-prem / Cloud. Also Hybrid
Public Cloud vendors)
Microservices Architecture
Data Infrastructure layer
Business Application layer - the use cases
Data Pipelines Messaging
Microservice/
Event Sourcing Stream Processing Data Integration Streaming ETL Log Aggregation

Apache Kafka is the de facto standard for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
18

Kafka is a cloud-native streaming middleware!
à More than just data ingestion or message queue
kai-waehner.de | @KaiWaehner | Building a Cloud-native Data Warehouse with Data Streaming
DWH
APP
STREAM
PROCESSING
CONNECTORS
ksqlDB
KStreams
APP
Streaming ETL
Data Processing
Real-time Analytics
Stateless and Stateful
Business Applications
Fully-managed
Pipelines
Connectivity to
Data Infrastructure,
SaaS, AI/ML
Data Governance
Filtering and Routing
Change Data Capture
Built-in Scale and Fault Tolerance
Oracle
DB
ORACLE CDC
SOURCE
PREMIUM
CONNECTOR
Real-time Data Sharing
across DCs and Multi-Cloud

Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
250,000 500 25,000 25
100 5 300 3
Kafka Scales with Your Business.

Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?

Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
23
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration

Real-Time Data as Foundation
for Live Commerce

Customer 360 and Real-Time Context-Driven Events
http://hyper-reality.co/
• CRM integration
• Real time product inventory
• Predictive recommendations
• Geo-location
• Payment integration
• …

Albertsons – Real-time Data Hub
26
TODO LINK to public Confluent webinar
Second largest American grocery company with 2200+ stores and 290,000+ employees

27

28

Nordstrom Analytical Platform (NAP)
29
American luxury department store chain
50+% of revenue online
A singular event can be used for
analytical, functional, operational, and
model building purposes
https://www.confluent.io/events/kafka-summit-americas-2021/nordstroms-event-sourced-architecture-and-kafka-as-a-service/

Context-specific Customer 360
30
Electrical retailer
Hyper-personalized online retail experience,
turning each customer visit into a one-on-one
marketing opportunity
Correlation of historical customer data with
real-time digital signals
Maximize customer satisfaction and revenue
growth, increased customer conversions
https://www.confluent.io/customers/ao/

Live Commerce with
Social Interactions

Live commerce with real-time data correlation
Online and offline shopping via social commerce
Including integration of CRM, loyalty, inventory, chatbots, location-based services, augmented reality, etc.
32

Live Commerce with Data in Motion
Your Business as Streams of Events, powered by Kafka
Influencer
Show
Payment
Shipping
Sales
Platform
Social
Platform

Live Commerce with Data in Motion
Interact with the customer via social commerce.
Recommend products that need to be sold.
Provide context-specific pricing.
All automated. In real-time. At scale.
Data Consolidation
Streaming ETL
(Kafka Connect + Kafka Streams)
Mobile App
(Swift on iOS)
(4) Receive
product recommendation
and context-specific price
Order Fulfillment &
Payment Service
Buy Now Pay Later (BNPL)
(C++)
Chatbot
External API
(REST API)
(1) Click on
shoes
(5) Order
product
(2) Q&A with chatbot
(6) Fulfill order
CRM / Loyalty
Salesforce
(Kafka Connect)
Source: channelize.io
Inventory
Live monitoring
(ksqlDB)
(3) Context-specific
pricing engine

Challenge: Build a conversational chatbot service that
incorporates complex technologies such as fulfillment,
natural-language understanding, and real-time analytics
Solution: Use Confluent to build a fast, super-scalable event-
driven architecture that could handle immense traffic spikes
and also provide other guarantees around delivery semantics
Results:
● Near-zero downtime even during huge traffic spikes
● Rapid acceleration of new-skill onboarding
● Doubling of NPS rating
Virtual Agent Platform:
(Marc Silbey, VP of Product at Expedia)

Augmented reality for enhanced online shopping
Scan, inform, buy. Whatever you see. Wherever you see it.
Streaming ETL
(Kafka Connect + Kafka Streams)
Mobile App
(Swift on iOS)
(3) Receive
enhanced
picture
Order Service
Buy product from favorite
vendor
(ksqlDB)
Picture
Enhancement
Show digital version with
product details
(external SaaS)
(1) Send
picture
(4) Order
product
(2) Enhance
picture
(5) Process
Order

Augmented reality for enhanced online shopping
Live demo (Kudos to Carsten Muetzlitz who has built the demo!)
https://www.kai-waehner.de/blog/2021/02/23/apache-kafka-augmented-reality-ar-vr-retail-demo-arkit-unity-unreal-engine/

Stitch Fix
Online personal styling service
Recommendation algorithms to personalize clothing items
based on size, budget and style
Data Scientists consume data from Kafka for transformation
of client data into features, model predictions, allocation of
A/B tests, etc.
38

Connect
REST API
Connect
logs-customer-nft-trade
suspicious-transaction
fraud-alert
Streams
Data
Correlation
Fraud
Detection
Crypto Payment and NFT Integration
Python
Fraud
Alerting
Streams

Architectures for
Apache Kafka in Retail

Walmart Cloud Native Platform (WCNP)
41

Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
for RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
42

Challenge:
- Needed to share data between web applications in
Google Cloud with Snowflake and other analytics tools
in Azure in real time
- Needed a single-region, multi-cloud disaster recovery
strategy
Solution:
Built a multi-cloud data architecture using:
- Confluent Cloud in GCP
- Confluent Cloud in Azure
- Fully managed connectors
- Cluster Linking
- ksqlDB (pre-production)
Results: In three months went from having no DR
strategy to a production multi-cloud DR capability based
on a real-time data architecture that also supported high
performance regional applications.
flowers.com
Multi-Cloud Disaster Recovery

‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
44
https://medium.com/porschedev

Omnichannel Retail
Time
P
C3 C2
C1
Personalized Sales
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Online Store
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

AWS Cloud Outage hit Disney World Visitors…
https://www.cnet.com/tech/services-and-software/disney-parks-were-already-facing-heat-from-fans-then-an-aws-outage-came-along/

CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Customer
data
Train
schedule
Payment
data
Loyalty
information
Hybrid Retail Architecture

Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Global Inventory
Management
Event Streaming at the Edge
in the Smart Retail Store
Item Availability

Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)

Ship-Shore Highway – Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

Event Streaming Is The Future Of Data
51
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Data
Streaming

Why Confluent?

The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
53

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
54

Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.

Kai Waehner
Field CTO
kai.waehner@confluent.io
confluent.io
kai-waehner.de
@KaiWaehner
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Rise of Data Streaming in Retail Industry

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Rise of Data Streaming in Retail Industry

Ähnlich wie Rise of Data Streaming in Retail Industry (20)

Mehr von Kai Wähner

Mehr von Kai Wähner (13)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Rise of Data Streaming in Retail Industry