Businesses are building digital platforms with modern architecture principles like domain driven design,  microservice based, and event-driven. These platforms are getting ever so modular, flexible and complex. While they are built with architecture principles like - loose coupling, individually scaling, plug-and-play components; regulations and security considerations on data - complexity leads to many unknown and grey areas in the entire architecture. Details on how the different components of this complex architecture interact with each other are lost. Generating insights becomes multi-teams, multi-staged activity and hence multi-days activity.  Multiple users and stakeholders of the platform want different and timely insights to take both corrective and preventive actions.Business teams want to know how business is doing in every corner of the country near real time at a zipcode granularity. Tech teams want to correlate flow changes with system health including that of downstream stability as it happens.Knowing these details also helps in providing the feedback to the platform itself, to make it more efficient and also to the underlying business process. In this talk we intend to share how we made all the business and technical insights of a complicated platform available in realtime with limited incremental effort and constant validation of the ideas and slices with business teams. Since the client was a Banking client, we will also touch base handling of financial data in a secure way and still enabling insights for a large group of stakeholders. We kept the self-service aspect at the center of our solution - to accommodate increasing components in the source platform, evolving requirements, even to support new platforms altogether. Configurability and Scalability were key here, it was important that all the data that was collected from the source platform was discoverable and presentable. This also led to evolving the solution in lines of domain data products, where the data is generated and consumed by those who understand it the best.

1. © 2021 Thoughtworks | Confidential
Real time insights for better
products, customer experience
and resilient platform
Balvinder Khurana and Sushant Joshi

2. Who we are
2
Balvinder Khurana
Data Architect and
Global data community lead
Sushant Joshi
Product Principal
@sushantjoshi
https://sushant-joshi.medium.com
Balvinder has 14+ years of experience in building large-
scale custom software and big data platform solutions for
complicated client problems. She has extensive experience
in Analysis, Design, Architecture,
and Development of Web based Enterprise systems and
Analytical systems using Agile practices like Scrum and XP.
Balvinder currently works as a Data Architect and Global
Data Community Lead for Thoughtworks
Sushant is a Product Principal at ThoughtWorks. His work
includes working with clients to assess product-market-fit,
create goal aligned roadmaps and product delivery. He
brings in his ever-curious mindset, business knowledge,
and interdisciplinary thinking to solve problems that form
our surroundings. His primary focus area is - product
discovery - through which he helps address key product
risks in the early stages of the product
Sushant is passionate about Indian digital ecosystems. He
is working with Indian companies to create better products.

3. Start your day with the your business dashboard!
3

4. Start your day with the your business dashboard!
4

5. Problem Landscape
5
Retail Bank
KYC - Compliance
to Customer Service
The Ask

6. Few examples:
How much amount was disbursed yesterday
in Mumbai?
How many car loans were sanctioned this
week?
What’s the health of APIs and underlying
systems for last 3 hours?
What are we
trying to achieve
6
Actionable Insights through Real Time, Self
Serviced Information of everything that’s
happening on the platform
What are the reasons
for drop offs?
Is it system or user?
Are my APIs
overloaded?
Average time for
disbursement
Opportunity loss in the
Funnel at different
stages
Login
Offers
Risk
Checks
Sanction
2FA
Disbursement
Which offers are
attractive?
How many customers
are moving further after
viewing offers
How long it takes to
send the OTP and
customer action?
Do users need alternate
mechanism?

7. Explosion of personas and explosion of requirements
7
Revenue Generated
Price Sensitivity
Number of Users
Customer micro-
Segment
Business
Deployment status
Load on a service
Downtime for Service
Developers
Service Availability
Service Traceability
Routing
Security team
Insights
What can I
understand?
Data Scientists
Customer 360
Customer Propensity
Customer-product fit
Customer facing
executives
7

8. 8
Isolated Solutions
Web
application
Mobile
application
User click
stream
Social media
Market data
System level
metrics
Customer
support
Competitor
data
Logs
Real time monitoring of all
infrastructure components and
service issues monitored using
Prometheus and charts created
on Grafana. Many other tools
like EFK stack, kafka etc. are
used.
Developers/
IT Support
Understand
system health
and avoid failures
Periodic data is provided
(sometimes manually) after
pulling out of tools like Kafka,
GTM in form of excel.
Product
Owners
No drop-outs,
all journeys
should be
completed
Business/
C-level execs
Data pulled out on-demand
(manually) and shared via
email/excel sheets.
How is my business
performing, where
are the leakages

9. 9
Business
● Siloed
○ Systems
○ Tools
● Different
○ Targets
○ Maturity
○ Objectives
Data
● Siloed
○ Data (Storage)
○ Business Units
● Different
○ Granularity
○ Formats
○ Architectures
○ Exploration Scopes
● Different
○ Tech Stack
○ Architecture
○ Tools
Technology
Limitations and pain points

10. 10
● Consolidation
● Coordination
● Manual synthesis
● Low confidence
Piecemeal Solutions

11. Size, Scale and Complexity
11
Web & Apps
Tele channels
Physical stores
Digital Ecosystem
partners
Kiosks, agents etc
Sales &
marketing
operations
Products
Technology
Risk &
compliance
Strategy
Enterprise
functions
Social media
Retail Loans
Retail
Deposits
...
Regulators
Investors
Aggregators
Partners
Product
based
Customer
segment
based
Channel
Corporate
Customer
risk
Fraud
Underwriting
Liquidity
HR
Employee
Facilities
Legal
How customers interact with
the organisation
Constraining and driving
forces

12. Bringing it all
together - The
“Superdata”
Solution
12

13. 13
13
What it is really? Make it quick and easy to explore a
hypothesis (business or technical),
accept or disprove it, and move on
to find the root cause.
Platform which enables people to
use their skills, extend their senses,
support their intuitions.
The Superdata solution a.k.a
Command center

14. Persona and business domains,
wants/requirements
Principles of Super-data
Traditionally data is looked from the weekly or
for monthly leadership review.
So someone requests and presents what needs
to be presented
From requests to predefined datasets
serving insights & exploration
14
Concept Validation
Product mindset
Design
#Data discoverability #Data
presentability
Holistic (Business + Tech)
Port the same to other domains

15. 15
Data Platform
Cloud DW
Data Lake
(Cloud Object
Storage)
arts
Data
Marts
ODS
Tech Solution
Dashboards
Data Service
API’s
Reports
Data Ingestion &
Integration
Batch
Ingestion
Unstructured
Source
Ingestion
API Ingestion
Streaming
Ingestion
Orchestration Service
Data Integration & Ml
ELT
Stream
Processing
ML
Toolkits*
Deep
Learning*
ETL
15
Source
Systems
Bank
Applications
Data
Tele Channels
Data
Physical
Store Data
Social Media
Data
Partners
data
15
15
DevOps/DataOps
Data Governance
Data Catalog Data Quality Security
Business Events
*Future Scope

16. Dashboards
Data Service
API’s
Reports
16
Tech Stack
Data Platform
Cloud DW
Data Lake
(Cloud Object
Storage)
Data
Marts
ODS
Data Ingestion &
Integration
Batch
Ingestion
Unstructured
Source
Ingestion
API Ingestion
Streaming
Ingestion
Orchestration Service
Data Integration & Ml
ELT
Stream
Processing
ML
Toolkits
Deep
Learning
ETL
Source
Systems
Bank
Applications
Data
Tele Channels
Data
Physical
Store Data
Social Media
Data
Partners
data
DevOps/DataOps
Data Governance
Data Catalog Data Quality Security
Business Events

17. 17
17
Data Platform
Cloud DW
Data Lake
(Cloud Object Storage) arts
Data Marts
ODS
Dashbo
ards
Data
Service
API’s
Report
s
Data Ingestion &
Integration
Batch Ingestion
Unstructured Source
Ingestion
API Ingestion
Streaming Ingestion
Orchestration Service
Data Integration & Ml
ELT
Stream
Processi
ng
ML Toolkits
Deep
Learning
ETL
Source
Systems
Bank Applications
Data
Tele Channels Data
Physical
Store Data
Social Media Data
Partners
data
DevOps/DataOps
Data Governance
Data Catalog Data Quality Security
Business Events
Serve Data as a Product
Auto Loan
Customer
Personal
Loan
Credit
Card
Social
Media
Customer
Profile
Domain driven
data boundaries
The boundaries cut across
the platform - from source
to consumption!

18. Data Product
Data Platform Architecture
Quantum
Fundamental unit of architecture
Self-Serve
Data
Product
Domain
Polyglot Data
Output Ports
Polyglot Data
Input Ports
Control
Ports Stats
Logs, metrics
Self
discovery
Management
18
Discoverable
Addressable
Self-describing
Trustworthy
Defined and Monitored silos
Secure
enforce globally configured access
control at each data product output port
Interoperable
governed by global open standard

19. Impact and learnings
19

20. 20
Our Learning
● Data-as-a-first-class-citizen
● Staying source agnostic
● Timely scaling considerations

21. ● Data driven product planning
● Resilient platform
● Data democratization
● Standardization of source platform
Impact
21
21
© 2021 Thoughtworks
“Can we ask for this data” to “Can we pull
up this data”
“we are not able to track numbers, since
last couple of hours dashboard is
showing the same numbers”

22. 22
What’s next
● Increased self-service
● Anomaly detection and alerting
● Adaptive journey completions
● Personalization and Nudges
● Domain driven boundaries for data

23. Thank you
Sushant Joshi
Twitter : @sushantjoshi
https://sushant-joshi.medium.com
Balvinder Khurana
23