21st Century Analytics with Zopa

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
01.10.2018
Migrating to 21st Century analytics
Shafreen Sayyed Varun Gangoor
AWS Solutions Architect Zopa Sr. Data Engineer

Agenda
• Trends and use cases
• Data Storage
• Data collection and automated data Ingestion
• Data processing and analysis
• Data consumption and visualisation
• Security and compliance
• Zopa’s Story
• Summary

Four trends are shaping Financial Services in EMEA today
Digital innovation
creating opportunities
to grow and
increasing
competitive pressure
Increasing regulatory
burden and ever-
increasing compliance
obligations
Resource constraints
on the road to
regaining financial
health
Laser focus on
operational risk

Typical First Use Cases for Financial Services
Risk Calculations
using Grid Computing
Instead of 10 servers
running 6 hours, why
not have 60 servers run
for an hour
Digital channels
Automating customer
contact
Alexa / Echo
Internet banking
Data streaming
Device Farm
Greenfield digital banks
Software Development
Dramatically accelerate
the development cycle.
Scale up and down
resources
Big Data and AI/ML
Storing and analyzing
large data sets in the
cloud, using data lakes,
state of the art analytics,
AI and machine learning
technology
Backup, Archive,
Disaster Recovery
Durable, redundant,
encrypted, lockable.
Worm compliance records
management
Spin up redundant data
centers quickly
Open Banking and
FinTech integration
Build and scale secure
APIs quickly to enable an
ecosystem of FinTech
technology

Where do I start?
Ingest /
Collect
Consume/
visualize
Store Process/
analyze
Data
1 4
0 9
5
Answers and
insights

Where do I start?
Ingest /
Collect
Consume/
visualize
Store Process/
analyze
Data
1 4
0 9
5
Answers and
insights
Start here
(with a
business case)

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage is Job #1

Object Storage is Foundational
Amazon
S3
Amazon RDSAmazon
DynamoDB
EBSAmazon
EFS

Data collection and automated
data Ingestion

Data collection
• Dedicated 1 Gbps and 10
Gbps fibre link to AWS
• Low cost, with consistent
low latency/jitter
• Direct access to AWS
services and your VPCs
• Tamper-resistant case and
electronics
• Ruggedized case that can
withstand 8.5 G
• Available in 50 TB or 80 TB
capacities
AWS Snowball
AWS Database
Migration Service
• Modernise, migrate, or
replicate your RDBMS
• Fan-in multiple sources to
single target
• Platform and schema
conversion
AWS Direct Connect

Database Migration
Service
Amazon
Kinesis
Amazon
S3 Acceleration
Amazon
S3 Upload Snowball
Snowball Edge
Snowmobile
Automated Data Ingestion

Data processing and analysis

Data ingestion
Amazon
Kinesis
AWS IoT
• Fully-managed real-time stream
processing
• Highly available across multiple
AZs
• Can capture and store:
• Terabytes of data per hour
• From hundreds of thousands
sources
• Collect data from your connected
devices
• Communicate securely back to your
devices
• Can easily support:
• Billions of devices
• Trillions of messages
“If you knew the state of every thing in the world, and could
reason on top of that data, what problems could you solve?”

SELECT STREAM author,
count(author) OVER ONE_MINUTE
FROM Tweets
WINDOW ONE_MINUTE AS
(PARTITION BY author
RANGE INTERVAL '1' MINUTE PRECEDING)
WHERE text LIKE ‘%#FSInsight%';
Amazon Kinesis Analytics – Simple SQL Interface

Scalable (secure, versioned, durable) storage +
Immutable data at every stage of its lifecycle +
Versioned schema and metadata
=
Data discovery, lineage
Storage + Catalog

AWS Glue
• Data Catalog
Discover and store metadata
• Job Execution
Serverless scheduling and execution
• Job Authoring
Auto-generated ETL code

Hive metastore-compatible, highly-
available metadata repository:
• Classification for identifying and
parsing files
• Versioning of table metadata as
schemas evolve
• Table definitions – usable by
Redshift, Athena, Glue, EMR
Populate using Hive DDL, bulk import,
or automatically through crawlers.
Glue Data Catalog

semi-structured
per-file schema
semi-structured
unified schema
identify file type
and parse files
enumerate
S3 objects
file 1
file 2
file N
…
int
array
intchar
struct
char int
array
struct
char
int
int
arrayint
char
char int
custom classifiers
app log parser
metrics parser
…
system classifiers
JSON parser
CSV parser
Apache log parser
…
bool
Crawlers: Automatic Schema Inference
bool

AWS Lambda
Metadata Index
(Amazon DynamoDB)
Search Index
(Amazon Elasticsearch)
ObjectCreated
ObjectDeleted PutItem
Update Stream
Update Index
Extract Search Fields
Indexing and Searching Using Metadata
Amazon
S3

Data processing and analysis
• Petabyte scale data
warehouse
• Fault-tolerant scalable
cluster with node auto-
recovery
• Auto backup into Amazon
S3
Amazon Redshift
Structured
data processing
• Fully-managed big data
platform
• Auto-scaling clusters
• Supports Hadoop:
Hive, Spark, Presto
Zeppelin, HBase, Flink
HDFS and Amazon S3
filesystems
Amazon EMR
Semi/unstructured
data processing
• No infrastructure to manage
• No data loading required
• Supports multiple data formats:
• CSV, TSV, Avro, ORC, Parquet
• Uses ANSI SQL to directly query
Amazon S3
Amazon Athena
Serverless
query processing

Consume and visualise data
• No infrastructure to
manage
• Event-driven processing
• Pay per 100 ms CPU
• Node.js, Python, Java and
C# (.NET Core)
AWS
Lambda
• Multiple classifier types
• Interactive UI for modelling and
dataset visualisation
Amazon
Machine Learning
• Fast, cloud-powered BI tool
• Scales to hundreds of thousands of
users
• Quick calculations with SPICE
Amazon
QuickSight

• Python code
generated by AWS
Glue
• Connect a notebook
or IDE to AWS Glue
• Existing code brought
into AWS Glue
Managed ETL with AWS Glue

• Schedule-based
• Event-based
• On demand
Job Execution with AWS Glue

Amazon Athena – Analyze Data in S3
• Interactive queries
• ANSI SQL
• No infrastructure or administration
• Zero spin up time
• Query data in its raw format
• AVRO, Text, CSV, JSON, weblogs, AWS service logs
• Convert to an optimized form like ORC or Parquet for the best
performance and lowest cost
• No loading of data, no ETL required
• Stream data from directly from Amazon S3, take advantage of Amazon
S3 durability and availability

Simple query editor
with syntax highlighting
and autocomplete
Data Catalog
Query History, Saved Queries, and
Catalog Management

Data consumption and
visualisation

QuickSight allows you to connect to data from a wide variety of AWS, third-party, and on-premises
sources including Amazon Athena
Amazon RDS
Amazon S3
Amazon Redshift
Amazon Athena
Using Amazon Athena with Amazon QuickSight

Security and compliance

Ingest ServingData
sources
Modern data architecture
Insights to enhance business applications, new digital services
Transactions
Web logs /
cookies
ERP
Data analysts
Data scientists
Business users
Engagement platformsConnected
devices
Automation / events
DATA PIPELINES
EVENT PIPELINES
Data
Event Action
Insights
Data
Lake
ML / Analytics
Predict /
Recommend
AI Services
Social media

Speed (Real-time)
Ingest ServingData
sources
Scale (Batch)
Modern data architecture
Insights to enhance business applications, new digital services
Data analysts
Data scientists
Business users
Engagement platforms
Schemaless
Amazon ElasticSearch
Direct Query
Amazon Athena
Near-Zero Latency
Amazon DynamoDB
Automation / events
Amazon S3
Staged Data
(Data Lake)
Semi/Unstructured
Amazon EMR
Transactions
Web logs /
cookies
Connected
devices
Access securely granted - trust and compliance
1. Restricted access roles set up within account
2. Highly governed access to service endpoints
3. Remote access to virtual desktops
AWS
Cloud Trail
AWS
IAM
Amazon
CloudWatch
AWS
KMS
ERP
Enterprise Apps
Oracle, Teradata,
SQL Server, MySQL, et
Data Warehouse
Amazon Redshift
Social media

Migrating to 21st Century analytics
Varun Gangoor
Senior Data Engineer
email: varun.gangoor@zopa.com

Who are we

A pioneering financial services company
World’s 1st
peer-to-peer
lending platform
in 2004
£3.5 billion
lent to date,
and our growth is
accelerating
390,000
people have taken a
Zopa loan
75,000
actively invest through
Zopa

What we do

Simple loans.
Smart investments.
BorrowersInvestors
Invests Repayments
Interest + capital Loans

Analytics @Zopa

Strictly Private & Confidential 47
Business Analytics
Centralised
Data
Warehouse
Business
Users
Great
Customer
Experience

Data Scientists
* Statistical model and
monitoring
* Raw data
* Cold data
Data Analysts
* Data quality and
analysis
* Raw and
aggregated data
* Cold and warm data
Product Analysts
* Product feature
analysis
* Raw and
aggregated data
* Hot and warm data
Leadership Team
* Custom dashboards
* Aggregated data
Data Consumers

Data Existence
Same data exists in
multiple location
Reporting
Running reports from DWH
instead of operational
systems
Data Quality
Knowledge base about our
systems and our data
Time
Reporting queries against
the historical data
Analytical Data Warehouse Requirements
Historical Data
Need to gather in one
accessible location
Integration
Variety of sources and data
formats
Data Growth
From gigabytes to
petabytes
Varying Data
Rapid changes in
application features
and data sources

Bigdata @Zopa

V's @Zopa
Validity
• Data Quality
• Data Governance
Variablility
• Change in Data
• Change in Model
Vocabulary
• Data Model
• Data Dictionary
Value
• Useful Data
Volume
• Terabytes
• Tables and Files
Variety
• Structured
• Semi-structured
Velocity
• Batch
• Realtime
Veracity
• Trustworthyness
• Availability

Tech Stack
Collect
Kinesis Firehose
Glue
Process/Analyse
Lambda
Batch
Glue
Redshift
Redshift Spectrum
Athena
Store
S3
Glacier
Consume
Sagemaker
Tableau
Python
SQL

Zopa's Enterprise Data Warehouse

Thank you
We are hiring !

Summary

Central Storage
Secure, Cost Effective
Storage in S3
S3
Access & User Interface
Give your users easy & secure access
API Gateway IAM Cognito
Protect & Secure
Use entitlements to ensure data is secure and users identities are verified
Security Token
Service
Cloudwatch Cloudtrail KMS
Athena Quicksight EMR Redshift
Processing & Analytics
Use predictive and prescriptive
analytics to gain better understanding
Firehose Direct Connect Snowball DMS
Data Ingestion
Get your data into S3
quickly and securely
Catalog & Search
Capture, Access, and Search Metadata
DynamoDB Amazon ESGlue Macie

Again, where do I start?
Ingest /
Collect
Consume/
visualize
Store Process/
analyze
Data
1 4
0 9
5 Answers and
insights
Seriously, start here
(with a business case)
Then collect your data

Build decoupled systems
• Use Amazon S3 as the data fabric of your data lake
• Data → Store → Process → Store → Analyze → Answers
Use the right tool for the job
• Data structure, latency, throughput, access patterns
Leverage AWS managed services
• Scalable/elastic, available, reliable, secure, no/low admin
Use log-centric design patterns
• Immutable log, batch, interactive & real-time views
Be cost-conscious
• Big data ≠ big cost

AWS Solution Builder - Data Lake on AWS
Reference Architecture deployment via
CloudFormation
Configures core services to tag, search
and catalogue datasets
Deploys a console to search and
browse available datasets
http://amzn.to/2nTVjcp

Thank you

21st Century Analytics with Zopa

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie 21st Century Analytics with Zopa

Ähnlich wie 21st Century Analytics with Zopa (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

21st Century Analytics with Zopa