Abebooks is one of Amazon Subsidiary and it treats data as an asset. It always looks the way to improve existing analytics solution and extract information from terabytes of data. One of the recent initiatives was the migration from legacy DW platform to the AWS Redshift. During this journey, our data engineers met lots of challenges and sometimes tried to reinvent the wheel. This talk will cover Abebooks journey towards Cloud DW. Moreover, we will cover the ETL tool selection process for the Cloud as well as the adoption process for the end users. This talk will help you understand the potential of the modern cloud DW and learn about our use case and save time for the future projects.

1. select outline
from victoria_aws_user_group
where topic_name = ‘Building Modern
Analytics Solutions‘
and solution_type = ’Cloud’
and solution_provider = ‘AWS’
and presenter = ‘Dmitry Anoshin’
And contact_info =
’Dmitry.Anoshin@gmail.com’

2. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
About Myself
Role of Analytics
About Abebooks
About Amazon
Innovation and Data
Abebooks DW modernization
ETL Tool selection for the Cloud
AWS Analytics Solution Prices example
Some Free Learning resources

3. ---------------------------------------------------------------
Disclaimer
---------------------------------------------------------------
The content of this presentation doesn’t represent Abebooks,
Amazon, and AWS. This information is based on my knowledge,
experience and doesn’t have any sensitive or confidential data.
All information and pictures are available online.

4. About Myself
• Work с Business Intelligence c 2007
• Canada since 09/2015
• Pacific Ocean since 05/2016

5. #dimaworkplace

6. Technical Skills Matrix
2015
2010
2007
Data
Warehouse
(Oracle,
Teradata,
Snowflake
,
Redshift)
ETL
(Pentaho
DI,
Informati
ca,
Matillion
ETL, AWS
Glue,
Azure
Data
Factory)
BI
(SAP
BusinessO
bjects,
Tableau,
Microstra
tegy,
Pentaho
BI, SAS
BI)
Bigdata
(Hadoop,
Data Lake)
Digital
Marketing
Data
Analytics
2018

7. Other Activities
Tableau
Cookbook
2019.X
• BI Tech Talk (100+ BI teams globally)
• Amazon Tableau User Group (2000+ users)
• Conferences (EDW 2018, 2019)
• Amazon internal conferences

8. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
Role of Analytics

9. Business Value
Stakeholders Employees Customers
Value
”The goal of any organization is to generate Value”
The Future of Competition.
https://www.amazon.com/Future-Competition-Co-Creating-Unique-Customers/dp/1578519535

10. BI Value Chain
Stakeholders Employees Customers
Value
Decisions
Data
Value creation based on
effective decisions
Effective decisions based on
accurate information

11. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
About Abebooks

12. About Abebooks
• Online marketplace for books, art &
collectibles.
• Amazon subsidiary since 2008 we are
a marketplace for used books and
increasingly non-book-collectibles
• 350 Mlns listings
• 3 in ‘Data Engineering Team’
• 2 locations: Victoria, BC and
Dusseldorf

13. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
A bit about Amazon Culture

14. Amazon Banana Stands

15. Amazon Fly Wheel
http://www.samseely.com/blog/2016/5/2/the-amazon-flywheel-part-1
In 90 Jeff Bezos shared
Amazon business model:
Amazon Fly Wheel =
Price!=>Customers"=>Sellers"

16. Leadership Principles
• Customer Obsession
• Ownership
• Invent and Simplify
• Are Right, A Lot
• Learn and Be Curious
• Hire and Develop the Best
• Insist on the Highest
Standards
• Think Big
• Bias for Action
• Frugality
• Earn Trust
• Dive Deep
• Have Backbone; Disagree
and Commit
• Deliver Results
https://www.amazon.jobs/en/principles

17. Leadership Principles (for me)
• Customer Obsession
• Ownership
• Invent and Simplify
• Are Right, A Lot
• Learn and Be Curious
• Hire and Develop the Best
• Insist on the Highest
Standards
• Think Big
• Bias for Action
• Frugality
• Earn Trust
• Dive Deep
• Have Backbone; Disagree
and Commit
• Deliver Results
https://www.amazon.jobs/en/principles

18. Working Backwards

19. Press Release, Narratives, FAQ

20. Diversity and Inclusion

21. It is still Day One

22. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
Innovation and Data

23. History of Innovation
Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

24. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

25. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

26. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

27. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

28. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

29. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

30. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

31. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

32. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

33. We are here
https://www.weforum.org/about/the-fourth-industrial-revolution-by-klaus-schwab
Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
??

34. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
4th Industrial
Revolution

35. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
4th Industrial
Revolution

36. Industrial
Revolution
2nd Industrial
Revolution
Digital
Revolution
4th Industrial
Revolution

38. AWS Rapid Pace of Innovation
Chart: #AWS services
https://www.slideshare.net/AmazonWebServices/a-culture-of-innovation-powered-by-aws

39. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
Analytics powered by AWS

40. For Data to be a differentiator, customers
need to be able to…
• Capture and store new non-
relational data at PB-EB scale
in real time
• Discover value in a new type
of analytics that go beyond
batch reporting to incorporate
real-time, predictive, voice,
and image recognition
• Democratize access to data in
a secure and governed way
New types of analytics
Dashboards Predictive Image
Recognition
VoiceReal-time
New types of data

41. OLTP ERP CRM LOB
Data Warehouse
Business
Intelligence
§ Relational data sources
§ TBs–PBs scale
§ Schema defined prior to data
load
§ Operational reporting and ad
hoc
Traditionally, analytics used to look
like this

42. Data & analytics partners extend the
traditional approach
Data Warehouse
Business
Intelligence
OLTP ERP CRM LOB Devices Web Sensors Social
Big Data processing,
real-time, Machine
Learning
Data Lake
§ Relational and non-relational
data
§ TBs–EBs scale
§ Diverse analytical engines
§ Low-cost storage & analytics

43. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
Abebooks DW Modernization

44. Abebooks BI/DW (before)
Storage LayerSource Layer
Ad-hoc SQL
SFTP
Data Warehouse
ETL (PL/SQL)
Files
Inventory
Sales
Access Layer

45. BI/DW Survey

46. Cloud Migration Strategy
Lift & Shift
• Typical Approach
• Move all-at-once
• Target platform then evolve
• Approach gets you to the cloud
quickly
• Relatively small barrier to learning
new technology since it tends to be a
close fit
Split & Flip
• Split application into logical
functional data layers
• Match the data functionality with
the right technology
• Leverage the wide selection of
tools on AWS to best fit the need
• Move data in phases — prototype,
learn and perfect

48. Choosing ETL Tool for Cloud
Use Cases
• OLTP to S3
• S3 to Redshift
• SFTP/API to
Redshift
• Data
Transformation
• Dimensional
Modelling
Tools
• Pentaho DI
• Informatica
• AWD Data
Pipeline/Glue
• Talend
• Matillion

49. ETL Criteria
High:
• Support
native
Redshift
driver
• Easily
capture
from
relational
db, CDC
• Ease of Use
for BI/DW
• Cover use
cases
• On-Premise
Medium:
• Support NoSQL
• Company “Winner”
• Deployment/Architecture
• Encryption
• Ease of Use for non
BI/DW
• Data Transformations
• Management
• Pricing
• Performance
Low:
• Version
Control
• Linux OS
• ETL
Monitoring
• Logging
• R/Pyhton

50. Why We Picked Matillion
• specific redshift support, built around Redshift
platform
• speed of ETL operations
• speed of development
• wide range of data sources supported
• ease of use outside of DE/DBA expertise
• Native with AWS
• $$$
• The biggest risk – putting our eggs in the Matillion
future, betting on a small and new player.

51. Source
Layer
OLTP
External
API
SFTP
Ad-hock
SQL
Business
Layer
Amazon
Redshift
Amazon
Elastic
Load
Balance
SQS SNSAmazon
Chime
AWS Virtual Private Cloud
Matillion
Amazon EC2
instance
Event/Notification
Services
Tableau
Web
Tableau
Desktop
Abebooks DW Modernization
(after)
ETL
ELT
Tableau Server
Amazon EC2
instance

52. Abebooks DW extending
Source
Layer
DynamoDB
Amazon
RDS
External
API
SFTP
APPs Ad-hock
SQL
Business
Layer
Amazon
Athena
Amazon
EMR
Amazon
Redshift
Amazon Redshift
Spectrum
Amazon
Elastic
Load
Balance
SQS SNS
Amazon
Chime
Tableau
Server
AWS Virtual Private Cloud
Matillion
Amazon EC2
instance
Event/Notification
Services
Amazon S3 Data Lake
Tableau
Web
Tableau
Desktop
Apache
Spark
Big Data Processing

53. How does it look and feel?

54. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
Prices

55. X-Small Package Medium Package
• BI Server** (EC2) (16vCPU/64RAM) –
585$
• ETL Server (EC2) (8 RAM) – 73$
• Data Lake (S3) 50 TB – 1200$
• Big Data Processing (EMR) 3node
• DW (Redshift) 10 TB – 2500$
• 3rd Party ETL (Matillion) 1780$
• Redshift Spectrum ~500$
• Support – 460$
Total: ~7098*$
Example of Monthly Prices in US$
* you might get significant discount for Yearly Reserved Instances
** not include BI tool license cost
https://calculator.s3.amazonaws.com/index.html
• BI Server (EC2) – 146$
• ETL Server (EC2) – 31$
• DW (Redshift) 2 TB – 622$
• S3 Storage (50Gb) – 1.15
• 3rd Party ETL (Matillion) 986$
• Support – 123$
Total: ~2348$*

56. ---------------------------------------------------------------
Outline
---------------------------------------------------------------
Free Learning Resources

57. Coursera:
• Data Warehouse for Business Intelligence Specialization
• Data Engineering on Google Cloud Platform
• Architecting with Google Cloud Platform
AWS Tutorials:
• Getting Started with Amazon Redshift
• Sizing Amazon Redshift
• Getting Started with Amazon Spectrum, Athena, Glue, EMR
• AWS Free Tier (for example 2 months of Redshift)
AWS Trainings:
• AWS Technical Essentials
Other:
Google Machine Learning Crash Course (Deep Learning with TensorFlow)
http://hackerrank.com
Sql-ex.ru