Presentation by Raffael Dzikowski and Sean Gustafson of Scout24 AG at the 2019 AWS Summit Berlin. Abstract: The Scout24 Data Platform powers all reporting, ad hoc analytics and machine learning products at AutoScout24 and ImmobilienScout24. In this talk, we will take a technical deep dive into our modern, cloud-based big data platform. We will discuss our evolution of approaches to ingestion, ETL, access control, reporting and machine learning with a focus on in-the-trenches learnings gained from our many failures and successes as we migrated from a traditional Oracle Data Warehouse to an AWS-based data lake.

1. S U M M I T
B E RLIN

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24 Data Platform
ATechnical Deep Dive
Sean Gustafson
Senior Technical Product Manager
Scout24
S e s s i o n I D
Raffael Dzikowski
Senior Data Engineer
Scout24

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
5
Core Geographies
and an overall presence
in 18 countries
80m
Household Reach
2
Major Household Brand Names
Scout24 AG
• SDAX
• € 489 million revenue (2017)
• ˜1500 employees

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Our technicalevolution
Production
Database
Monolith app
Data Warehouse Data
Warehouse
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Microservice
Persistence
Persistence
Persistence
Persistence
Persistence
Persistence
Persistence

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Our datawarehousewasabottleneck

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scout24 wants to become a truly data-driven company
Fast & easy data-driven
product development…
…supported by
Data & Analytics

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scout24 wants to become a truly data-driven company
Everywhere in the company... ...without bloating up Data &
Analytics

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Our solution:
Build an internal “platform” for data

9. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
We thinkof our Data Platform asa Product
Just like AWS, Salesforce, etc. – the platform is a generic layer upon which
Scout24’s products can be built
BUT, we have a very, very small number of customers.
That means, product teams get personalized support and there is lots of
opportunity for collaboration.

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Wedon’tdictateanything.
Wejusttrytomakecertainthingseasierbyofferinga“pavedpath”
Productteamsarefullyempoweredtomaketheirownchoicesaboutwhatisthe
bestuseoftheirresources.

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
“Inalmostallcases,wewillnot
mandatethatinternalteamuse
theseplatformsandservices—
theseplatform teamswill
to win over andsatisfy their
internalcustomers,
evencompetingwithexternal
vendors.”

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Guiding principle of theplatform
Autonomy for producers and consumers
Self-service Analytics
Self-service Data Ingestion
Self-service ETL

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Self-service ETL
Self-service Analytics
Central Data Lake on Amazon S3
Data
Scientist
AnalystPM Leader
Engineer
Analyst
Self-service Data Ingestion
Data
Producer

15. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OurApproach

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Multi-AccountSetting

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data LakeBucketTypes

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataLakeAccessRoles
Data Lake Account
ImmobilienScout24
Data Lake Account
AutoScout24
Regular DL Bucket Access Role
Restricted DL Bucket Access Roles
Personal DL Bucket Access Roles

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
IngestionGoals
Microservices
Ingestion
Goals
Batch Support
Streaming Support
Rest APIScalability

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
IngestionOptions
Amazon
KinesisData
Firehose
Kafka Connect

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
IngestionOptions
Amazon
KinesisData
Firehose
Kafka Connect
Simple
Config
Simple
Config

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
FirehoseIngestionArchitecture
Data Lake Account
Amazon
KinesisData
Firehose
Producer to
Firehose Role
Producer Account
Data Producer
Firehose to
Datalake Role
Datalake Bucket

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
FirehoseIngestionArchitecture
Data Lake Account
Amazon
KinesisData
Firehose
Producer to
Firehose Role
Producer Account
Data Producer
Firehose to
Datalake Role
Datalake Bucket
STS Assume
Role
STS Assume
Role

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
FirehoseIngestionArchitecture
Data Lake Account
Amazon
KinesisData
Firehose
Producer to
Firehose Role
Producer Account
Data Producer
Firehose to
Datalake Role
Datalake Bucket
STS Assume
Role
STS Assume
Role
Send Data

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
FirehoseIngestionArchitecture
Data Lake Account
Amazon
KinesisData
Firehose
Producer to
Firehose Role
Producer Account
Data Producer
Firehose to
Datalake Role
Datalake Bucket
STS Assume
Role
STS Assume
Role
Send Data
Write Data

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnect

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnect
Elasticsearch
Amazon S3
ActiveMQ
Cassandra
Kafka
RDMBS…

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnect
Elasticsearch
Amazon S3
ActiveMQ
Cassandra
Kafka
RDMBS…
Kafka
Connect
Cluster

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnect
Elasticsearch
Amazon S3
ActiveMQ
Cassandra
Kafka
RDMBS…
Kafka
Connect
ClusterRead Data from
Topic(s)

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnect
Elasticsearch
Amazon S3
ActiveMQ
Cassandra
Kafka
RDMBS…
Kafka
Connect
ClusterRead Data from
Topic(s)
Write
Data

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scout24InfinityCluster

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scout24InfinityCluster
Amazon ECS
Infinity Service
Simple
Config

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scout24InfinityCluster
Amazon ECS
Infinity Service
Simple
Config
Central Logging to
Elasticsearch
Monitoring in Datadog
Managed By Scout24
Cloud Platform
Engineering

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
RelatedBreakouts
15:00 in Hall 1
To Infinity and Beyond – Handling Heterogeneous Container
Clusters in AWS
Christine Trahe, Platform Engineer @ Scout24
16:00 in Hall 1
Boost your AWS Infrastructure
Philipp Garbe, AWS Container Hero @ Scout24

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnecton Infinity
Simple
Config
(Infinity)
Amazon ECS
Infinity Service

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnecton Infinity

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnecton Infinity

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnecton Infinity

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnecton Infinity
Simple
Config
(Kafka Connect)
Kafka Connect Service
Infinity Service

41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnectDeployment

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnectDeployment

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnectDeployment

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnectDeployment

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KafkaConnectDeployment

46. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataWario –OurWrapper forAWS DataPipeline

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataWario –OurWrapper forAWS DataPipeline
Simple
Config
AWS Data
Pipeline
DataWario

49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataWario –OurWrapper forAWS DataPipeline
Builtin Support for
Scout24 Ecosystem
Shortens
Development Cycles
Only Exposes
Configuration
Essentials
Introduces Custom
Step Types
Automatically
Manages Artifacts
Simple
Config
AWS Data
Pipeline
DataWario

50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataWarioArchitecture

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataWarioArchitecture

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataWarioArchitecture

53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DataWarioArchitecture

54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Library ofCommon DataTransformations

55. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
QueryChallenges

57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’sAhead

58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’sAhead
OneScout Hive Metastore
Unlock the Datalake for Scout24’s
Toolset and Users with Different
Skillsets
Data Analysis for Various User
Groups
Provide a Timely and Accurate
Update of the Metadata Layer

59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’sAhead
Personal Analytics ClusterOneScout Hive Metastore
Unlock the Datalake for Scout24’s
Toolset and Users with Different
Skillsets
Data Analysis for Various User
Groups
Provide a Timely and Accurate
Update of the Metadata Layer

60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’sAhead
Automatic Hive Partition DetectionPersonal Analytics ClusterOneScout Hive Metastore
Unlock the Datalake for Scout24’s
Toolset and Users with Different
Skillsets
Data Analysis for Various User
Groups
Provide a Timely and Accurate
Update of the Metadata Layer

61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OneScout HiveMetastore –ASchematicView
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake

62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OneScout HiveMetastore – Recapof Ecosystem

63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

65. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

66. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

67. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

68. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

69. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

70. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

71. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

72. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

73. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

74. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EMR MetastoreConfigurationOptions

75. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24Hive MetastoreProxy –AMotivation

76. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24Hive MetastoreProxy –AMotivation

77. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24Hive MetastoreProxy –AMotivation

78. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24Hive MetastoreProxy –AMotivation

79. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24Hive MetastoreProxy –AMotivation

80. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24Hive MetastoreProxy –AMotivation

81. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
TheScout24Hive MetastoreProxy –AMotivation

82. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ThePersonalAnalyticsCluster –AnOverview

83. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ThePersonalAnalyticsCluster –AnOverview
Amazon EMR
Personal Analytics
Cluster
Simple
Config

84. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ThePersonalAnalyticsCluster –AnOverview
Amazon EMR
Personal Analytics
Cluster
Simple
Config
Easy Access via Web
Interface
Zeppelin and Jupyter
Notebook Restore
OneClick Deployment
Managed Scaling and
Shutdown
Support for Pre-baked
AMIs and Configs

85. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ThePersonalAnalyticsCluster –AnOverview

86. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake

87. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake
Partitioned Table

88. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake
Partitioned Table
Data Ingestion

89. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake
Partitioned Table
Data Ingestion

90. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake
Partitioned Table
Data Ingestion
Table Access

91. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake
Partitioned Table
Data Ingestion
Table Access

92. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake
Partitioned Table
Data Ingestion
Table Access
Automatic Partition Detection

93. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automated PartitionDetection –AMotivation
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake
Partitioned Table
Data Ingestion
Table Access
Automatic Partition Detection

94. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

95. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

96. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

97. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

98. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

99. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

100. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

101. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

102. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
PartitionDetectionArchitecture

103. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

104. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Build our own vs.AWSmanaged services
Metastore  Glue
Presto  Athena
DataWario  Glue, Step function, Lambda, …
Personal Analytics Cluster  Glue notebooks, Sagemaker
We hope to throw out most of the custom components we build.

105. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Our datawarehousewasabottleneck

106. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Our data platform holds nothing back

107. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Raffael Dzikowski
Senior Data Engineer
Scout24
Sean Gustafson
Senior Technical Product Manager
Scout24

108. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

110. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Extra slides

111. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Centralized Federated
Control Autonomy
Perfection Scale
Pull Push
Product is Data Product is Platform
Reporting Reporting, Advanced Analytics,
Machine Learning, etc.
DataWarehouse vs. DataPlatform

112. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Core
DB
APPAPPAPPAPPAPPAPPAPPAPPAPP
MicroStrategy
Presto
Central Data Lake on S3
CRM Core
DB
Micro
Service
REST API / Firehose
Data
Scientist
Jupyter
Zeppelin
Analyst
PM
Data
Producer
Personal
Analytics
Clusters
SQL
Alation
Data Catalog
Metastore
Leader
Engineer
Analyst
DataWario
Spark

113. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OurJourney to Presto
Personal Analytics
Cluster
Hive Tables and Presto Views
Datalake

114. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OurJourney to Presto

115. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OurJourney to Presto
Amazon
Athena

116. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OurJourney to Presto
Amazon
Athena

117. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OurJourney to Presto
Amazon
Athena
On EMR

118. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
OurJourney to Presto
Amazon
Athena
On EMR
Cross Account Support
(OneScout Hive
Metastore)
Leverages Datalake
Access Roles (EMRFS)
Scheduled Scaling
Configurations
Fits our GDPRConcept
(multiple isolated
Clusters)

119. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

120. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
SCOUT24
DATA LANDSCAPE
MANIFESTO
ROLES, RESPONSIBILITIES, AND VALUES
FOR A DATA-DRIVEN COMPANY AT SCALE

121. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data is a key asset of our
company.
#1 Preamble

122. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
#2 Our Responsibility
We, Data & Analytics, are
responsible for providing a
solid Data Platform as well
as clear guidelines and
training how to participate
in the Data Landscape. Data Platform
DnA
Data Landscape

123. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
#3 Data Autonomy, Not Anarchy
Data autonomy puts data
producers & data consumers in
control of their data & of
their metrics and thereby allows
us to be data-driven at scale, but
this comes with responsibility. Data Platform
Data
Producer Consumer
DnA
Data Landscape

124. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
#4 Producer’s Responsibility
Data producers are responsible
for publishing data to the
central Data Lake, for the
data's quality, and for
publishing metadata that
makes it easy to find and
consume the data.
Data Platform
Metadata
Data
Producer
DnA
Data Landscape

125. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
#5 Consumer’s Responsibility
Data consumers are responsible
for the definition & visualization
of metrics and for driving the
implementation and
maintenance of these metrics.
Data Platform
Producer Consumer
DnA
Data Landscape

126. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
#6 Exception: Core KPIs
We, Data & Analytics, take the
full ownership and
responsibility of the few top
company-wide core KPIs.
Data Platform
Producer Consumer
DnA
Data Landscape
Core
metric

127. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
#7 Transparency Over Continuity
We value data transparency
over data continuity, which
means we may break metric
comparability if it is for the
cause of enabling better
insights. Data Platform
Producer Consumer
DnA
Data Landscape
Core
metric

128. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The Ultimate Goal
Data Platform
Metadata
Data
Producer Consumer
DnA
Data Landscape
Core
metric
A federal landscape of data
producers and consumers with
just enough rules to ensure
seamless co-operation without
severely impeding autonomy.