SlideShare ist ein Scribd-Unternehmen logo

Modern Data Architecture

ā€¢
ā€¢
Alexey Grishchenko
Alexey Grishchenko

This is the presentation for the talk I gave at JavaDay Kiev 2015. This is about an evolution of data processing systems from simple ones with single DWH to the complex approaches like Data Lake, Lambda Architecture and Pipeline architecture

Daten & Analysen
1 von 100
1Pivotal Confidentialā€“Internal Use Only 1Pivotal Confidentialā€“Internal Use Only Modern Data Architecture Alexey Grishchenko
2Pivotal Confidentialā€“Internal Use Only About me Enterprise Architect @ Pivotal ļ‚Ÿ 7 years in data processing ļ‚Ÿ 5 years with MPP ļ‚Ÿ 4 years with Hadoop ļ‚Ÿ Spark contributor ļ‚Ÿ http://0x0fff.com
3Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End
4Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End
5Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS
6Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS What about BI?
7Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS Just put it there!
8Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS BI
9Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS BI Was it fast?
10Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End 10ms Back End DBMS BI 100ms 200ms 1-2 min
11Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End 10ms Back End DBMS BI 100ms 200ms 1-2 min yes, single serverā€¦
12Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 200ms 1-2 min More users got workstations
13Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 400ms 800ms 1-2 min
14Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 400ms 800ms 1-2 min Split!
15Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 300ms 600ms 1-2 min
16Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 300ms 600ms 1-2 min Even more users?
17Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 300ms 600ms 1-2 min Split!
18Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 400ms 1-2 min Front End Back End Front End Back End
19Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 400ms 1-2 min Front End Back End Front End Back End What about automated systems?
20Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 1 sec 5-10 min Front End Back End Front End Back End Front End Back End Front End Back End
21Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 1 sec 5-10 min Front End Back End Front End Back End Front End Back End Front End Back End Database, please, live!
22Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 1 sec 5-10 min Front End Back End Front End Back End Front End Back End Front End Back End
23Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 800ms 15-20 min Front End Back End Front End Back End Front End Back End Front End Back End
24Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 800ms 15-20 min Front End Back End Front End Back End Front End Back End Front End Back End What if ā€œsplitā€ didnā€™t help this time?
25Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 800ms 15-20 min Front End Back End Front End Back End Front End Back End Front End Back End Split more! Eventually it will helpā€¦
26Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS
27Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS
28Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up!
29Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up! Sales went 20% down!
30Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 600ms 2-3 hrs Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up! Sales went 20% down!
31Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 600ms 2-3 hrs Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up! Sales went 20% down! Stop loading my system with your stupid reports!
32Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 2 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day
33Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 2 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day We need more reports!
34Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 3-4 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day Data Mining OLAPā€¦
35Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 3-4 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day Data Mining OLAPā€¦ We need secondary site!
36Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦
37Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late
38Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late
39Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late Where is our DWH? We need this data now!
40Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late
41Pivotal Confidentialā€“Internal Use Only ETL The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS
42Pivotal Confidentialā€“Internal Use Only ETL The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS Why is this data so old?
43Pivotal Confidentialā€“Internal Use Only ETL The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS
44Pivotal Confidentialā€“Internal Use Only ETL Advanced Architecture ā€“ ELT 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS DBMS DBMS DBMSā€¦ ETL DDS Data Marts Reports Aggregates OLAP DBMS DBMS DBMSā€¦ ELT DDS Data Marts Reports Aggregates OLAP ODS ODS ODSā€¦
45Pivotal Confidentialā€“Internal Use Only ELT Advanced Architecture ā€“ ELT 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS
46Pivotal Confidentialā€“Internal Use Only ELT Advanced Architecture ā€“ CDC 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS DBMS DBMS DBMSā€¦ ELT DDS Data Marts Reports Aggregates OLAP ODS ODS ODSā€¦ DBMS DBMS DBMSā€¦ ELT DDS Data Marts Reports Aggregates OLAP ODS ODS ODSā€¦ CDC 1 day 1 hour
47Pivotal Confidentialā€“Internal Use Only ELT CDC Advanced Architecture ā€“ CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH
48Pivotal Confidentialā€“Internal Use Only ELT CDC Advanced Architecture ā€“ CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Why is our secondary siteā€™s DWH so old?
49Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Moving Forward
50Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are Moving Forward
51Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days Moving Forward
52Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing Moving Forward
53Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing ļƒ˜ DWH MPP storage is expensive Moving Forward
54Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing ļƒ˜ DWH MPP storage is expensive Data Lake
55Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing ļƒ˜ DWH MPP storage is expensive Lambda Data Lake
56Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Data Lake 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Hadoop DBMS DBMS DBMSā€¦ ELT DDS OLAP Data Marts Aggregates Reports ODS ODS ODSā€¦ CDC DWH ODS UDS Analytical Archives BI Data Mining OLAP SQL-on-Hadoop Data Mining At Scale
57Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Data Lake 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH
58Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Data Lake 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late Data Mining BI OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ?
59Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Lambda 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late Data Mining BI OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? Source Data Speed Layer Batch Layer Serving Layer Query Query Master Dataset Batch View Batch View Batch View Real-time View Real-time View Real-time View
60Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Lambda 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late Data Mining BI OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ?
61Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures ā€“ Lambda 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining
62Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are
63Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems
64Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster?
65Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster? ļƒ˜ How to sync data in real-time systems?
66Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster? ļƒ˜ How to sync data in real-time systems? ļƒ˜ How to better sync DWH?
67Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster? ļƒ˜ How to sync data in real-time systems? ļƒ˜ How to better sync DWH? Pipelining
68Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining
69Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP
70Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP
71Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Table
72Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table
73Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch
74Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL
75Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DWH
76Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DDS DWH
77Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DDS DataMart DWH
78Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DDS DataMart DWH JDBC
79Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL ODS DDS DataMart DWH JDBC
80Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatch
81Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatch loadETL
82Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL
83Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL STG BatchApp Hadoop HDFS SQL On Hadoop
84Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL STG BatchApp Hadoop HDFS SQL On Hadoop RTI App
85Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL STG BatchApp Hadoop HDFS SQL On Hadoop RTI AppReplicate
86Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining
87Pivotal Confidentialā€“Internal Use Only ELT CDC FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH OLAP Data Mining RTBIā€¦ FE BE FE BE FE BE CDC Hadoop In-Memory Data Store BI Modern Data Architecture ā€“ Pipelining Replication Queue 3-5 minutes late In-Memory Data Store OLAPā€¦ DWHHadoop BI Data Mining RTBI DBMS DBMS DBMSWAL Replication 3-5 minutes late
88Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL
89Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI HTTP Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Pivotal Cloud Foundry FE ā€¦ App App App Queue BE ā€¦ App App App ļ‚Ÿ Pivotal Labs ā€“ agile software development for next-generation applications ļ‚Ÿ Pivotal Cloud Foundry ā€“ PaaS for customer applications ļ‚Ÿ RabbitMQ ā€“ distributed message queue service on top of PCF ļ‚Ÿ Spring IO ā€“ foundation platform for modern applications
90Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Pivotal GemFire App Pivotal GemFire and Apache Geode (incubating) ā€“ in-memory data grid enabling real-time data processing and real-time decision making for enterprises
91Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Spring XD Streaming Spring XD ā€“ unified, distributed and extensible framework for data pipelining: ingesting, batching, processing and exporting
92Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming ES DDS DataMart Pivotal Greenplum PostgreSQL SP Table ODS ETL ETL Streaming Data Pivotal HD Pivotal HAWQ Data Mart ļ‚Ÿ Pivotal HD ā€“ leading Hadoop distribution based on ODP ļ‚Ÿ Pivotal HAWQ and Apache HAWQ (incubating) ā€“ bringing the power of MPP to the Hadoop cluster, best in class SQL-on- Hadoop solution ļ‚Ÿ Apache Spark ā€“ component of the Pivotal HD distribution, modern framework for distributed data processing
93Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data Mart ODS ETL ETL PostgreSQL SP Table ļ‚Ÿ Pivotal PostgreSQL ā€“ commercially supported by Pivotal open source distribution of PostgreSQL
94Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ Data MartPostgreSQL SP Table ETL ETL ES DDS DataMart Pivotal Greenplum ODS Pivotal Greenplum ā€“ leading analytical MPP database, foundation for the enterprise data warehousing systems and advanced analytics
95Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture Pivotal GemFire App Spring XD Streaming BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Data Lake
96Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Spring XD Streaming ES DDS DataMart Pivotal Greenplum PostgreSQL SP Table ODS ETL ETL Pivotal GemFire App Streaming Data Pivotal HD Pivotal HAWQ Data Mart BI Lambda Architecture
97Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture ES DDS DataMart Pivotal Greenplum PostgreSQL SP Table ODS ETL ETL Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Streaming Pivotal HD BI Pivotal GemFire App Spring XD Streaming Data Pivotal HAWQ Data Mart Pipelining
98Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL
99Pivotal Confidentialā€“Internal Use Only 99Pivotal Confidentialā€“Internal Use Only Questions?
BUILT FOR THE SPEED OF BUSINESS

Empfohlen

Modern Data Architecture
Modern Data Architecture Modern Data Architecture
Modern Data Architecture Mark Hewitt
Ā 
Architectā€™s Open-Source Guide for a Data Mesh Architecture
Architectā€™s Open-Source Guide for a Data Mesh ArchitectureArchitectā€™s Open-Source Guide for a Data Mesh Architecture
Architectā€™s Open-Source Guide for a Data Mesh ArchitectureDatabricks
Ā 
Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...
Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...
Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...DATAVERSITY
Ā 
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDatabricks
Ā 
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?DATAVERSITY
Ā 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Jeffrey T. Pollock
Ā 
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?DATAVERSITY
Ā 
Microsoft Data Platform - What's included
Microsoft Data Platform - What's includedMicrosoft Data Platform - What's included
Microsoft Data Platform - What's includedJames Serra
Ā 

Empfohlen

Modern Data Architecture
Modern Data Architecture Modern Data Architecture
Modern Data Architecture Mark Hewitt
Ā 
Architectā€™s Open-Source Guide for a Data Mesh Architecture
Architectā€™s Open-Source Guide for a Data Mesh ArchitectureArchitectā€™s Open-Source Guide for a Data Mesh Architecture
Architectā€™s Open-Source Guide for a Data Mesh ArchitectureDatabricks
Ā 
Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...
Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...
Data Architecture, Solution Architecture, Platform Architecture ā€” Whatā€™s the ...DATAVERSITY
Ā 
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDatabricks
Ā 
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?DATAVERSITY
Ā 
Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Webinar Data Mesh - Part 3
Webinar Data Mesh - Part 3Jeffrey T. Pollock
Ā 
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?
Emerging Trends in Data Architecture ā€“ Whatā€™s the Next Big Thing?DATAVERSITY
Ā 
Microsoft Data Platform - What's included
Microsoft Data Platform - What's includedMicrosoft Data Platform - What's included
Microsoft Data Platform - What's includedJames Serra
Ā 
Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for DinnerKent Graziano
Ā 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPDatabricks
Ā 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureDatabricks
Ā 
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardDelta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardParis Data Engineers !
Ā 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshJeffrey T. Pollock
Ā 
The ABCs of Treating Data as Product
The ABCs of Treating Data as ProductThe ABCs of Treating Data as Product
The ABCs of Treating Data as ProductDATAVERSITY
Ā 
Snowflake Best Practices for Elastic Data Warehousing
Snowflake Best Practices for Elastic Data WarehousingSnowflake Best Practices for Elastic Data Warehousing
Snowflake Best Practices for Elastic Data WarehousingAmazon Web Services
Ā 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture DesignKujambu Murugesan
Ā 
Snowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleSnowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleAdam Doyle
Ā 
8 Steps to Creating a Data Strategy
8 Steps to Creating a Data Strategy8 Steps to Creating a Data Strategy
8 Steps to Creating a Data StrategySilicon Valley Data Science
Ā 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks FundamentalsDalibor Wijas
Ā 
Data Mesh
Data MeshData Mesh
Data MeshPiethein Strengholt
Ā 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogDATAVERSITY
Ā 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouseJames Serra
Ā 
Business Intelligence & Data Analyticsā€“ An Architected Approach
Business Intelligence & Data Analyticsā€“ An Architected ApproachBusiness Intelligence & Data Analyticsā€“ An Architected Approach
Business Intelligence & Data Analyticsā€“ An Architected ApproachDATAVERSITY
Ā 
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Dr. Arif Wider
Ā 
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...HostedbyConfluent
Ā 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceDenodo
Ā 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureDATAVERSITY
Ā 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data GovernanceDATAVERSITY
Ā 
Apache Spark Architecture
Apache Spark ArchitectureApache Spark Architecture
Apache Spark ArchitectureAlexey Grishchenko
Ā 
MapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APIMapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APImcsrivas
Ā 

Weitere Ƥhnliche Inhalte

Was ist angesagt?

Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for DinnerKent Graziano
Ā 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPDatabricks
Ā 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureDatabricks
Ā 
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardDelta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardParis Data Engineers !
Ā 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshJeffrey T. Pollock
Ā 
The ABCs of Treating Data as Product
The ABCs of Treating Data as ProductThe ABCs of Treating Data as Product
The ABCs of Treating Data as ProductDATAVERSITY
Ā 
Snowflake Best Practices for Elastic Data Warehousing
Snowflake Best Practices for Elastic Data WarehousingSnowflake Best Practices for Elastic Data Warehousing
Snowflake Best Practices for Elastic Data WarehousingAmazon Web Services
Ā 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture DesignKujambu Murugesan
Ā 
Snowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleSnowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleAdam Doyle
Ā 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks FundamentalsDalibor Wijas
Ā 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogDATAVERSITY
Ā 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouseJames Serra
Ā 
Business Intelligence & Data Analyticsā€“ An Architected Approach
Business Intelligence & Data Analyticsā€“ An Architected ApproachBusiness Intelligence & Data Analyticsā€“ An Architected Approach
Business Intelligence & Data Analyticsā€“ An Architected ApproachDATAVERSITY
Ā 
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Dr. Arif Wider
Ā 
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...HostedbyConfluent
Ā 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceDenodo
Ā 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureDATAVERSITY
Ā 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data GovernanceDATAVERSITY
Ā 

Was ist angesagt? (20)

Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for Dinner
Ā 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCP
Ā 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
Ā 
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardDelta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Ā 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
Ā 
The ABCs of Treating Data as Product
The ABCs of Treating Data as ProductThe ABCs of Treating Data as Product
The ABCs of Treating Data as Product
Ā 
Snowflake Best Practices for Elastic Data Warehousing
Snowflake Best Practices for Elastic Data WarehousingSnowflake Best Practices for Elastic Data Warehousing
Snowflake Best Practices for Elastic Data Warehousing
Ā 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture Design
Ā 
Snowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleSnowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at Scale
Ā 
8 Steps to Creating a Data Strategy
8 Steps to Creating a Data Strategy8 Steps to Creating a Data Strategy
8 Steps to Creating a Data Strategy
Ā 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
Ā 
Data Mesh
Data MeshData Mesh
Data Mesh
Ā 
Activate Data Governance Using the Data Catalog
Activate Data Governance Using the Data CatalogActivate Data Governance Using the Data Catalog
Activate Data Governance Using the Data Catalog
Ā 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
Ā 
Business Intelligence & Data Analyticsā€“ An Architected Approach
Business Intelligence & Data Analyticsā€“ An Architected ApproachBusiness Intelligence & Data Analyticsā€“ An Architected Approach
Business Intelligence & Data Analyticsā€“ An Architected Approach
Ā 
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...
Ā 
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Ā 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
Ā 
Improving Data Literacy Around Data Architecture
Improving Data Literacy Around Data ArchitectureImproving Data Literacy Around Data Architecture
Improving Data Literacy Around Data Architecture
Ā 
Data Architecture for Data Governance
Data Architecture for Data GovernanceData Architecture for Data Governance
Data Architecture for Data Governance
Ā 

Andere mochten auch

Apache Spark Architecture
Apache Spark ArchitectureApache Spark Architecture
Apache Spark ArchitectureAlexey Grishchenko
Ā 
MapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APIMapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APImcsrivas
Ā 
MapR Tutorial Series
MapR Tutorial SeriesMapR Tutorial Series
MapR Tutorial Seriesselvaraaju
Ā 
Architectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop DistributionArchitectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop Distributionmcsrivas
Ā 
Simplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache SparkSimplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache SparkDatabricks
Ā 
Hands on MapR -- Viadea
Hands on MapR -- ViadeaHands on MapR -- Viadea
Hands on MapR -- Viadeaviadea
Ā 
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)Amazon Web Services
Ā 
MapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR Technologies
Ā 
Apache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & InternalsApache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & InternalsAnton Kirillov
Ā 
Apache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and SmarterApache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and SmarterDatabricks
Ā 
MapR Data Analyst
MapR Data AnalystMapR Data Analyst
MapR Data Analystselvaraaju
Ā 
Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark InternalsPietro Michiardi
Ā 

Andere mochten auch (14)

Apache Spark Architecture
Apache Spark ArchitectureApache Spark Architecture
Apache Spark Architecture
Ā 
MapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APIMapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase API
Ā 
Deep Learning for Fraud Detection
Deep Learning for Fraud DetectionDeep Learning for Fraud Detection
Deep Learning for Fraud Detection
Ā 
Apache Spark & Hadoop
Apache Spark & HadoopApache Spark & Hadoop
Apache Spark & Hadoop
Ā 
MapR Tutorial Series
MapR Tutorial SeriesMapR Tutorial Series
MapR Tutorial Series
Ā 
Architectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop DistributionArchitectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop Distribution
Ā 
Simplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache SparkSimplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache Spark
Ā 
Hands on MapR -- Viadea
Hands on MapR -- ViadeaHands on MapR -- Viadea
Hands on MapR -- Viadea
Ā 
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
Ā 
MapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR and Cisco Make IT Better
MapR and Cisco Make IT Better
Ā 
Apache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & InternalsApache Spark in Depth: Core Concepts, Architecture & Internals
Apache Spark in Depth: Core Concepts, Architecture & Internals
Ā 
Apache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and SmarterApache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and Smarter
Ā 
MapR Data Analyst
MapR Data AnalystMapR Data Analyst
MapR Data Analyst
Ā 
Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark Internals
Ā 

Ƅhnlich wie Modern Data Architecture

The ninja elephant, scaling the analytics database in Transwerwise
The ninja elephant, scaling the analytics database in TranswerwiseThe ninja elephant, scaling the analytics database in Transwerwise
The ninja elephant, scaling the analytics database in TranswerwiseFederico Campoli
Ā 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL databaseDave Stokes
Ā 
20120426 high availability MySQL
20120426 high availability MySQL20120426 high availability MySQL
20120426 high availability MySQLJui-Nan Lin
Ā 
High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013Server Density
Ā 
Pluk2013 bodybuilding ratheesh
Pluk2013 bodybuilding ratheeshPluk2013 bodybuilding ratheesh
Pluk2013 bodybuilding ratheeshRatheesh Kaniyala
Ā 
All About Storeconfigs
All About StoreconfigsAll About Storeconfigs
All About StoreconfigsBrice Figureau
Ā 
The Importance of Data
The Importance of DataThe Importance of Data
The Importance of DataTrendz Lab
Ā 
PGDAY FR 2014 : presentation de Postgresql chez leboncoin.fr
PGDAY FR 2014 : presentation de Postgresql chez leboncoin.frPGDAY FR 2014 : presentation de Postgresql chez leboncoin.fr
PGDAY FR 2014 : presentation de Postgresql chez leboncoin.frjlb666
Ā 
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and HealthierPhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and HealthierDave Stokes
Ā 
IMS11 BMC Susbystem Optimizer - subzero
IMS11 BMC Susbystem Optimizer - subzeroIMS11 BMC Susbystem Optimizer - subzero
IMS11 BMC Susbystem Optimizer - subzeroRobert Hain
Ā 
Splunk-EMC
Splunk-EMCSplunk-EMC
Splunk-EMCBoni Bruno
Ā 
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...Severalnines
Ā 
Pinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberPinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberXiang Fu
Ā 
The 5 Minute DBA-DBA Skills for Non-DBA
The 5 Minute DBA-DBA Skills for Non-DBAThe 5 Minute DBA-DBA Skills for Non-DBA
The 5 Minute DBA-DBA Skills for Non-DBApercona2013
Ā 
VeeamON 2023 Architecting Veeam Backup for Microsoft 365 at Scale
VeeamON 2023 Architecting Veeam Backup for Microsoft 365 at ScaleVeeamON 2023 Architecting Veeam Backup for Microsoft 365 at Scale
VeeamON 2023 Architecting Veeam Backup for Microsoft 365 at ScaleJim Jones
Ā 
The future of tape april 16
The future of tape april 16The future of tape april 16
The future of tape april 16Josef Weingand
Ā 
Spectra Logic's BlackPearl Developers Summit 2016
Spectra Logic's BlackPearl Developers Summit 2016Spectra Logic's BlackPearl Developers Summit 2016
Spectra Logic's BlackPearl Developers Summit 2016spectralogic
Ā 
Why Wordnik went non-relational
Why Wordnik went non-relationalWhy Wordnik went non-relational
Why Wordnik went non-relationalTony Tam
Ā 

Ƅhnlich wie Modern Data Architecture (20)

Pinto+Stratopan+Love
Pinto+Stratopan+LovePinto+Stratopan+Love
Pinto+Stratopan+Love
Ā 
The ninja elephant, scaling the analytics database in Transwerwise
The ninja elephant, scaling the analytics database in TranswerwiseThe ninja elephant, scaling the analytics database in Transwerwise
The ninja elephant, scaling the analytics database in Transwerwise
Ā 
The care and feeding of a MySQL database
The care and feeding of a MySQL databaseThe care and feeding of a MySQL database
The care and feeding of a MySQL database
Ā 
20120426 high availability MySQL
20120426 high availability MySQL20120426 high availability MySQL
20120426 high availability MySQL
Ā 
High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013
Ā 
Ds @ bol
Ds @ bolDs @ bol
Ds @ bol
Ā 
Pluk2013 bodybuilding ratheesh
Pluk2013 bodybuilding ratheeshPluk2013 bodybuilding ratheesh
Pluk2013 bodybuilding ratheesh
Ā 
All About Storeconfigs
All About StoreconfigsAll About Storeconfigs
All About Storeconfigs
Ā 
The Importance of Data
The Importance of DataThe Importance of Data
The Importance of Data
Ā 
PGDAY FR 2014 : presentation de Postgresql chez leboncoin.fr
PGDAY FR 2014 : presentation de Postgresql chez leboncoin.frPGDAY FR 2014 : presentation de Postgresql chez leboncoin.fr
PGDAY FR 2014 : presentation de Postgresql chez leboncoin.fr
Ā 
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and HealthierPhpTek Ten Things to do to make your MySQL servers Happier and Healthier
PhpTek Ten Things to do to make your MySQL servers Happier and Healthier
Ā 
IMS11 BMC Susbystem Optimizer - subzero
IMS11 BMC Susbystem Optimizer - subzeroIMS11 BMC Susbystem Optimizer - subzero
IMS11 BMC Susbystem Optimizer - subzero
Ā 
Splunk-EMC
Splunk-EMCSplunk-EMC
Splunk-EMC
Ā 
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Ā 
Pinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberPinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ Uber
Ā 
The 5 Minute DBA-DBA Skills for Non-DBA
The 5 Minute DBA-DBA Skills for Non-DBAThe 5 Minute DBA-DBA Skills for Non-DBA
The 5 Minute DBA-DBA Skills for Non-DBA
Ā 
VeeamON 2023 Architecting Veeam Backup for Microsoft 365 at Scale
VeeamON 2023 Architecting Veeam Backup for Microsoft 365 at ScaleVeeamON 2023 Architecting Veeam Backup for Microsoft 365 at Scale
VeeamON 2023 Architecting Veeam Backup for Microsoft 365 at Scale
Ā 
The future of tape april 16
The future of tape april 16The future of tape april 16
The future of tape april 16
Ā 
Spectra Logic's BlackPearl Developers Summit 2016
Spectra Logic's BlackPearl Developers Summit 2016Spectra Logic's BlackPearl Developers Summit 2016
Spectra Logic's BlackPearl Developers Summit 2016
Ā 
Why Wordnik went non-relational
Why Wordnik went non-relationalWhy Wordnik went non-relational
Why Wordnik went non-relational
Ā 

KĆ¼rzlich hochgeladen

Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdf
Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdfKantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdf
Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdfSocial Samosa
Ā 
Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”soniya singh
Ā 
1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·
1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·
1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·vhwb25kk
Ā 
High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...
High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...
High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...soniya singh
Ā 
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.pptdokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.pptSonatrach
Ā 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics
Ā 
Defining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data StoryDefining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data StoryJeremy Anderson
Ā 
原ē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£
原ē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£åŽŸē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£
原ē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£208367051
Ā 
Customer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptxCustomer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptxEmmanuel Dauda
Ā 
RS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”DelhiRS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”Delhijennyeacort
Ā 
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDRafezzaman
Ā 
办ē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€
办ē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€åŠžē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€
办ē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€fhwihughh
Ā 
Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Colleen Farrelly
Ā 
ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degree
ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degree
ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeyuu sss
Ā 
MK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docxMK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docxUnduhUnggah1
Ā 
Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsapps
Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /WhatsappsBeautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsapps
Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsappssapnasaifi408
Ā 
Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)
Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)
Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)jennyeacort
Ā 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024thyngster
Ā 
Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...
Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...
Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...limedy534
Ā 
Multiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfMultiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfchwongval
Ā 

KĆ¼rzlich hochgeladen (20)

Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdf
Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdfKantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdf
Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdf
Ā 
Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls in Defence Colony Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Ā 
1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·
1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·
1:1定制(UQęƕäøščƁļ¼‰ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•äæ®ę”¹ē•™äæ”å­¦åŽ†č®¤čƁ原ē‰ˆäø€ęØ”äø€ę ·
Ā 
High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...
High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...
High Class Call Girls Noida Sector 39 Aarushi šŸ”8264348440šŸ” Independent Escort...
Ā 
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.pptdokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.ppt
Ā 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Ā 
Defining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data StoryDefining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data Story
Ā 
原ē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£
原ē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£åŽŸē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£
原ē‰ˆ1:1å®šåˆ¶å—åå­—ę˜Ÿå¤§å­¦ęƕäøščƁļ¼ˆSCUęƕäøščƁļ¼‰#ę–‡å‡­ęˆē»©å•#ēœŸå®žē•™äæ”å­¦åŽ†č®¤čƁę°øä¹…å­˜ę”£
Ā 
Customer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptxCustomer Service Analytics - Make Sense of All Your Data.pptx
Customer Service Analytics - Make Sense of All Your Data.pptx
Ā 
RS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”DelhiRS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”Delhi
RS 9000 Call In girls Dwarka Mor (DELHI)ā‡›9711147426šŸ”Delhi
Ā 
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
Ā 
办ē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€
办ē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€åŠžē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€
办ē†å­¦ä½čƁēŗ½ēŗ¦å¤§å­¦ęƕäøščƁ(NYUęƕäøščƁ书ļ¼‰åŽŸē‰ˆäø€ęƔäø€
Ā 
Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024
Ā 
ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degree
ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degree
ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degreeę¾³ę“²äø­å¤®ę˜†å£«å…°å¤§å­¦ęƕäøščÆęˆē»©å•pdfē”µå­ē‰ˆåˆ¶ä½œäæ®ę”¹#ęƕäøšę–‡å‡­åˆ¶ä½œ#å›žå›½å…„čŒ#diploma#degree
Ā 
MK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docxMK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docx
Ā 
Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsapps
Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /WhatsappsBeautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsapps
Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsapps
Ā 
Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)
Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)
Call Us āž„97111āˆš47426šŸ¤³Call Girls in Aerocity (Delhi NCR)
Ā 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Ā 
Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...
Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...
Effects of Smartphone Addiction on the Academic Performances of Grades 9 to 1...
Ā 
Multiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfMultiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdf
Ā 

Modern Data Architecture

  • 1. 1Pivotal Confidentialā€“Internal Use Only 1Pivotal Confidentialā€“Internal Use Only Modern Data Architecture Alexey Grishchenko
  • 2. 2Pivotal Confidentialā€“Internal Use Only About me Enterprise Architect @ Pivotal ļ‚Ÿ 7 years in data processing ļ‚Ÿ 5 years with MPP ļ‚Ÿ 4 years with Hadoop ļ‚Ÿ Spark contributor ļ‚Ÿ http://0x0fff.com
  • 3. 3Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End
  • 4. 4Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End
  • 5. 5Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS
  • 6. 6Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS What about BI?
  • 7. 7Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS Just put it there!
  • 8. 8Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS BI
  • 9. 9Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End Back End DBMS BI Was it fast?
  • 10. 10Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End 10ms Back End DBMS BI 100ms 200ms 1-2 min
  • 11. 11Pivotal Confidentialā€“Internal Use Only How it startedā€¦ Front End 10ms Back End DBMS BI 100ms 200ms 1-2 min yes, single serverā€¦
  • 12. 12Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 200ms 1-2 min More users got workstations
  • 13. 13Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 400ms 800ms 1-2 min
  • 14. 14Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 400ms 800ms 1-2 min Split!
  • 15. 15Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 300ms 600ms 1-2 min
  • 16. 16Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 300ms 600ms 1-2 min Even more users?
  • 17. 17Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 300ms 600ms 1-2 min Split!
  • 18. 18Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 400ms 1-2 min Front End Back End Front End Back End
  • 19. 19Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 400ms 1-2 min Front End Back End Front End Back End What about automated systems?
  • 20. 20Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 1 sec 5-10 min Front End Back End Front End Back End Front End Back End Front End Back End
  • 21. 21Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 1 sec 5-10 min Front End Back End Front End Back End Front End Back End Front End Back End Database, please, live!
  • 22. 22Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 1 sec 5-10 min Front End Back End Front End Back End Front End Back End Front End Back End
  • 23. 23Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 800ms 15-20 min Front End Back End Front End Back End Front End Back End Front End Back End
  • 24. 24Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 800ms 15-20 min Front End Back End Front End Back End Front End Back End Front End Back End What if ā€œsplitā€ didnā€™t help this time?
  • 25. 25Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 800ms 15-20 min Front End Back End Front End Back End Front End Back End Front End Back End Split more! Eventually it will helpā€¦
  • 26. 26Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS
  • 27. 27Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS
  • 28. 28Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up!
  • 29. 29Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 300ms 35-40 min Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up! Sales went 20% down!
  • 30. 30Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 600ms 2-3 hrs Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up! Sales went 20% down!
  • 31. 31Pivotal Confidentialā€“Internal Use Only First Issues Front End 10ms Back End DBMS BI 100ms 600ms 2-3 hrs Front End Back End Front End Back End Front End Back End Front End Back End DBMS DBMSDBMSDBMS Sales went 10% up! Sales went 20% down! Stop loading my system with your stupid reports!
  • 32. 32Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 2 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day
  • 33. 33Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 2 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day We need more reports!
  • 34. 34Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 3-4 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day Data Mining OLAPā€¦
  • 35. 35Pivotal Confidentialā€“Internal Use Only BI The Era of Data Warehouse 100ms DBMS 300ms 3-4 days FE BE DBMS DBMSDBMSDBMS FE BE FE BE FE BE FE BE ETL DWH 1 day Data Mining OLAPā€¦ We need secondary site!
  • 36. 36Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦
  • 37. 37Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late
  • 38. 38Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late
  • 39. 39Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late Where is our DWH? We need this data now!
  • 40. 40Pivotal Confidentialā€“Internal Use Only The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE WAL Replication 3-5 minutes late
  • 41. 41Pivotal Confidentialā€“Internal Use Only ETL The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS
  • 42. 42Pivotal Confidentialā€“Internal Use Only ETL The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS Why is this data so old?
  • 43. 43Pivotal Confidentialā€“Internal Use Only ETL The Era of Data Warehouse 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS
  • 44. 44Pivotal Confidentialā€“Internal Use Only ETL Advanced Architecture ā€“ ELT 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ETL DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS DBMS DBMS DBMSā€¦ ETL DDS Data Marts Reports Aggregates OLAP DBMS DBMS DBMSā€¦ ELT DDS Data Marts Reports Aggregates OLAP ODS ODS ODSā€¦
  • 45. 45Pivotal Confidentialā€“Internal Use Only ELT Advanced Architecture ā€“ ELT 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS
  • 46. 46Pivotal Confidentialā€“Internal Use Only ELT Advanced Architecture ā€“ CDC 100ms 300ms 3-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 1 day BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late DWH BI Data Mining OLAPā€¦ 5-7 days DBMS DBMS DBMS DBMS DBMS DBMS DBMS DBMSā€¦ ELT DDS Data Marts Reports Aggregates OLAP ODS ODS ODSā€¦ DBMS DBMS DBMSā€¦ ELT DDS Data Marts Reports Aggregates OLAP ODS ODS ODSā€¦ CDC 1 day 1 hour
  • 47. 47Pivotal Confidentialā€“Internal Use Only ELT CDC Advanced Architecture ā€“ CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH
  • 48. 48Pivotal Confidentialā€“Internal Use Only ELT CDC Advanced Architecture ā€“ CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Why is our secondary siteā€™s DWH so old?
  • 49. 49Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Moving Forward
  • 50. 50Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are Moving Forward
  • 51. 51Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days Moving Forward
  • 52. 52Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing Moving Forward
  • 53. 53Pivotal Confidentialā€“Internal Use Only ELT CDC 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing ļƒ˜ DWH MPP storage is expensive Moving Forward
  • 54. 54Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing ļƒ˜ DWH MPP storage is expensive Data Lake
  • 55. 55Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Our problems are ļƒ˜ Time to action takes up to 7 days ļƒ˜ Amount of data is growing ļƒ˜ DWH MPP storage is expensive Lambda Data Lake
  • 56. 56Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Data Lake 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH Hadoop DBMS DBMS DBMSā€¦ ELT DDS OLAP Data Marts Aggregates Reports ODS ODS ODSā€¦ CDC DWH ODS UDS Analytical Archives BI Data Mining OLAP SQL-on-Hadoop Data Mining At Scale
  • 57. 57Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Data Lake 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs BI Data Mining OLAPā€¦ FE BE FE BE FE BE FE BE FE BE WAL Replication 3-5 minutes late NAS NAS Backup / Restore 3 days late BI Data Mining OLAPā€¦ 4-7 days DBMS DBMS DBMS DBMS DBMS CDC DWH
  • 58. 58Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Data Lake 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late Data Mining BI OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ?
  • 59. 59Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Lambda 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late Data Mining BI OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? Source Data Speed Layer Batch Layer Serving Layer Query Query Master Dataset Batch View Batch View Batch View Real-time View Real-time View Real-time View
  • 60. 60Pivotal Confidentialā€“Internal Use Only ELT CDC Modern Architectures ā€“ Lambda 100ms 300ms 1-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 3-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late Data Mining BI OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ?
  • 61. 61Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures ā€“ Lambda 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining
  • 62. 62Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are
  • 63. 63Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems
  • 64. 64Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster?
  • 65. 65Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster? ļƒ˜ How to sync data in real-time systems?
  • 66. 66Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster? ļƒ˜ How to sync data in real-time systems? ļƒ˜ How to better sync DWH?
  • 67. 67Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC Modern Architectures 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Our problems are ļƒ˜ Too many standby systems ļƒ˜ How to replicate Hadoop cluster? ļƒ˜ How to sync data in real-time systems? ļƒ˜ How to better sync DWH? Pipelining
  • 68. 68Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining
  • 69. 69Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP
  • 70. 70Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP
  • 71. 71Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Table
  • 72. 72Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table
  • 73. 73Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch
  • 74. 74Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL
  • 75. 75Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DWH
  • 76. 76Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DDS DWH
  • 77. 77Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DDS DataMart DWH
  • 78. 78Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦SOAP OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL load ODS DDS DataMart DWH JDBC
  • 79. 79Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch ETL cp Batch ETL ODS DDS DataMart DWH JDBC
  • 80. 80Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatch
  • 81. 81Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatch loadETL
  • 82. 82Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL
  • 83. 83Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL STG BatchApp Hadoop HDFS SQL On Hadoop
  • 84. 84Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL STG BatchApp Hadoop HDFS SQL On Hadoop RTI App
  • 85. 85Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining FE BI App App App ā€¦HTTP BE Srv Srv Srv ā€¦ OLTP SP JDBC Log Table CDC copy Parse Batch load ODS DDS DataMart DWH JDBC API Queue ETL ETLBatchApp ETLBatch load loadETL STG BatchApp Hadoop HDFS SQL On Hadoop RTI AppReplicate
  • 86. 86Pivotal Confidentialā€“Internal Use Only In-Memory Data Store ELT CDC 100ms 300ms 0-4 days FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH 0-24 hrs OLAP Data Mining BIā€¦ FE BE FE BE FE BE NAS NAS Backup / Restore 2 days late OLAPā€¦ 3-6 days DBMS DBMS DBMS WAL Replication 3-5 minutes late CDC DWHHadoop Hadoop ? In-Memory Data Store RTDM BI Data Mining Modern Data Architecture ā€“ Pipelining
  • 87. 87Pivotal Confidentialā€“Internal Use Only ELT CDC FE BE DBMS DBMS FE BE DBMS FE BE ELT DWH OLAP Data Mining RTBIā€¦ FE BE FE BE FE BE CDC Hadoop In-Memory Data Store BI Modern Data Architecture ā€“ Pipelining Replication Queue 3-5 minutes late In-Memory Data Store OLAPā€¦ DWHHadoop BI Data Mining RTBI DBMS DBMS DBMSWAL Replication 3-5 minutes late
  • 88. 88Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL
  • 89. 89Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI HTTP Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Pivotal Cloud Foundry FE ā€¦ App App App Queue BE ā€¦ App App App ļ‚Ÿ Pivotal Labs ā€“ agile software development for next-generation applications ļ‚Ÿ Pivotal Cloud Foundry ā€“ PaaS for customer applications ļ‚Ÿ RabbitMQ ā€“ distributed message queue service on top of PCF ļ‚Ÿ Spring IO ā€“ foundation platform for modern applications
  • 90. 90Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Pivotal GemFire App Pivotal GemFire and Apache Geode (incubating) ā€“ in-memory data grid enabling real-time data processing and real-time decision making for enterprises
  • 91. 91Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Spring XD Streaming Spring XD ā€“ unified, distributed and extensible framework for data pipelining: ingesting, batching, processing and exporting
  • 92. 92Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming ES DDS DataMart Pivotal Greenplum PostgreSQL SP Table ODS ETL ETL Streaming Data Pivotal HD Pivotal HAWQ Data Mart ļ‚Ÿ Pivotal HD ā€“ leading Hadoop distribution based on ODP ļ‚Ÿ Pivotal HAWQ and Apache HAWQ (incubating) ā€“ bringing the power of MPP to the Hadoop cluster, best in class SQL-on- Hadoop solution ļ‚Ÿ Apache Spark ā€“ component of the Pivotal HD distribution, modern framework for distributed data processing
  • 93. 93Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data Mart ODS ETL ETL PostgreSQL SP Table ļ‚Ÿ Pivotal PostgreSQL ā€“ commercially supported by Pivotal open source distribution of PostgreSQL
  • 94. 94Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ Data MartPostgreSQL SP Table ETL ETL ES DDS DataMart Pivotal Greenplum ODS Pivotal Greenplum ā€“ leading analytical MPP database, foundation for the enterprise data warehousing systems and advanced analytics
  • 95. 95Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture Pivotal GemFire App Spring XD Streaming BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL Data Lake
  • 96. 96Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Spring XD Streaming ES DDS DataMart Pivotal Greenplum PostgreSQL SP Table ODS ETL ETL Pivotal GemFire App Streaming Data Pivotal HD Pivotal HAWQ Data Mart BI Lambda Architecture
  • 97. 97Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture ES DDS DataMart Pivotal Greenplum PostgreSQL SP Table ODS ETL ETL Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Streaming Pivotal HD BI Pivotal GemFire App Spring XD Streaming Data Pivotal HAWQ Data Mart Pipelining
  • 98. 98Pivotal Confidentialā€“Internal Use Only Pivotal and Modern Data Architecture BI Pivotal Cloud Foundry HTTP FE ā€¦ App App App Queue BE ā€¦ App App App Pivotal GemFire App Spring XD Streaming Streaming Data Pivotal HD Pivotal HAWQ ES DDS DataMart Pivotal Greenplum Data MartPostgreSQL SP Table ODS ETL ETL
  • 99. 99Pivotal Confidentialā€“Internal Use Only 99Pivotal Confidentialā€“Internal Use Only Questions?
  • 100. BUILT FOR THE SPEED OF BUSINESS