Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault Modelling and Data Governance conference on Oct. 17, 2019: Physical and Virtual Dstsa Vault 2.0 Automation with WhereScape at a Major Dutch Bank

•

1 like•143 views

The document summarizes Erik van der Hoeven's journey transforming customer data into wisdom through various data warehouse projects using DataVault 2.0 and automation with WhereScape. It outlines a future state architecture piloted with WhereScape, including the first implementations of a DataVault 2.0 data model and enterprise data model. It concludes that WhereScape helps provide a predictable, mature BI delivery process and framework for tool integration, while recommendations include creating a vision, using an architecture, building team skills, and automating with WhereScape.

Technology

Physical &Virtual DataVault 2.0 Automation
Erik van der Hoeven
17 October 2019
with

Agenda
• My journey
• Customer journey
• Future State Architecture
• WhereScape pilot
• First DV 2.0 implementation
• First EDM implementation
• Conclusion
• Recommendations

My journey of: Transforming your data into Wisdom!
2008 -2012
4 Kim
ball projects
Autom
ated pattern loading
2
DataVault 1.0 projects
DataVault 2.0Certificate
First DataVault 2.0 project
2013 2014 -2017 2018 2019 - present2018
First Big Data Project
Non Lean and mature process Lean and mature process
Erik van der Hoeven

Customer journey
What made this journey special
• Historization with DataVault 2.0
• Automation with WhereScape
• Tool integration
• Contract first
First EDM
im
plem
entation
04-2019
FutureStateArchitecture
Pilot
First DV
2.0 im
plem
entation
08-2019 - 09-201905-2019 10-2019

Future State Architecture
Source
Historised
Persistent Store
(RawVault)
Consumption
Data Delivery
Layer
Delivery
Transform
ations
Semantic Layer
Business
Transform
ations
Functional Tooling Database

PilotWhereScape
Define Test Score Launch
Architect Programmanager
Fin. Director
BI Team BI Team
Architect Programmanager
Fin. Director
Real value
Trust in the supplier
Trust in the BITeam
Loyalty from the BITeam +
Mature(2) dev team
Common understanding
Trust in the FSA
Foundation for future dev.

First DV 2.0 implementation
Solution Architecture
Savings
Mortgages
BizTalk Server
Azure
Integration
Services
PORTAL
Logging
Virtual
MDS
On Premise Cloud
Search
Customers
Persistent
RawVault
DataDelivery

First LDM implementation
Raw Vault: Data Vault Model
ODS Savings
Business Vault: EDM Based Data Model
ACBS MDS …
Data
Delivery
Layer
stg_SRC_SourceTable
Entity_S Entiy_L_EntityEntity_H
dv_CONS_Dimension
CONS_Dimension
dv_CONS_Fact
CONS_Fact
stg_SRC2_SourceTable
Virtual Staging
Staging
Raw Vault: Staging
Virtual Transformations + Business Rule Versioning
Persistent
Cached
Virtual
CACHED
Virtual
VirtualData Delivery: Consumer based model
Deployment Development
VirtualRawVault Base Views
Compress Timelines
Fix functional Timeline, Datatype Conversion, Attribute Naming
Determine Business Keys, Create SID
Lookup Reference Data, Split table, Calculate new Change Hash(es)
Union sources
dbo.SRC_SourceTable_H dbo.SRC_SourceTable_S dbo.SRC_SourceTable_L
bv_SRC_SourceTable_H bv_SRC_SourceTable_S bv_SRC_SourceTable_L
SQL Table Denodo Base View Denodo Interface View Denodo View

ConclusionWhereScape
• WhereScape helps us to have a predictable mature BI
Delivery Process
• It offers us an open framework where it helps us to
integrate with different tools
• It ensures that our Data Warehouse is fully auditable
and reliable (BCBS 239)
Pro
Con
• Integrating WhereScape with Denodo is not out of the
box.
• Open platform requires you to make choices
• Data & Model Driven

Recommendation
1. Create aVision for BI
2. Use an architecture to create a path
3. Build a loyal and skilful BITeam
4. UseWhereScape to automate your DataWarehouse
5. Aim high
6. Start now

What's hot

Data Vault 2.0 - Getting Started | Certus Solutions

Certus Solutions

Snowflake Company Presentation

AndrewJiang18

Banking organizations are having to deal with highly complex fraud rings today. Fraudsters are spreading their activities across a large number of transactions and geographical regions in a more coordinated fashion, making traditional anomaly identification methods nearly obsolete. Join us for this on-demand webinar with Jie Wu, Director of Product Marketing, DataStax and guest speaker Scott Heath, Chief Revenue Officer, Expero for a discussion on how to use real-time graph database to effectively detect fraud and reduce risk in real time. View recording: https://youtu.be/XzpTICtJ2Fk Explore all DataStax webinars: https://www.datastax.com/resources/webinars

Webinar - Fighting Bank Fraud with Real-time Graph Database

DataStax

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

Cathrine Wilhelmsen

Watch: https://bit.ly/3iZUf2o Data Virtualization has gone beyond its initial promise and is now becoming a critical component of an adaptive, agile enterprise data fabric. But there are common misconceptions around Data Virtualization and how it works. In this session, we will examine these misconceptions and set the record straight. According to Gartner, organizations with data virtualization will spend 40% less on building and managing data integration processes for connecting distributed data assets, and 60% of all organizations on track to implement some sort of data virtualization by 2022. This solidifies data virtualization as a critical piece of technology for modern data architecture. Agenda: - Why Data Virtualization? And why now? - How Data Virtualization can turbocharge your enterprise data strategy - Typical use cases about Data Virtualization - Demystifying the misconceptions about Data Virtualization Our speakers bring years of experience launching data strategies, and they will share success stories and lessons learned. Learn how you can enhance the competitive edge for your business in this webinar hosted by Orion Innovation and Denodo. We look forward to seeing you online.

Demystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy

Denodo

Google BigQuery is one of the largest, fastest, and most capable cloud data warehouses on the market. In this webinar, we review BigQuery best practices and show you how Matillion ETL can help you get the most out of the platform to gain a competitive edge. In this webinar: - Discover how to work quickly and efficiently with Google BigQuery - Find out the best ways to monitor and control costs - Hear tips and tricks for loading and transforming massive amounts of data in BigQuery with Matillion ETL - Get expert advice on improving your performance in BigQuery for quicker data analysis - Learn how to optimize BigQuery for your marketing analytics needs

Lets Talk Google BigQuery

Matillion

As companies grow, so does the volume of their data. Without the proper solutions in place to quickly store, measure and analyze that data, its usefulness quickly declines. See our latest webinar to learn about how companies are increasingly turning towards cloud-based data warehousing to derive more value out of their data and apply their findings to make smarter business decisions. The webinar covers core topics including: - The benefits of using Snowflake’s unique architecture for interacting with data. - How Matillion can help you quickly load and transform your data to maximize its value. - Expert advice on how to apply data warehousing and ETL best practices. Watch the full webinar: https://youtu.be/mIOm3j431OQ

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

Matillion

Webinar: Which Storage Architecture is Best for Splunk Analytics?

Storage Switzerland

Database migration services

Vito Rinaldi

Analytics in a Day Virtual Workshop

CCG

QVD Metadata Profile App in Qlik Sense

Syntrix Consulting Group

Everyone is moving their data to the cloud - but with all the different choices for a cloud based data warehouse, how do you know which one to choose? How do you know which warehouse will be the best and most flexible for your needs? In this webinar learn: - Why and what is a data warehouse - do you even need one? - The best criteria when evaluating which data warehouse to choose - What problems a data warehouse solves - What is “Big Data” and how does this provide business value - How Matillion can help you work with your data in your data warehouse of choice

How to Choose a Data Warehouse

Matillion

Data Architecture for Machine Learning

Denodo

Snowflake: The Good, the Bad, and the Ugly

Tyler Wishnoff

Disaster Recovery is an expensive proposition. But since the consequences of not being prepared for a disaster are so severe, it is an expense that organizations make. But that’s not to say organizations are not always looking for way to do DR better, faster and for less money. In this live webinar join Storage Switzerland and ClearSky to learn how organizations can lower the cost of DR preparation and execution.

Webinar: Cut Disaster Recovery Expenses – Improve Recovery Times

Storage Switzerland

Watch full webinar here: https://bit.ly/2ZVbLAY The Denodo Platform makes all data across the organization available to potentially everyone, internally and externally, in real time. But what does this mean for security? In fact, the Denodo Platform is built from the ground up to ensure that people can only access the data that they are authorized to see. But this is only the beginning. The Denodo Platform enables organizations to implement highly granular access policies that can be applied to roles, groups, or individual users, and those that can govern data sets, tables, or even individual cells. In this session, Vincent Fages-Gouyou, EMEA product management director at Denodo, takes you for a deep dive into the robust security capabilities of the Denodo Platform, which includes a short demo of the Denodo Platform in action. You will learn the ins and outs of: - The core Denodo security architecture - Policy-based protocols, which can govern access based on time of day, query threshold, and many other user-defined parameters - Role-based control, with row- and column-level masking - Tag-based security policies, enabling the definition of security policies in the semantic layer - Denodo Platform support for third-party security solutions

Secure Data Sharing with the Denodo Platform

Denodo

Walking Around the Data Lake

All Things Open

Elastic community Abidjan #225 meetup 08 May 2021

Yassine, LASRI

Big Data Storage Challenges and Solutions

WSO2

Watch full webinar here: https://buff.ly/2HMdbUp What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics. Attend this session to learn: • What data virtualization really is, • How it differs from other enterprise data integration technologies • Real-world examples of data virtualization in action from companies such as Logitech, Autodesk and Festo.

DAMA Webinar: Turn Grand Designs into a Reality with Data Virtualization

Denodo

What's hot (20)

Data Vault 2.0 - Getting Started | Certus Solutions

Snowflake Company Presentation

Webinar - Fighting Bank Fraud with Real-time Graph Database

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

Demystifying Data Virtualization: Why it’s Now Critical for Your Data Strategy

Lets Talk Google BigQuery

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

Webinar: Which Storage Architecture is Best for Splunk Analytics?

Database migration services

Analytics in a Day Virtual Workshop

QVD Metadata Profile App in Qlik Sense

How to Choose a Data Warehouse

Data Architecture for Machine Learning

Snowflake: The Good, the Bad, and the Ugly

Webinar: Cut Disaster Recovery Expenses – Improve Recovery Times

Secure Data Sharing with the Denodo Platform

Walking Around the Data Lake

Elastic community Abidjan #225 meetup 08 May 2021

Big Data Storage Challenges and Solutions

DAMA Webinar: Turn Grand Designs into a Reality with Data Virtualization

Similar to Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault Modelling and Data Governance conference on Oct. 17, 2019: Physical and Virtual Dstsa Vault 2.0 Automation with WhereScape at a Major Dutch Bank

Before the rise of Big Data, the Enterprise Data Warehouse (EDW) reigned supreme in Business Intelligence architecture. However, modern data rates and volumes often outstripped the capacity of traditional Data Warehousing tools and modeling strategies to keep pace. Many companies turned to unstructured Data Lakes as an means of keeping up with the influx. Consequently, they often discovered that the road from Data Lake to Business Intelligence was filled with its own steep challenges. As a result, any savings in terms of throughput and storage costs was more than offset by the high extraction and analytics costs of turning an unstructured Data Lake into an insights-yielding asset. Enter Data Vault 2.0, the Enterprise Data Warehouse reimagined to meet today’s data rate, volume and analytics demands. Not strictly an alternative to Data Lakes, Data Vault can easily integrate with your Data Lake and Big Data ingestion pipelines and analytics toolchain. This talk will introduce the fundamental concepts and advantages of Data Vault 2.0, and explain its approach to modeling data around your business domain’s “Hubs”, “Links” and “Satellites”. Finally, the talk will examine a real case study of building a Data Vault, including some challenges and drawbacks we encountered and addressed along the way.

Big Data or Data Warehousing? How to Leverage Both in the Enterprise

Dean Hallman

Data Vault 2.0: Big Data Meets Data Warehousing

All Things Open

Resume Vikram_S

Vikram Soundararajan

CV RCD- Eng

Ricardo Calderón Duana

You can find the webinar replay in the IDERA Resource Center: http://ow.ly/t10N50A4snJ Microsoft introduced Big Data Clusters at the recent Ignite conference. SQL Server Big Data Clusters only run on Kubernetes—a container orchestration platform that lets you run SQL Server on Linux, offering high availability and disaster recovery. While high availability and disaster recovery were a challenge on Linux in SQL Server 2017, Kubernetes makes this process painless and simplifies the overall deployment process. Join IDERA and Joey D’Antoni as he walks through how to deploy, configure, and maintain your SQL Server containers so they are available and meet your performance needs. Speaker: Joey D'Antoni is a Senior Consultant and SQL Server MVP with over a decade of experience working in both Fortune 500 and smaller firms. He is a Principal Architect for Denny Cherry and Associates and lives in Malvern, PA. He is a frequent speaker at major tech events, and blogger about all topics technology. He believes that no single platform is the answer to all technology problems. He holds a BS in Computer Information Systems from Louisiana Tech University and an MBA from North Carolina State University, and is the co-author of the Microsoft white paper "Using Power BI in a Hybrid Environment.

Geek Sync | Linux, Containers, and SQL Server—Get Ready for Big Data Clusters...

IDERA Software

Webinar presented live on August 8, 2017 The CSCC has published version 2.0 of Cloud Customer Architecture for Big Data & Analytics – a reference architecture that describes elements and components needed to support big data and analytics solutions using cloud computing. Version 2.0 of the architecture includes support for new use cases and cognitive computing. Big data analytics (BDA) and cloud computing are a top priority for CIOs. As cloud computing and big data technologies converge, they offer a cost-effective delivery model for cloud-based analytics. Many companies are experimenting with different cloud configurations to understand and refine requirements for their big data analytics solutions. This webinar will cover: - Business reasons to adopt cloud computing for big data and analytics capabilities - An architectural overview of a big data analytics solution in a cloud environment with a description of the capabilities offered by cloud providers - Proven architecture patterns that have been deployed in successful enterprise BDA projects The presentation draws from the CSCC's deliverable, Cloud Customer Architecture for Big Data and Analytics V2.0 http://www.cloud-council.org/deliverables/cloud-customer-architecture-for-big-data-and-analytics.htm Download the presentation deck here: http://www.cloud-council.org/webinars/cloud-customer-architecture-for-big-data-and-analytics-v2.htm

Cloud Customer Architecture for Big Data and Analytics V2.0

Cloud Standards Customer Council

datavault2.pptx

Mounika662749

Vikas_C_R_Summary_081116

Vikas Renukaradhya

Webinar on MongoDB BI Connectors

Sumit Sarkar

Personal Information (1) - Copy

Vinod Kanmantha

Maharshi_Amin_416

mamin1411

Cendien - Senior Cognos Developer & Consultant

Israel Denis

This presentation gives an overview of StreamCentral technology targeted for IT professionals. StreamCentral is software to model and build Big Data Solutions. StreamCentral consists of a Big Data Solutions Modeler that not only makes it easy to model traditional BI/DW and Big Data solutions but also auto deploys the model on the latest innovations in Big Data Management solutions (like HP Vertica and SQL Server Parallel Data Warehouse). StreamCentral Big Data Server executes the model definition in real-time. StreamCentral drastically reduces the time to market, risk and cost associated with building traditional BI/DW and Big Data solutions!

StreamCentral for the IT Professional

Raheel Retiwalla

IBM THINK 2020 - Cloud Data Lake with IBM Cloud Data Services

Torsten Steinbach

In this talk we share our experiences developing and deploying a microservices-based application. You will learn about the distributed data management challenges that arise in a microservices architecture. We will describe how we solved them using event sourcing to reliably publish events that drive eventually consistent workflows and pdate CQRS-based views. You will also learn how we build and deploy the application using a Jenkins-based deployment pipeline that creates Docker images that run on Amazon EC2.

Building and deploying microservices with event sourcing, CQRS and Docker (Ha...

Chris Richardson

Building event-driven Serverless Apps with Azure Functions and Azure Cosmos DB

Microsoft Tech Community

RichardChauvetShortResume2015

Richard Chauvet

Actively looking for new opportunity in Business Intelligence, Data Integration and Data Warehouse. Hands on experience in data analysis, providing ETL Solutions and building reports, dashboards and framework for business. Tools/Technologies: -Databases: SQL Server 2012, Teradata, MySQL. -Reporting Tools: Pentaho Report Designer, Tableau. -Dashboard Tool: Pentaho CDF, Pentaho CDE, Saiku. -ETL Tools: Pentaho PDI, SSIS -Scripting Languages: Python,UNIX Shell Scripting, Java script. -Cloud: AWS – S3, RDS, EC2, DMS, Glue , snowflake, Metallion

Abhishek jaiswal

Professional Portfolio

MoniqueO Opris

Optimizing Code Reusability for SharePoint using Linq to SharePoint & the MVP...

Sparkhound Inc.

More from Patrick Van Renterghem

Ethical AI at VDAB, presented by Vincent Buekenhout (Ethical AI Lead, VDAB) a...

Patrick Van Renterghem

Implementing error-proof, business-critical Machine Learning, presentation by...

Patrick Van Renterghem

Building Trust and Explainability into Chatbots: the Partena Ziekenfonds Busi...

Patrick Van Renterghem

AI & Ethics: The Belgian Industry Vision & Initiatives, presentation by Jelle...

Patrick Van Renterghem

Organisations need to make sure that they use AI in an appropriate way. Martijn and Hugo explain how to ensure that the developments are ethically sound and comply with regulations, how to have end-to-end governance, and how to address bias and fairness, interpretability and explainability, and robustness and security. During the conference, we looked at an example AI development process with focussing on the risks to be managed and the controls that can be established.

Responsible AI: An Example AI Development Process with Focus on Risks and Con...

Patrick Van Renterghem

Fairness and Transparency: Algorithmic Explainability, some Legal and Ethical...

Patrick Van Renterghem

How obedient digital twins and intelligent beings contribute to ethics and ex...

Patrick Van Renterghem

He Said, She Said: Finding and Fixing Bias in NLP (Natural Language Processin...

Patrick Van Renterghem

Introduction to Bias in Machine Learning, presented by Matthias Feys, CTO @ M...

Patrick Van Renterghem

Business Case: Ozitem Groupe, where 80% of the company is working remotely. R...

Patrick Van Renterghem

Digital Workplace Case Study: How the Municipality of Duffel successfully swi...

Patrick Van Renterghem

Unleashing the Full Potential of People, Teams and SOLVAY, presented by Bruce...

Patrick Van Renterghem

The Building Blocks of a Digital Workplace, presented by Sam Marshall at the ...

Patrick Van Renterghem

Engie's Digital Workplace and "Connecting the company" business case, present...

Patrick Van Renterghem

Face your communication challenges when implementing a digital workplace, bas...

Patrick Van Renterghem

The first steps in Recticel's Digital Workplace program by Kenneth Meuleman (...

Patrick Van Renterghem

Presentation by Dave Geentjens at the "Successful Digital Workplace Adoption"...

Patrick Van Renterghem

Tim scottkoenverheyenpresentation

Patrick Van Renterghem

The start of GDPR implementations in Europe was, for most organizations, also the start of rethinking their Data Warehouse strategy. The experience of past implementations gave a better view on the do's and don'ts. One of the important lessons learned was the approach of handling information quality. It's not something you handle on top of your data warehouse. To be successful, information quality goes hand in hand with your data warehouse implementation.

Presentation by Ivan Schotsmans (DV Community) at the Data Vault Modelling an...

Patrick Van Renterghem

Presentation by Luc Delanglez (DataLumen) at the Data Vault Modelling and Dat...

Patrick Van Renterghem

More from Patrick Van Renterghem (20)