SlideShare a Scribd company logo
1 of 11
Download to read offline
Big Data trap
francis@qmining.com
@fraka6
Data/Big Data Knowledge Action
People care about Knowledge/actions not data
Agenda
● Big data dilemma
● When are we doing Big Data?
● Maturity/Evolution steps
● The big data trap
● Optimal design = real time data-mining
● Increase your chances of success
The Big Data Dilemma
Big Data =
Data + IO bounded (disk)
CPU
<100%Data
IO bounded
QA
BI
Maturity
Barriers of entry Levels
Just another barrier of entry
Trap = no KPI
● No KPI -> batch processing -> big data
● KPI -> real time -> no big data complexity
Optimal design = real-time data-mining
● Events -> everything is an event
● + Rule -> create signal from events
● + KPIs -> selection of signals (top level)
● + Incident = signal static/dynamic thresholds
● + Root causes analysis
○ Bayesian inference (ratio signal)
○ Signal correlation (std signal)
○ Rule filtering (domain specific)
Increase chances of success
● Data driven culture
● Data quality culture (Avoid logs)
● Reach Analytics/BI level
● KISS
Recap
● Big Data = Small Data + IO bound
● Big data->Data->Analytics->Mining->Predictive
○ Data Quality = BIGGEST PROBLEM
○ Big Data = another barrier of entry
● Big data trap = no KPI
● KISS = real time data mining
hum...
Questions?
francis@qmining.com

More Related Content

Similar to Big data trap

The big data dead valley dilemma and much more.
The big data dead valley dilemma and much more.The big data dead valley dilemma and much more.
The big data dead valley dilemma and much more.
Francis Piéraut
 
Lean Digital | Data Driven Factory
Lean Digital | Data Driven FactoryLean Digital | Data Driven Factory
Lean Digital | Data Driven Factory
XL Groupe
 

Similar to Big data trap (20)

Big data: Just another barrier of entry
Big data: Just another barrier of entryBig data: Just another barrier of entry
Big data: Just another barrier of entry
 
Process mining: The role of Data in Business Processes
Process mining: The role of Data in Business ProcessesProcess mining: The role of Data in Business Processes
Process mining: The role of Data in Business Processes
 
10 ways to stumble with big data
10 ways to stumble with big data10 ways to stumble with big data
10 ways to stumble with big data
 
Dive into H2O: NYC
Dive into H2O: NYCDive into H2O: NYC
Dive into H2O: NYC
 
Pitfalls and pro-tips for effective and transparent Business Intelligence too...
Pitfalls and pro-tips for effective and transparent Business Intelligence too...Pitfalls and pro-tips for effective and transparent Business Intelligence too...
Pitfalls and pro-tips for effective and transparent Business Intelligence too...
 
The big data dead valley dilemma and much more.
The big data dead valley dilemma and much more.The big data dead valley dilemma and much more.
The big data dead valley dilemma and much more.
 
Dear Fiscal Board - Chief Data Officer shares thoughts and experiences with P...
Dear Fiscal Board - Chief Data Officer shares thoughts and experiences with P...Dear Fiscal Board - Chief Data Officer shares thoughts and experiences with P...
Dear Fiscal Board - Chief Data Officer shares thoughts and experiences with P...
 
QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing
QA Fest 2018. Slavik Pashanin. Analytics Data Driven TestingQA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing
QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing
 
How to succeed at data without even trying!
How to succeed at data without even trying!How to succeed at data without even trying!
How to succeed at data without even trying!
 
MLOps.pptx
MLOps.pptxMLOps.pptx
MLOps.pptx
 
A Journey Through Software
A Journey Through SoftwareA Journey Through Software
A Journey Through Software
 
The Big Bad Data
The Big Bad DataThe Big Bad Data
The Big Bad Data
 
Lean Digital | Data Driven Factory
Lean Digital | Data Driven FactoryLean Digital | Data Driven Factory
Lean Digital | Data Driven Factory
 
Moving from BI to AI : For decision makers
Moving from BI to AI : For decision makersMoving from BI to AI : For decision makers
Moving from BI to AI : For decision makers
 
Unified Operations Vision
Unified Operations VisionUnified Operations Vision
Unified Operations Vision
 
Getting Started with Big Data Analytics
Getting Started with Big Data AnalyticsGetting Started with Big Data Analytics
Getting Started with Big Data Analytics
 
Data driven testing - slavik pashanin (eng)
Data driven testing - slavik pashanin (eng)Data driven testing - slavik pashanin (eng)
Data driven testing - slavik pashanin (eng)
 
Winning in Today's Data-Centric Economy (Part 1)
Winning in Today's Data-Centric Economy (Part 1)Winning in Today's Data-Centric Economy (Part 1)
Winning in Today's Data-Centric Economy (Part 1)
 
Melt iron heterogeneous computing - lspe v3
Melt iron   heterogeneous computing - lspe v3Melt iron   heterogeneous computing - lspe v3
Melt iron heterogeneous computing - lspe v3
 
Big Data overview
Big Data overviewBig Data overview
Big Data overview
 

More from Francis Piéraut

More from Francis Piéraut (14)

4th industrial revolution fuel by combining big data and deeplearning a qui...
4th industrial revolution fuel by combining big data and deeplearning   a qui...4th industrial revolution fuel by combining big data and deeplearning   a qui...
4th industrial revolution fuel by combining big data and deeplearning a qui...
 
Startups ultime experience
Startups ultime experienceStartups ultime experience
Startups ultime experience
 
The ultimate trick to learn faster
The ultimate trick  to learn fasterThe ultimate trick  to learn faster
The ultimate trick to learn faster
 
ML_tools&libs-part1.pptx
ML_tools&libs-part1.pptxML_tools&libs-part1.pptx
ML_tools&libs-part1.pptx
 
ML_big_picture-2.0.pptx
ML_big_picture-2.0.pptxML_big_picture-2.0.pptx
ML_big_picture-2.0.pptx
 
Big data barrier of entry (flash)
Big data barrier of entry (flash) Big data barrier of entry (flash)
Big data barrier of entry (flash)
 
Appengine vs Amazon; pros &amp; cons for startups
Appengine vs Amazon; pros &amp; cons for startupsAppengine vs Amazon; pros &amp; cons for startups
Appengine vs Amazon; pros &amp; cons for startups
 
No BI without Machine Learning
No BI without Machine LearningNo BI without Machine Learning
No BI without Machine Learning
 
Java Empowered by Jython
Java Empowered by JythonJava Empowered by Jython
Java Empowered by Jython
 
easy_install digipy &amp; mlboost
easy_install digipy &amp; mlboosteasy_install digipy &amp; mlboost
easy_install digipy &amp; mlboost
 
Machine Learning empowered by Python April2009
Machine Learning empowered by Python April2009Machine Learning empowered by Python April2009
Machine Learning empowered by Python April2009
 
Intro to Machine Learning Enpowered by Python (Montreal Python)
Intro to Machine Learning Enpowered by Python (Montreal Python)Intro to Machine Learning Enpowered by Python (Montreal Python)
Intro to Machine Learning Enpowered by Python (Montreal Python)
 
Master Defense Slides (translated)
Master Defense Slides (translated)Master Defense Slides (translated)
Master Defense Slides (translated)
 
Soutenance 17 Avril 2003
Soutenance 17 Avril 2003Soutenance 17 Avril 2003
Soutenance 17 Avril 2003
 

Recently uploaded

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
ZurliaSoop
 
Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...
nirzagarg
 
Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...
Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...
Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...
Klinik kandungan
 
Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...
nirzagarg
 
怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制
怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制
怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制
vexqp
 
怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制
怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制
怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制
vexqp
 
怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制
怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制
怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制
vexqp
 
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi ArabiaIn Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
ahmedjiabur940
 
+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...
+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...
+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...
Health
 
PLE-statistics document for primary schs
PLE-statistics document for primary schsPLE-statistics document for primary schs
PLE-statistics document for primary schs
cnajjemba
 
Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...
Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...
Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...
gajnagarg
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样
一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样
一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样
wsppdmt
 
Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...
Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...
Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...
gajnagarg
 

Recently uploaded (20)

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
 
Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Purnia [ 7014168258 ] Call Me For Genuine Models We...
 
Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...
Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...
Jual obat aborsi Bandung ( 085657271886 ) Cytote pil telat bulan penggugur ka...
 
Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...
Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...
 
Capstone in Interprofessional Informatic // IMPACT OF COVID 19 ON EDUCATION
Capstone in Interprofessional Informatic  // IMPACT OF COVID 19 ON EDUCATIONCapstone in Interprofessional Informatic  // IMPACT OF COVID 19 ON EDUCATION
Capstone in Interprofessional Informatic // IMPACT OF COVID 19 ON EDUCATION
 
怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制
怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制
怎样办理伦敦大学毕业证(UoL毕业证书)成绩单学校原版复制
 
Sequential and reinforcement learning for demand side management by Margaux B...
Sequential and reinforcement learning for demand side management by Margaux B...Sequential and reinforcement learning for demand side management by Margaux B...
Sequential and reinforcement learning for demand side management by Margaux B...
 
怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制
怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制
怎样办理伦敦大学城市学院毕业证(CITY毕业证书)成绩单学校原版复制
 
怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制
怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制
怎样办理旧金山城市学院毕业证(CCSF毕业证书)成绩单学校原版复制
 
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi ArabiaIn Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
 
+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...
+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...
+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...
 
SR-101-01012024-EN.docx Federal Constitution of the Swiss Confederation
SR-101-01012024-EN.docx  Federal Constitution  of the Swiss ConfederationSR-101-01012024-EN.docx  Federal Constitution  of the Swiss Confederation
SR-101-01012024-EN.docx Federal Constitution of the Swiss Confederation
 
PLE-statistics document for primary schs
PLE-statistics document for primary schsPLE-statistics document for primary schs
PLE-statistics document for primary schs
 
Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...
Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...
Top profile Call Girls In dimapur [ 7014168258 ] Call Me For Genuine Models W...
 
The-boAt-Story-Navigating-the-Waves-of-Innovation.pptx
The-boAt-Story-Navigating-the-Waves-of-Innovation.pptxThe-boAt-Story-Navigating-the-Waves-of-Innovation.pptx
The-boAt-Story-Navigating-the-Waves-of-Innovation.pptx
 
Aspirational Block Program Block Syaldey District - Almora
Aspirational Block Program Block Syaldey District - AlmoraAspirational Block Program Block Syaldey District - Almora
Aspirational Block Program Block Syaldey District - Almora
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
 
Predicting HDB Resale Prices - Conducting Linear Regression Analysis With Orange
Predicting HDB Resale Prices - Conducting Linear Regression Analysis With OrangePredicting HDB Resale Prices - Conducting Linear Regression Analysis With Orange
Predicting HDB Resale Prices - Conducting Linear Regression Analysis With Orange
 
一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样
一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样
一比一原版(UCD毕业证书)加州大学戴维斯分校毕业证成绩单原件一模一样
 
Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...
Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...
Top profile Call Girls In Chandrapur [ 7014168258 ] Call Me For Genuine Model...
 

Big data trap

  • 2. Data/Big Data Knowledge Action People care about Knowledge/actions not data
  • 3. Agenda ● Big data dilemma ● When are we doing Big Data? ● Maturity/Evolution steps ● The big data trap ● Optimal design = real time data-mining ● Increase your chances of success
  • 4. The Big Data Dilemma
  • 5. Big Data = Data + IO bounded (disk) CPU <100%Data IO bounded
  • 6. QA BI Maturity Barriers of entry Levels Just another barrier of entry
  • 7. Trap = no KPI ● No KPI -> batch processing -> big data ● KPI -> real time -> no big data complexity
  • 8. Optimal design = real-time data-mining ● Events -> everything is an event ● + Rule -> create signal from events ● + KPIs -> selection of signals (top level) ● + Incident = signal static/dynamic thresholds ● + Root causes analysis ○ Bayesian inference (ratio signal) ○ Signal correlation (std signal) ○ Rule filtering (domain specific)
  • 9. Increase chances of success ● Data driven culture ● Data quality culture (Avoid logs) ● Reach Analytics/BI level ● KISS
  • 10. Recap ● Big Data = Small Data + IO bound ● Big data->Data->Analytics->Mining->Predictive ○ Data Quality = BIGGEST PROBLEM ○ Big Data = another barrier of entry ● Big data trap = no KPI ● KISS = real time data mining