Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way
Collecting Uncertain Data the Reactive Way

Check these out next

Apprentissage statistique et analyse prédictive en Python avec scikit-learn p...
Apprentissage statistique et analyse prédictive en Python avec scikit-learn p...
La Cuisine du Web
Recommender Systems with Apache Spark's ALS Function
Recommender Systems with Apache Spark's ALS Function
Will Johnson
Basic of python for data analysis
Basic of python for data analysis
Pramod Toraskar
Introduction to Machine Learning with SciKit-Learn
Introduction to Machine Learning with SciKit-Learn
Benjamin Bengfort
Next Generation Programming in R
Next Generation Programming in R
Florian Uhlitz
PPT ON MACHINE LEARNING by Ragini Ratre
PPT ON MACHINE LEARNING by Ragini Ratre
RaginiRatre
Java Collection framework
Java Collection framework
ankitgarg_er
Collections In Java
Collections In Java
Binoj T E
1 von 69

Collecting Uncertain Data the Reactive Way

11. Dec 2015••
Downloaden Sie, um offline zu lesen
Daten & Analysen

Related Resources ---------------------- Video (starts at 25 minutes): https://skillsmatter.com/skillscasts/7038-lightning-talks-2 From Scala Exchange 2015: https://skillsmatter.com/conferences/1948-scala-exchange-2014 Reactive Machine Learning: http://www.reactivemachinelearning.com/ Data Engineering blogging: https://medium.com/data-engineering Talk Summary ----------------- Before you can ever get started building large-scale data analytic systems, you need to start with one crucial element: data. Collecting data, especially collecting lots of data, is harder than it seems. Data ingested with the wrong data model can be worse than no data at all. A data collection system that is too slow can bring an entire platform grinding to a halt. Don't panic! Scalable, non-destructive data collection is possible. This talk will focus on strategies for data collection based on real world experience building large scale machine learning systems. It will introduce ideas from the emerging paradigm of reactive machine learning that are based on older ideas about immutable facts and pervasive, intrinsic uncertainty.

Jeff Smith
Designing machine learning systems that scale

Recomendados

Introducing Reactive Machine LearningIntroducing Reactive Machine Learning
Introducing Reactive Machine LearningJeff Smith2.4K Aufrufe•81 Folien
Spark for Reactive Machine Learning: Building Intelligent Agents at ScaleSpark for Reactive Machine Learning: Building Intelligent Agents at Scale
Spark for Reactive Machine Learning: Building Intelligent Agents at ScaleJeff Smith1.2K Aufrufe•67 Folien
Reactive Learning AgentsReactive Learning Agents
Reactive Learning AgentsJeff Smith1.3K Aufrufe•96 Folien
Unafraid of Change: Optimizing ETL, ML, and AI in Fast-Paced Environments wit...Unafraid of Change: Optimizing ETL, ML, and AI in Fast-Paced Environments wit...
Unafraid of Change: Optimizing ETL, ML, and AI in Fast-Paced Environments wit...Databricks516 Aufrufe•28 Folien
Machine Learning - Dataset PreparationMachine Learning - Dataset Preparation
Machine Learning - Dataset PreparationAndrew Ferlitsch1.9K Aufrufe•16 Folien
Is there a perfect data-parallel programming language? (Experiments with More...Is there a perfect data-parallel programming language? (Experiments with More...
Is there a perfect data-parallel programming language? (Experiments with More...Julian Hyde366 Aufrufe•97 Folien

Más contenido relacionado

Presentaciones para ti(20)

Apprentissage statistique et analyse prédictive en Python avec scikit-learn p...Apprentissage statistique et analyse prédictive en Python avec scikit-learn p...
Apprentissage statistique et analyse prédictive en Python avec scikit-learn p...
La Cuisine du Web•4.9K Aufrufe
Recommender Systems with Apache Spark's ALS FunctionRecommender Systems with Apache Spark's ALS Function
Recommender Systems with Apache Spark's ALS Function
Will Johnson•24K Aufrufe
Basic of python for data analysisBasic of python for data analysis
Basic of python for data analysis
Pramod Toraskar•339 Aufrufe
Introduction to Machine Learning with SciKit-LearnIntroduction to Machine Learning with SciKit-Learn
Introduction to Machine Learning with SciKit-Learn
Benjamin Bengfort•16.8K Aufrufe
Next Generation Programming in RNext Generation Programming in R
Next Generation Programming in R
Florian Uhlitz•1.7K Aufrufe
PPT ON MACHINE LEARNING by Ragini RatrePPT ON MACHINE LEARNING by Ragini Ratre
PPT ON MACHINE LEARNING by Ragini Ratre
RaginiRatre•262 Aufrufe
Java Collection frameworkJava Collection framework
Java Collection framework
ankitgarg_er•14.2K Aufrufe
Collections In JavaCollections In Java
Collections In Java
Binoj T E•9.6K Aufrufe
5 collection framework5 collection framework
5 collection framework
Minal Maniar•2.6K Aufrufe
07 Retrieving Objects07 Retrieving Objects
07 Retrieving Objects
Ranjan Kumar•391 Aufrufe
Data preprocessing for Machine Learning with R and PythonData preprocessing for Machine Learning with R and Python
Data preprocessing for Machine Learning with R and Python
Akhilesh Joshi•643 Aufrufe
Java collectionJava collection
Java collection
Arati Gadgil•2.4K Aufrufe
Machine Learning in RMachine Learning in R
Machine Learning in R
Alexandros Karatzoglou•16.2K Aufrufe
Java - Collections frameworkJava - Collections framework
Java - Collections framework
Riccardo Cardin•5.2K Aufrufe
Towards Parallel Nonmonotonic Reasoning with Billions of FactsTowards Parallel Nonmonotonic Reasoning with Billions of Facts
Towards Parallel Nonmonotonic Reasoning with Billions of Facts
PlanetData Network of Excellence•513 Aufrufe
Java collections conceptJava collections concept
Java collections concept
kumar gaurav•6.8K Aufrufe
Text Classification in Python – using Pandas, scikit-learn, IPython Notebook ...Text Classification in Python – using Pandas, scikit-learn, IPython Notebook ...
Text Classification in Python – using Pandas, scikit-learn, IPython Notebook ...
Jimmy Lai•29.6K Aufrufe
Data Structures for Statistical Computing in PythonData Structures for Statistical Computing in Python
Data Structures for Statistical Computing in Python
Wes McKinney•89K Aufrufe
Collections in Java NotesCollections in Java Notes
Collections in Java Notes
Shalabh Chaudhary•1.4K Aufrufe
Data Mining: Concepts and Techniques_ Chapter 6: Mining Frequent Patterns, ...Data Mining: Concepts and Techniques_ Chapter 6: Mining Frequent Patterns, ...
Data Mining: Concepts and Techniques_ Chapter 6: Mining Frequent Patterns, ...
Salah Amean•7.5K Aufrufe

Similar a Collecting Uncertain Data the Reactive Way(20)

Machine LearningMachine Learning
Machine Learning
butest•381 Aufrufe
Machine LearningMachine Learning
Machine Learning
butest•1.4K Aufrufe
Intrusion DetectionIntrusion Detection
Intrusion Detection
butest•925 Aufrufe
Data mining Data mining
Data mining
Jhadesunil•221 Aufrufe
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John MelonakosPT-4054, "OpenCL™ Accelerated Compute Libraries" by John Melonakos
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John Melonakos
AMD Developer Central •1.6K Aufrufe
Machine Learning and Real-World ApplicationsMachine Learning and Real-World Applications
Machine Learning and Real-World Applications
MachinePulse•22.9K Aufrufe
Numpy Talk at SIAMNumpy Talk at SIAM
Numpy Talk at SIAM
Enthought, Inc.•4.2K Aufrufe
Supervised vs unsupervised learning - infographicSupervised vs unsupervised learning - infographic
Supervised vs unsupervised learning - infographic
Intellspot•726 Aufrufe
Machine Learning : why we should know and how it worksMachine Learning : why we should know and how it works
Machine Learning : why we should know and how it works
Kevin Lee•128 Aufrufe
Presentation of Visual TrackingPresentation of Visual Tracking
Presentation of Visual Tracking
Yu-Sheng (Yosen) Chen•1.6K Aufrufe
PPTPPT
PPT
butest•848 Aufrufe
Fusing semantic dataFusing semantic data
Fusing semantic data
Andriy Nikolov•426 Aufrufe
Introduction to Deep Learning and neon at GalvanizeIntroduction to Deep Learning and neon at Galvanize
Introduction to Deep Learning and neon at Galvanize
Intel Nervana•1.4K Aufrufe
Towards Efficient Privacy-preserving Image Feature Extraction in Cloud ComputingTowards Efficient Privacy-preserving Image Feature Extraction in Cloud Computing
Towards Efficient Privacy-preserving Image Feature Extraction in Cloud Computing
Si Chen•1.7K Aufrufe
International Journal of Computational Engineering Research(IJCER)International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
ijceronline•73 Aufrufe
Lecture-1-Introduction-to-Data-Mining.pdfLecture-1-Introduction-to-Data-Mining.pdf
Lecture-1-Introduction-to-Data-Mining.pdf
Jojo314349•5 Aufrufe
I DunderstnI Dunderstn
I Dunderstn
hajaanwar•578 Aufrufe
Bayesian Network 을 활용한 예측 분석Bayesian Network 을 활용한 예측 분석
Bayesian Network 을 활용한 예측 분석
datasciencekorea•8.7K Aufrufe
Nimrita koul Machine LearningNimrita koul Machine Learning
Nimrita koul Machine Learning
Nimrita Koul•271 Aufrufe
What the Brain says about Machine Intelligence What the Brain says about Machine Intelligence
What the Brain says about Machine Intelligence
Numenta•11.5K Aufrufe

Más de Jeff Smith(13)

Questioning Conversational AIQuestioning Conversational AI
Questioning Conversational AI
Jeff Smith•183 Aufrufe
Neuroevolution in ElixirNeuroevolution in Elixir
Neuroevolution in Elixir
Jeff Smith•139 Aufrufe
Tools for Making Machine Learning more ReactiveTools for Making Machine Learning more Reactive
Tools for Making Machine Learning more Reactive
Jeff Smith•133 Aufrufe
Building Learning AgentsBuilding Learning Agents
Building Learning Agents
Jeff Smith•432 Aufrufe
Reactive for Machine Learning TeamsReactive for Machine Learning Teams
Reactive for Machine Learning Teams
Jeff Smith•423 Aufrufe
Reactive Machine Learning On and Beyond the JVMReactive Machine Learning On and Beyond the JVM
Reactive Machine Learning On and Beyond the JVM
Jeff Smith•1.4K Aufrufe
Bringing Data Scientists and Engineers TogetherBringing Data Scientists and Engineers Together
Bringing Data Scientists and Engineers Together
Jeff Smith•990 Aufrufe
Characterizing Intelligence with ElixirCharacterizing Intelligence with Elixir
Characterizing Intelligence with Elixir
Jeff Smith•573 Aufrufe
Reactive Machine Learning and Functional ProgrammingReactive Machine Learning and Functional Programming
Reactive Machine Learning and Functional Programming
Jeff Smith•2.2K Aufrufe
Huhdoop?: Uncertain Data Management on Non-Relational Database SystemsHuhdoop?: Uncertain Data Management on Non-Relational Database Systems
Huhdoop?: Uncertain Data Management on Non-Relational Database Systems
Jeff Smith•5.9K Aufrufe
Breadth or Depth: What's in a column-store?Breadth or Depth: What's in a column-store?
Breadth or Depth: What's in a column-store?
Jeff Smith•1.1K Aufrufe
Save the server, Save the worldSave the server, Save the world
Save the server, Save the world
Jeff Smith•864 Aufrufe
NoSQL in PerspectiveNoSQL in Perspective
NoSQL in Perspective
Jeff Smith•889 Aufrufe

Último(20)

신논현오피【ØPSS07쩜CØM】오피쓰 신논현방문하기 ®신논현휴게텔ꗝ신논현오피≫신논현안마신논현오피【ØPSS07쩜CØM】오피쓰 신논현방문하기 ®신논현휴게텔ꗝ신논현오피≫신논현안마
신논현오피【ØPSS07쩜CØM】오피쓰 신논현방문하기 ®신논현휴게텔ꗝ신논현오피≫신논현안마
pieliedie88•0 Aufrufe
목포오피【opss07ㆍ컴】오피쓰 목포정보→목포휴게텔ꔟ목포오피‣목포안마목포오피【opss07ㆍ컴】오피쓰 목포정보→목포휴게텔ꔟ목포오피‣목포안마
목포오피【opss07ㆍ컴】오피쓰 목포정보→목포휴게텔ꔟ목포오피‣목포안마
pieliedie88•2 Aufrufe
les six sigma.pdfles six sigma.pdf
les six sigma.pdf
SofianeIdjer•0 Aufrufe
Distributed Data Across Cloud and On-Premises: Opportunities and ChallengesDistributed Data Across Cloud and On-Premises: Opportunities and Challenges
Distributed Data Across Cloud and On-Premises: Opportunities and Challenges
Denodo •0 Aufrufe
온라인홀덤top7588⬤com⊐ *추천인: AAKK* 온라인홀덤top7588⬤com⊐ *추천인: AAKK*
온라인홀덤top7588⬤com⊐ *추천인: AAKK*
yqwggcy3463•2 Aufrufe
신당오피【OPSS07。COM】오피쓰 신당오피사이트 ✮신당휴게텔ꖫ신당오피❉신당안마신당오피【OPSS07。COM】오피쓰 신당오피사이트 ✮신당휴게텔ꖫ신당오피❉신당안마
신당오피【OPSS07。COM】오피쓰 신당오피사이트 ✮신당휴게텔ꖫ신당오피❉신당안마
pieliedie88•3 Aufrufe
ꔘtop7588⬤com텍사스홀덤 【추천인: AAKK】ꔘtop7588⬤com텍사스홀덤 【추천인: AAKK】
ꔘtop7588⬤com텍사스홀덤 【추천인: AAKK】
yqwggcy3463•2 Aufrufe
sentiment analysis.pdfsentiment analysis.pdf
sentiment analysis.pdf
manju451965•0 Aufrufe
의정부대박【OPSS07。COM】오피쓰≸의정부오피 의정부안마ナ의정부오피ꕘ의정부건마ナ의정부오피의정부대박【OPSS07。COM】오피쓰≸의정부오피 의정부안마ナ의정부오피ꕘ의정부건마ナ의정부오피
의정부대박【OPSS07。COM】오피쓰≸의정부오피 의정부안마ナ의정부오피ꕘ의정부건마ナ의정부오피
pieliedie88•3 Aufrufe
#Vashikaran Specialist in lahore #taweez #Vashikaran Specialist in lahore #taweez
#Vashikaran Specialist in lahore #taweez
KaleIlamWaliAmila•0 Aufrufe
Unproductive data analysis_v7.pptxUnproductive data analysis_v7.pptx
Unproductive data analysis_v7.pptx
ssuser9d6cab•0 Aufrufe
화성op【OpSS07。cØm】화성오피ꔑ오피쓰ノ화성건마ノ화성마사지☢화성오피화성op【OpSS07。cØm】화성오피ꔑ오피쓰ノ화성건마ノ화성마사지☢화성오피
화성op【OpSS07。cØm】화성오피ꔑ오피쓰ノ화성건마ノ화성마사지☢화성오피
pieliedie88•2 Aufrufe
Site Visit to 10 MTPA Dolvi Works.pdfSite Visit to 10 MTPA Dolvi Works.pdf
Site Visit to 10 MTPA Dolvi Works.pdf
VikrantKaswan2•1 Aufruf
노원oP【OPSS07。COM】노원오피ꖱ오피쓰 ➭노원건마 ➭노원마사지⭊노원오피노원oP【OPSS07。COM】노원오피ꖱ오피쓰 ➭노원건마 ➭노원마사지⭊노원오피
노원oP【OPSS07。COM】노원오피ꖱ오피쓰 ➭노원건마 ➭노원마사지⭊노원오피
pieliedie88•3 Aufrufe
Bridging Data Gaps with a Solid Data Foundation - A Key Imperative for Today’...Bridging Data Gaps with a Solid Data Foundation - A Key Imperative for Today’...
Bridging Data Gaps with a Solid Data Foundation - A Key Imperative for Today’...
Denodo •0 Aufrufe
Employment in media industry.pptxEmployment in media industry.pptx
Employment in media industry.pptx
MollyBrown86•0 Aufrufe
SIZING-OF-FACILITIES-Model.pdfSIZING-OF-FACILITIES-Model.pdf
SIZING-OF-FACILITIES-Model.pdf
KarlBelleza1•0 Aufrufe
Importance of a Logical First Architecture in a Cloud First Data LandscapeImportance of a Logical First Architecture in a Cloud First Data Landscape
Importance of a Logical First Architecture in a Cloud First Data Landscape
Denodo •0 Aufrufe
ꔊtop7588⬤com텍사스홀덤 【추천인: AAKK】ꔊtop7588⬤com텍사스홀덤 【추천인: AAKK】
ꔊtop7588⬤com텍사스홀덤 【추천인: AAKK】
yqwggcy3463•0 Aufrufe
SOCIAL-INEQUALITY.pptxSOCIAL-INEQUALITY.pptx
SOCIAL-INEQUALITY.pptx
JuzGeronimo•0 Aufrufe

Collecting Uncertain Data the Reactive Way

  1. Collecting Uncertain Data the Reactive Way Jeff Smith @jeffksmithjr
  2. x.ai is a personal assistant who schedules meetings for you
  3. Reactive Machine Learning
  4. Machine Learning Systems
  5. Machine Learning Systems
  6. Machine Learning Systems
  7. Traits of Reactive Systems
  8. Traits of Reactive Systems
  9. Reactive Strategies
  10. Reactive Strategies
  11. Reactive Machine Learning
  12. Reactive Machine Learning
  13. Reactive Machine Learning
  14. Collecting Data
  15. What’s for dinner?
  16. Reactive Data Collection
  17. Modeling Uncertain Data
  18. Certain Data Model case class ZebraReading(sensorId: Int, locationId: Int, timestamp: Long, count: Int)
  19. Uncertainty Interval 27 33
  20. Uncertain Data Model case class PreyReading(sensorId: Int, locationId: Int, timestamp: Long, animalsLowerBound: Double, animalsUpperBound: Double, percentZebras: Double)
  21. Scaling Data Collection
  22. Simple Data Architecture
  23. Simple Data Architecture
  24. Mutable State case class Region(id: Int) import collection.mutable.HashMap var densities = new HashMap[Region, Double]() densities.put(Region(4), 52.4)
  25. Scaling with Queues
  26. Scaling with Queues
  27. Out of Order Updates
  28. Out of Order Updates densities.put(Region(6), 73.6) densities.put(Region(6), 0.5) densities.get(Region(6)).get
  29. Out of Order Updates densities.put(Region(6), 73.6) densities.put(Region(6), 0.5) densities.get(Region(6)).get densities.put(Region(6), 0.5) densities.put(Region(6), 73.6) densities.get(Region(6)).get
  30. Concurrent Collections import collection.mutable._ var synchronizedDensities = new LinkedHashMap[Region, Double]() with SynchronizedMap[Region, Double]
  31. Scaling with Locks
  32. Scaling with Locks
  33. Immutable Facts case class PreyReading(sensorId: Int, locationId: Int, timestamp: Long, animalsLowerBound: Double, animalsUpperBound: Double, percentZebras: Double) implicit val preyReadingFormatter = Json.format[PreyReading]
  34. Immutable Facts val reading = PreyReading(36, 12, currentTimeMillis(), 12.0, 18.0, 0.60) val setDoc = bucket.set[PreyReading](readingId(reading), reading)
  35. Scaling with Distributed Databases
  36. Scaling with Distributed Databases
  37. Handling Incomplete Data
  38. Distributed Data Storage
  39. Querying Complete Data (bucket.searchValues[PreyReading]("prey", "by_sensor_id") (new Query().setIncludeDocs(true))) .enumerate.apply(Iteratee.foreach { doc => println(s"Prey Reading: $doc")})
  40. Complete Data
  41. Partition Tolerance
  42. Partition Tolerance
  43. Partition Tolerance
  44. Partition Tolerance
  45. Querying Incomplete Data (bucket.searchValues[PreyReading]("prey", "by_sensor_id") (new Query().setIncludeDocs(true))) .enumerate.apply(Iteratee.foreach { doc => println(s"Prey Reading: $doc")})
  46. Incomplete Data
  47. Incomplete Data
  48. Reactive Data Collection
  49. For Later
  50. reactivemachinelearning.com medium.com/data-engineering M A N N I N G Jeff Smith
  51. x.ai @xdotai hello@human.x.ai New York, New York
  52. skillsmatter.com/conferences/ 6862-scala-exchange-2015#skillscasts
  53. Thank You
  54. Collecting Uncertain Data the Reactive Way Jeff Smith @jeffksmithjr