SlideShare ist ein Scribd-Unternehmen logo
1 von 46
ANALYZING LARGE-SCALE USER DATA

                    SPEAKER: Aaron Kimball
                             CTO
                             WibiData


Friday, July 27, 2012
Friday, July 27, 2012
Analyzing	
  Large-­‐Scale	
  User	
  Data
                      with	
  Hadoop	
  and	
  HBase

                        Aaron	
  Kimball	
  –	
  CTO



                                                       WibiData,	
  
Friday, July 27, 2012
We	
  can	
  now	
  collect	
  
                        more	
  data	
  than	
  at	
  
                        any	
  Dme	
  in	
  history.


Friday, July 27, 2012
Yesterday’s	
  engineering	
  
    challenge:	
  FiJng	
  the	
  
    problem	
  into	
  the	
  
    hardware.
Friday, July 27, 2012
Today’s	
  constrained	
  
               resource	
  is	
  
               understanding.

Friday, July 27, 2012
How	
  do	
  we	
  best	
  apply	
  
          data




                          …to	
  beMer	
  serving	
  our	
  
Friday, July 27, 2012
The	
  best	
  products	
  are	
  user-­‐
         • IntuiDve	
  UI
         • ConDnuously	
  learning
                   – Guided	
  search
                   – Smarter	
  recommenda1ons
         • More	
  effec1ve	
  service



Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
What	
  are	
  we	
  building	
  




Friday, July 27, 2012
Requirements




                 1.	
  Understand	
  the	
  user	
  
                 populaDon
Friday, July 27, 2012
Requirements

                        2.	
  Respond	
  to	
  
                              users	
  in	
  real	
  
                              Dme



Friday, July 27, 2012
Requirements




                 3.	
  Support	
  graceful	
  data	
  
                 evoluDon
Friday, July 27, 2012
Large-­‐scale	
  data	
  science	
  is	
  
         • What	
  does	
  a	
  user	
  look	
  like?
                   – What	
  data	
  is	
  available	
  about	
  the	
  user?
                   – Which	
  features	
  are	
  important?
                   – Which	
  features	
  are	
  correlated?
         • How	
  do	
  I	
  model	
  this	
  in	
  MapReduce?
         • How	
  do	
  I	
  serve	
  results	
  in	
  a	
  Dmely	
  

Friday, July 27, 2012
Friday, July 27, 2012
Tools	
  of	
  the	
  trade
         • Store	
  all	
  data	
  about	
  a	
  
           user	
  in	
  one	
  place
         • Support	
  real-­‐Dme	
  
           get/put,	
  as	
  well	
  as	
  
           MapReduce



Friday, July 27, 2012
Tools	
  of	
  the	
  trade
                        • Use	
  complex	
  data	
  
                          types	
  to	
  model	
  
                          complex	
  data
                        • Support	
  extended	
  
                          data	
  models	
  over	
  
                          Dme

Friday, July 27, 2012
Tools	
  of	
  the	
  trade
         • Abstract	
  computaDonal	
  
           model	
  away	
  from	
  
           MapReduce
         • Support	
  computaDon	
  
           over	
  all	
  users…	
  or	
  one	
  
           user	
  at	
  a	
  Dme

Friday, July 27, 2012
Friday, July 27, 2012
Friday, July 27, 2012
Friday, July 27, 2012
Friday, July 27, 2012
Friday, July 27, 2012
Friday, July 27, 2012
Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes



                                                Viewing/recording	
  
                                                history




Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes



                                                Viewing/recording	
  
                                                history




Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes


                                                                                                                                        	
  	
  	
  	
  	
  Libraries
                                                                                                                                        Device	
  and	
  User	
  Analysis



                                                Viewing/recording	
  
                                                history

                                                Personalized	
  offers	
  
                                                       and	
  
                                                recommenda=ons




Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes


                                                                                                                                        	
  	
  	
  	
  	
  Libraries
                                                                                                                                        Device	
  and	
  User	
  Analysis



                                                Viewing/recording	
  
                                                history

                                                Personalized	
  offers	
  
                                                       and	
  
                                                recommenda=ons




Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes


                                                                                                                                        	
  	
  	
  	
  	
  Libraries
                                                                                                                                        Device	
  and	
  User	
  Analysis



                                                Viewing/recording	
  
                                                history

                                                Personalized	
  offers	
  
                                                       and	
  
                                                recommenda=ons



                                                  Analysis	
  for	
  
                                                    product	
  
                                                   roadmap
Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes


                                                                                                                                        	
  	
  	
  	
  	
  Libraries
                                                                                                                                        Device	
  and	
  User	
  Analysis



                                                Viewing/recording	
  
                                                history

                                                Personalized	
  offers	
  
                                                       and	
  
                                                recommenda=ons



                                                  Analysis	
  for	
  
                                                    product	
  
                                                   roadmap
Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes


                                                                                                                                        	
  	
  	
  	
  	
  Libraries
                                                                                                                                        Device	
  and	
  User	
  Analysis



                                                Viewing/recording	
  
                                                history

                                                Personalized	
  offers	
  
                                                       and	
  
                                                recommenda=ons



                                                  Analysis	
  for	
  
                                                    product	
                                               Tech	
  support	
  
                                                   roadmap                                                     portal
Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes


                                                                                                                                        	
  	
  	
  	
  	
  Libraries
                                                                                                                                        Device	
  and	
  User	
  Analysis



                                                Viewing/recording	
  
                                                history

                                                Personalized	
  offers	
  
                                                       and	
  
                                                recommenda=ons



                                                  Analysis	
  for	
  
                                                    product	
                                               Tech	
  support	
  
                                                   roadmap                                                     portal
Friday, July 27, 2012
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  :	
  for	
  set-­‐top	
  boxes


                                                                                                                                        	
  	
  	
  	
  	
  Libraries
                                                                                                                                        Device	
  and	
  User	
  Analysis



                                                Viewing/recording	
  
                                                history

                                                Personalized	
  offers	
  
                                                       and	
  
                                                recommenda=ons


                                                                                                                                                              Improve
                                                  Analysis	
  for	
  
                                                                                                                                                              d	
  reports	
  
                                                    product	
                                               Tech	
  support	
  
                                                                                                                                                              for	
  
                                                   roadmap                                                     portal
Friday, July 27, 2012                                                                                                                                         adver=se
The	
  future
         • More	
  personalizaDon
         • AdapDve	
  UIs	
  (self	
  arranging	
  
           dashboards)
         • Targeted	
  content,	
  ads
         • More	
  effecDve	
  customer	
  service


Friday, July 27, 2012
Conclusions
         • ApplicaDons	
  are	
  becoming	
  
           increasingly	
  
           user-­‐centric
         • Data	
  drives	
  this	
  capability,	
  but	
  
           harnessing	
  it	
  requires	
  a	
  new	
  
           distributed	
  architecture

Friday, July 27, 2012
www.wibidata.com	
  /	
  
                        Aaron	
  Kimball	
  –	
  aaron@wibidata.com




Friday, July 27, 2012
Friday, July 27, 2012

Weitere ähnliche Inhalte

Ähnlich wie ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012

Best Practices - Seeqnce - 23/24-02-2012
Best Practices - Seeqnce - 23/24-02-2012Best Practices - Seeqnce - 23/24-02-2012
Best Practices - Seeqnce - 23/24-02-2012
Youssef Chaker
 
Alabfi em-20120624
Alabfi em-20120624Alabfi em-20120624
Alabfi em-20120624
zepheiraorg
 
E assessment - ZAWF at Aberdeen Workshop
 E assessment - ZAWF at Aberdeen Workshop E assessment - ZAWF at Aberdeen Workshop
E assessment - ZAWF at Aberdeen Workshop
Digit Class
 
Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)
Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)
Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)
Rick. Bahague
 
My fire st petersburg 27 june 2012 (d hladky)
My fire st petersburg 27 june 2012 (d hladky)My fire st petersburg 27 june 2012 (d hladky)
My fire st petersburg 27 june 2012 (d hladky)
AI4BD GmbH
 
Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...
Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...
Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...
Vito Ostuni
 

Ähnlich wie ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012 (20)

An Analytics Toolkit Tour
An Analytics Toolkit TourAn Analytics Toolkit Tour
An Analytics Toolkit Tour
 
THE TRILLION ROW SPREADSHEET(tm) from Structure:Data 2012
THE TRILLION ROW SPREADSHEET(tm) from Structure:Data 2012THE TRILLION ROW SPREADSHEET(tm) from Structure:Data 2012
THE TRILLION ROW SPREADSHEET(tm) from Structure:Data 2012
 
SPONSORED WORKSHOP by Amplidata from Structure:Data 2012:
SPONSORED WORKSHOP by Amplidata from Structure:Data 2012:  SPONSORED WORKSHOP by Amplidata from Structure:Data 2012:
SPONSORED WORKSHOP by Amplidata from Structure:Data 2012:
 
BIG DATA: AN AUGMENTED INTELLIGENCE FOR STRATEGIC DECISION MAKING from Struct...
BIG DATA: AN AUGMENTED INTELLIGENCE FOR STRATEGIC DECISION MAKING from Struct...BIG DATA: AN AUGMENTED INTELLIGENCE FOR STRATEGIC DECISION MAKING from Struct...
BIG DATA: AN AUGMENTED INTELLIGENCE FOR STRATEGIC DECISION MAKING from Struct...
 
Lecture 4: Social Web Personalization (2012)
Lecture 4: Social Web Personalization (2012)Lecture 4: Social Web Personalization (2012)
Lecture 4: Social Web Personalization (2012)
 
Best Practices - Seeqnce - 23/24-02-2012
Best Practices - Seeqnce - 23/24-02-2012Best Practices - Seeqnce - 23/24-02-2012
Best Practices - Seeqnce - 23/24-02-2012
 
REALIZING REAL-TIME VALUE ON THE REAL-TIME WEB from Structure:Data 2012
REALIZING REAL-TIME VALUE ON THE REAL-TIME WEB from Structure:Data 2012REALIZING REAL-TIME VALUE ON THE REAL-TIME WEB from Structure:Data 2012
REALIZING REAL-TIME VALUE ON THE REAL-TIME WEB from Structure:Data 2012
 
Alabfi em-20120624
Alabfi em-20120624Alabfi em-20120624
Alabfi em-20120624
 
E assessment - ZAWF at Aberdeen Workshop
 E assessment - ZAWF at Aberdeen Workshop E assessment - ZAWF at Aberdeen Workshop
E assessment - ZAWF at Aberdeen Workshop
 
eXo Software Factory Overview
eXo Software Factory OvervieweXo Software Factory Overview
eXo Software Factory Overview
 
Enhancing AT through ID Techniques
Enhancing AT through ID TechniquesEnhancing AT through ID Techniques
Enhancing AT through ID Techniques
 
THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUMETHE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
 
Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)
Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)
Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)
 
Enhancing AT through ID techniques handouts
Enhancing AT through ID techniques handoutsEnhancing AT through ID techniques handouts
Enhancing AT through ID techniques handouts
 
Final Year Project Guidance
Final Year Project GuidanceFinal Year Project Guidance
Final Year Project Guidance
 
My fire st petersburg 27 june 2012 (d hladky)
My fire st petersburg 27 june 2012 (d hladky)My fire st petersburg 27 june 2012 (d hladky)
My fire st petersburg 27 june 2012 (d hladky)
 
Guerrilla Usability Testing for Agile/Lean
Guerrilla Usability Testing for Agile/LeanGuerrilla Usability Testing for Agile/Lean
Guerrilla Usability Testing for Agile/Lean
 
The state of drupal 8 - Drupalcamp Gent
The state of drupal 8  - Drupalcamp GentThe state of drupal 8  - Drupalcamp Gent
The state of drupal 8 - Drupalcamp Gent
 
Choosing a backend for your mobile app? Don’t roll the dice!
Choosing a backend for your mobile app? Don’t roll the dice!Choosing a backend for your mobile app? Don’t roll the dice!
Choosing a backend for your mobile app? Don’t roll the dice!
 
Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...
Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...
Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...
 

Mehr von Gigaom

Mehr von Gigaom (20)

Structure 2014 - The strategic value of the cloud - Joe Weinman
Structure 2014 - The strategic value of the cloud - Joe WeinmanStructure 2014 - The strategic value of the cloud - Joe Weinman
Structure 2014 - The strategic value of the cloud - Joe Weinman
 
Structure 2014 - The right and wrong way to scale - Rackspace
Structure 2014 - The right and wrong way to scale - RackspaceStructure 2014 - The right and wrong way to scale - Rackspace
Structure 2014 - The right and wrong way to scale - Rackspace
 
Structure 2014 - The future of cloud computing survey results
Structure 2014 - The future of cloud computing survey resultsStructure 2014 - The future of cloud computing survey results
Structure 2014 - The future of cloud computing survey results
 
Structure 2014 - Launchpad Competition
Structure 2014 - Launchpad CompetitionStructure 2014 - Launchpad Competition
Structure 2014 - Launchpad Competition
 
Structure 2014 - Disrupting the data center - Intel sponsor workshop
Structure 2014 - Disrupting the data center - Intel sponsor workshopStructure 2014 - Disrupting the data center - Intel sponsor workshop
Structure 2014 - Disrupting the data center - Intel sponsor workshop
 
Structure 2014 - Cloud trends - Battery
Structure 2014 - Cloud trends - BatteryStructure 2014 - Cloud trends - Battery
Structure 2014 - Cloud trends - Battery
 
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
 
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
 
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit BendovStructure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
 
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
 
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA, Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
 
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari GesherStructure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
 
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris HaddadStructure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
 
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
 
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrathStructure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
 
Structure Data 2014: IS VIDEO BIG DATA?, Steve Russell
Structure Data 2014: IS VIDEO BIG DATA?, Steve RussellStructure Data 2014: IS VIDEO BIG DATA?, Steve Russell
Structure Data 2014: IS VIDEO BIG DATA?, Steve Russell
 
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan WaiteStructure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
 
How Data is Remaking E-commerce - from Roadmap 2013
How Data is Remaking E-commerce - from Roadmap 2013How Data is Remaking E-commerce - from Roadmap 2013
How Data is Remaking E-commerce - from Roadmap 2013
 
25 Favorite Experiences in Tech - from Roadmap 2013
25 Favorite Experiences in Tech - from Roadmap 201325 Favorite Experiences in Tech - from Roadmap 2013
25 Favorite Experiences in Tech - from Roadmap 2013
 
How Moore’s Law is Influencing Design - from Roadmap 2013
How Moore’s Law is Influencing Design - from Roadmap 2013How Moore’s Law is Influencing Design - from Roadmap 2013
How Moore’s Law is Influencing Design - from Roadmap 2013
 

Kürzlich hochgeladen

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Kürzlich hochgeladen (20)

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 

ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012

  • 1. ANALYZING LARGE-SCALE USER DATA SPEAKER: Aaron Kimball CTO WibiData Friday, July 27, 2012
  • 3. Analyzing  Large-­‐Scale  User  Data with  Hadoop  and  HBase Aaron  Kimball  –  CTO WibiData,   Friday, July 27, 2012
  • 4. We  can  now  collect   more  data  than  at   any  Dme  in  history. Friday, July 27, 2012
  • 5. Yesterday’s  engineering   challenge:  FiJng  the   problem  into  the   hardware. Friday, July 27, 2012
  • 6. Today’s  constrained   resource  is   understanding. Friday, July 27, 2012
  • 7. How  do  we  best  apply   data …to  beMer  serving  our   Friday, July 27, 2012
  • 8. The  best  products  are  user-­‐ • IntuiDve  UI • ConDnuously  learning – Guided  search – Smarter  recommenda1ons • More  effec1ve  service Friday, July 27, 2012
  • 9. What  are  we  building   Friday, July 27, 2012
  • 10. What  are  we  building   Friday, July 27, 2012
  • 11. What  are  we  building   Friday, July 27, 2012
  • 12. What  are  we  building   Friday, July 27, 2012
  • 13. What  are  we  building   Friday, July 27, 2012
  • 14. What  are  we  building   Friday, July 27, 2012
  • 15. What  are  we  building   Friday, July 27, 2012
  • 16. What  are  we  building   Friday, July 27, 2012
  • 17. What  are  we  building   Friday, July 27, 2012
  • 18. What  are  we  building   Friday, July 27, 2012
  • 19. Requirements 1.  Understand  the  user   populaDon Friday, July 27, 2012
  • 20. Requirements 2.  Respond  to   users  in  real   Dme Friday, July 27, 2012
  • 21. Requirements 3.  Support  graceful  data   evoluDon Friday, July 27, 2012
  • 22. Large-­‐scale  data  science  is   • What  does  a  user  look  like? – What  data  is  available  about  the  user? – Which  features  are  important? – Which  features  are  correlated? • How  do  I  model  this  in  MapReduce? • How  do  I  serve  results  in  a  Dmely   Friday, July 27, 2012
  • 24. Tools  of  the  trade • Store  all  data  about  a   user  in  one  place • Support  real-­‐Dme   get/put,  as  well  as   MapReduce Friday, July 27, 2012
  • 25. Tools  of  the  trade • Use  complex  data   types  to  model   complex  data • Support  extended   data  models  over   Dme Friday, July 27, 2012
  • 26. Tools  of  the  trade • Abstract  computaDonal   model  away  from   MapReduce • Support  computaDon   over  all  users…  or  one   user  at  a  Dme Friday, July 27, 2012
  • 34.                                                      :  for  set-­‐top  boxes Viewing/recording   history Friday, July 27, 2012
  • 35.                                                      :  for  set-­‐top  boxes Viewing/recording   history Friday, July 27, 2012
  • 36.                                                      :  for  set-­‐top  boxes          Libraries Device  and  User  Analysis Viewing/recording   history Personalized  offers   and   recommenda=ons Friday, July 27, 2012
  • 37.                                                      :  for  set-­‐top  boxes          Libraries Device  and  User  Analysis Viewing/recording   history Personalized  offers   and   recommenda=ons Friday, July 27, 2012
  • 38.                                                      :  for  set-­‐top  boxes          Libraries Device  and  User  Analysis Viewing/recording   history Personalized  offers   and   recommenda=ons Analysis  for   product   roadmap Friday, July 27, 2012
  • 39.                                                      :  for  set-­‐top  boxes          Libraries Device  and  User  Analysis Viewing/recording   history Personalized  offers   and   recommenda=ons Analysis  for   product   roadmap Friday, July 27, 2012
  • 40.                                                      :  for  set-­‐top  boxes          Libraries Device  and  User  Analysis Viewing/recording   history Personalized  offers   and   recommenda=ons Analysis  for   product   Tech  support   roadmap portal Friday, July 27, 2012
  • 41.                                                      :  for  set-­‐top  boxes          Libraries Device  and  User  Analysis Viewing/recording   history Personalized  offers   and   recommenda=ons Analysis  for   product   Tech  support   roadmap portal Friday, July 27, 2012
  • 42.                                                      :  for  set-­‐top  boxes          Libraries Device  and  User  Analysis Viewing/recording   history Personalized  offers   and   recommenda=ons Improve Analysis  for   d  reports   product   Tech  support   for   roadmap portal Friday, July 27, 2012 adver=se
  • 43. The  future • More  personalizaDon • AdapDve  UIs  (self  arranging   dashboards) • Targeted  content,  ads • More  effecDve  customer  service Friday, July 27, 2012
  • 44. Conclusions • ApplicaDons  are  becoming   increasingly   user-­‐centric • Data  drives  this  capability,  but   harnessing  it  requires  a  new   distributed  architecture Friday, July 27, 2012
  • 45. www.wibidata.com  /   Aaron  Kimball  –  aaron@wibidata.com Friday, July 27, 2012