SlideShare ist ein Scribd-Unternehmen logo
1 von 94
THE 3V’S OF BIG DATA: VARIETY, VELOCITY,
           and VOLUME
                      SPEAKER: Nick Weir
                               CEO
                               ChoozON


Tuesday, November 27, 12
Mining	
  Big	
  Data	
  and	
  the	
  3V’s:
                           The	
  Business	
  and	
  Science	
  of	
  Big	
  Data
                                                                             Nick	
  Weir,	
  Ph.D.	
  
                                                                                             CEO
                                                                           ChoozOn	
  Corpora,on
                                                                             Twi$er:	
  @usamaf

                                                                                    March	
  21,	
  2012
                                                           GigaOM	
  Structure:	
  Data	
  Conference
                                                                                      New	
  York,	
  NY




Tuesday, November 27, 12
Outline
         • Big	
  Data	
  all	
  around	
  us
         • The	
  role	
  of	
  Data	
  Mining	
  and	
  PredicEve	
  AnalyEcs
         • On-­‐line	
  data	
  and	
  facts
         • Case	
  studies	
  from	
  mulEple	
  verEcals:
               1. Yahoo!	
  Big	
  Data
               2. Social	
  Network	
  Data
               3. ChoozOn	
  Big	
  Data	
  from	
  offers




Tuesday, November 27, 12
What	
  MaPers	
  in	
  the	
  Age	
  of	
  Big	
  Data?	
  
               1.Being	
  able	
  to	
  exploit	
  all	
  the	
  data	
  that	
  is	
  available	
  
                 • not	
  just	
  what	
  you've	
  got	
  available	
  
                 • what	
  you	
  can	
  acquire	
  and	
  use	
  to	
  enhance	
  your	
  acEons

               2.	
  ProliferaEng	
  analyEcs	
  throughout	
  your	
  organizaEon
                     • make	
  every	
  part	
  of	
  your	
  business	
  smarter

               3.	
  Driving	
  significant	
  business	
  value	
  
                     • embedding	
  analyEcs	
  into	
  every	
  area	
  of	
  your	
  business	
  can	
  help	
  you	
  drive	
  
                       top	
  line	
  revenues	
  and/or	
  boPom	
  line	
  cost	
  efficiencies	
  


Tuesday, November 27, 12
Why	
  Big	
  Data?




Tuesday, November 27, 12
Why	
  Big	
  Data?
     A	
  new	
  term,	
  with	
  associated	
  “Data	
  Scien:st”	
  posi:ons:




Tuesday, November 27, 12
Why	
  Big	
  Data?
     A	
  new	
  term,	
  with	
  associated	
  “Data	
  Scien:st”	
  posi:ons:
     • Big	
  Data:	
  is	
  a	
  mix	
  of	
  structured,	
  semi-­‐structured,	
  and	
  unstructured	
  
       data:
           –Typically	
  breaks	
  barriers	
  for	
  tradiEonal	
  RDB	
  storage
           –Typically	
  breaks	
  limits	
  of	
  indexing	
  by	
  “rows”
           –Typically	
  requires	
  intensive	
  pre-­‐processing	
  before	
  each	
  query	
  to	
  extract	
  
            “some	
  structure”	
  –	
  usually	
  using	
  Map-­‐Reduce	
  type	
  operaEons




Tuesday, November 27, 12
Why	
  Big	
  Data?
     A	
  new	
  term,	
  with	
  associated	
  “Data	
  Scien:st”	
  posi:ons:
     • Big	
  Data:	
  is	
  a	
  mix	
  of	
  structured,	
  semi-­‐structured,	
  and	
  unstructured	
  
       data:
           –Typically	
  breaks	
  barriers	
  for	
  tradiEonal	
  RDB	
  storage
           –Typically	
  breaks	
  limits	
  of	
  indexing	
  by	
  “rows”
           –Typically	
  requires	
  intensive	
  pre-­‐processing	
  before	
  each	
  query	
  to	
  extract	
  
            “some	
  structure”	
  –	
  usually	
  using	
  Map-­‐Reduce	
  type	
  operaEons
     • Above	
  leads	
  to	
  “messy”	
  situaEons	
  with	
  no	
  standard	
  recipes	
  or	
  
       architecture:	
  hence	
  the	
  need	
  for	
  “data	
  scienEsts”	
  
           –Conduct	
  “Data	
  ExpediEons”	
  
           –Discovery	
  and	
  learning	
  on	
  the	
  spot

Tuesday, November 27, 12
What	
  Makes	
  Data	
  “Big	
  Data”?




Tuesday, November 27, 12
What	
  Makes	
  Data	
  “Big	
  Data”?
     • Big	
  Data	
  is	
  Characterized	
  by	
  the	
  3-­‐V’s:
       –Volume:	
  larger	
  than	
  “normal”	
  –	
  challenging	
  to	
  load	
  and	
  process
                • Expensive	
  to	
  do	
  ETL
                • Expensive	
  to	
  figure	
  out	
  how	
  to	
  index	
  and	
  retrieve
                • MulEple	
  dimensions	
  that	
  are	
  “key”
           –Velocity:	
  Rate	
  of	
  arrival	
  poses	
  real-­‐,me	
  constraints	
  on	
  what	
  are	
  typically	
  “batch	
  
              ETL”	
  opera,ons
                • If	
  you	
  fall	
  behind	
  catching	
  up	
  is	
  extremely	
  expensive	
  (replicate	
  very	
  expensive	
  systems)
                • Must	
  keep	
  up	
  with	
  rate	
  and	
  service	
  queries	
  on-­‐the-­‐fly
           –Variety:	
  Mix	
  of	
  data	
  types	
  and	
  varying	
  degrees	
  of	
  structure
                • Non-­‐standard	
  schema
                • Lots	
  of	
  BLOB’s	
  and	
  CLOB’s
                • DB	
  queries	
  don’t	
  know	
  what	
  to	
  do	
  with	
  semi-­‐structured	
  and	
  unstructured	
  data.


Tuesday, November 27, 12
The	
  DisEncEon	
  between	
  “Data”	
  and	
  “Big	
  Data”	
  is	
  fast	
  
                                           disappearing
         • Most	
  real	
  data	
  sets	
  nowadays	
  come	
  with	
  a	
  serious	
  mix	
  of	
  semi-­‐structured	
  and	
  
           unstructured	
  components:
               –   Images
               –   Video
               –   Text	
  descripEons	
  and	
  news,	
  blogs,	
  etc…
               –   User	
  and	
  customer	
  commentary
               –   ReacEons	
  on	
  social	
  media:	
  e.g.	
  TwiPer	
  is	
  a	
  mix	
  of	
  data	
  anyway
         • Using	
  standard	
  transforms,	
  enEty	
  extracEon,	
  and	
  new	
  generaEon	
  tools	
  to	
  transform	
  
           unstructured	
  raw	
  data	
  into	
  semi-­‐structured	
  analyzable	
  data	
  
         • Hadoop	
  vs.	
  Not	
  Hadoop	
  -­‐	
  	
  when	
  to	
  use	
  what	
  kind	
  of	
  techniques	
  requiring	
  Map-­‐Reduce	
  
           and	
  grid	
  compuEng




Tuesday, November 27, 12
Text	
  Data:	
  The	
  Big	
  Driver

         • While	
  we	
  speak	
  of	
  “big	
  data”	
  and	
  the	
  “Variety”	
  in	
  3-­‐V’s
         • Reality:	
  biggest	
  driver	
  of	
  growth	
  of	
  Big	
  Data	
  has	
  been	
  text	
  data
         • In	
  fact	
  Map-­‐Reduce	
  became	
  popularized	
  by	
  Google	
  to	
  address	
  the	
  
           problem	
  of	
  processing	
  large	
  amounts	
  of	
  text	
  data:	
  
               – Indexing	
  a	
  full	
  copy	
  of	
  the	
  web
               – Frequent	
  re-­‐indexing
               – Many	
  operaEons	
  with	
  each	
  being	
  a	
  simple	
  operaEon	
  but	
  done	
  at	
  large	
  scale
         • Most	
  work	
  on	
  analysis	
  of	
  “images”	
  and	
  “video”	
  data	
  has	
  really	
  been	
  
           reduced	
  to	
  analysis	
  of	
  surrounding	
  text



Tuesday, November 27, 12
To	
  Hadoop	
  or	
  not	
  to	
  Hadoop?
      When	
  to	
  use	
  techniques	
  requiring	
  Map-­‐Reduce	
  and	
  grid	
  compu?ng?
      • Typically	
  organizaEons	
  try	
  to	
  use	
  Map-­‐Reduce	
  for	
  everything	
  to	
  do	
  with	
  Big	
  Data
            – Inefficient	
  and	
  ojen	
  irraEonal
            – Certain	
  operaEons	
  require	
  specialized	
  storage
                 • UpdaEng	
  segment	
  memberships	
  over	
  large	
  numbers	
  of	
  users
                 • Defining	
  new	
  segments	
  on	
  user	
  or	
  usage	
  data
      • Map-­‐Reduce	
  is	
  useful	
  when	
  a	
  very	
  simple	
  operaEon	
  is	
  to	
  be	
  applied	
  on	
  a	
  large	
  body	
  of	
  
        unstructured	
  data
            – Typically	
  this	
  is	
  during	
  enEty	
  and	
  aPribute	
  extracEon
            – SEll	
  need	
  Big	
  Data	
  analysis	
  post-­‐Hadoop
      • Map-­‐Reduce	
  is	
  not	
  efficient	
  or	
  effecEve	
  for	
  tasks	
  involving	
  deeper	
  staEsEcal	
  modeling
            – 	
  Good	
  for	
  gathering	
  counts	
  and	
  simple	
  (sufficient)	
  staEsEcs
                 • E.g.	
  how	
  many	
  Emes	
  a	
  keyword	
  occurs,	
  quick	
  aggregaEon	
  of	
  simple	
  facts	
  in	
  unstructured	
  data,	
  esEmates	
  of	
  variances,	
  
                   density,	
  etc…
            – Mostly	
  pre-­‐processing	
  for	
  Data	
  Mining


Tuesday, November 27, 12
Hadoop	
  Use	
  Cases	
  by	
  Data	
  Type




Tuesday, November 27, 12
Tuesday, November 27, 12
Tuesday, November 27, 12
Big	
  Data	
  Applica:ons	
  and	
  Uses
                                                                                  100%	
  Capture
                                IT	
  Log	
  &	
  Security	
  
                              Forensics	
  &	
  AnalyEcs         Find	
  New	
  Signal           Predict	
  Events


                                                                             Product	
  Planning
                             Automated	
  Device	
  Data                       Failure	
  Analysis
                                   AnalyEcs
                                                                               ProacEve	
  Fixes

                                                                                RecommendaEon
                                     AdverEsing                                     SegmentaEon
                                      AnalyEcs                                       Social	
  Media


                                                                            Hadoop	
  +	
  	
  MPP	
  +	
  EDW
                                   Big	
  Data                   Cost	
  ReducEon                Ad	
  Hoc	
  Insight
                              Warehouse	
  AnalyEcs
                                                                            PredicEve	
  AnalyEcs




Tuesday, November 27, 12
So	
  this	
  Internet	
  thing	
  is	
  going	
  to	
  be	
  big!

                           Big	
  Opportunity,	
  Big	
  Data,	
  Big	
  Challenges!




Tuesday, November 27, 12
Stats	
  about	
  on-­‐line	
  usage




Tuesday, November 27, 12
Stats	
  about	
  on-­‐line	
  usage
        • How	
  many	
  people	
  are	
  on-­‐line	
  today?	
  
             – 2.1	
  Billion	
  (per	
  Comscore	
  esEmates)
             – 30%	
  of	
  world	
  PopulaEon




Tuesday, November 27, 12
Stats	
  about	
  on-­‐line	
  usage
        • How	
  many	
  people	
  are	
  on-­‐line	
  today?	
  
             – 2.1	
  Billion	
  (per	
  Comscore	
  esEmates)
             – 30%	
  of	
  world	
  PopulaEon

        • How	
  much	
  Eme	
  is	
  spent	
  on-­‐line	
  per	
  month	
  by	
  the	
  whole	
  world?
             – 4M	
  person-­‐years	
  per	
  month




Tuesday, November 27, 12
Stats	
  about	
  on-­‐line	
  usage
        • How	
  many	
  people	
  are	
  on-­‐line	
  today?	
  
             – 2.1	
  Billion	
  (per	
  Comscore	
  esEmates)
             – 30%	
  of	
  world	
  PopulaEon

        • How	
  much	
  Eme	
  is	
  spent	
  on-­‐line	
  per	
  month	
  by	
  the	
  whole	
  world?
             – 4M	
  person-­‐years	
  per	
  month

        • How	
  many	
  hours	
  per	
  month	
  per	
  Internet	
  User?
             – 16	
  hours	
  (global	
  average)
             – 32	
  hours	
  (U.S.	
  Average)



Tuesday, November 27, 12
Stats	
  about	
  on-­‐line	
  usage
        • How	
  many	
  people	
  are	
  on-­‐line	
  today?	
  
             – 2.1	
  Billion	
  (per	
  Comscore	
  esEmates)
             – 30%	
  of	
  world	
  PopulaEon

        • How	
  much	
  Eme	
  is	
  spent	
  on-­‐line	
  per	
  month	
  by	
  the	
  whole	
  world?
             – 4M	
  person-­‐years	
  per	
  month

        • How	
  many	
  hours	
  per	
  month	
  per	
  Internet	
  User?
             – 16	
  hours	
  (global	
  average)
             – 32	
  hours	
  (U.S.	
  Average)

        *Sources: Feb.2012 - from Go-Gulf.com compiled from Comscoredatamine.com, Nielsen.com, thisDigitalLife.com, PewInternet.org




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?
         – More	
  than	
  250M




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?
         – More	
  than	
  250M
   • Facebook:	
  Updates	
  per	
  day?




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?
         – More	
  than	
  250M
   • Facebook:	
  Updates	
  per	
  day?
         – More	
  than	
  800M




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?
         – More	
  than	
  250M
   • Facebook:	
  Updates	
  per	
  day?
         – More	
  than	
  800M
   • YouTube:	
  Views/day




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?
         – More	
  than	
  250M
   • Facebook:	
  Updates	
  per	
  day?
         – More	
  than	
  800M
   • YouTube:	
  Views/day
         – 4	
  Billion	
  views
         – 60	
  hours	
  of	
  video	
  uploaded	
  every	
  minute!




Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?
         – More	
  than	
  250M
   • Facebook:	
  Updates	
  per	
  day?
         – More	
  than	
  800M
   • YouTube:	
  Views/day
         – 4	
  Billion	
  views
         – 60	
  hours	
  of	
  video	
  uploaded	
  every	
  minute!
   • Social	
  Networks:	
  users	
  who	
  have	
  used	
  sites	
  for	
  spying	
  on	
  their	
  partners?
      –56%



Tuesday, November 27, 12
Volume,	
  Velocity,	
  Variety
   • Google:	
  	
  How	
  many	
  queries	
  per	
  day?
         – More	
  than	
  1	
  Billion
   • Twi$er:	
  How	
  many	
  Tweets/day?
         – More	
  than	
  250M
   • Facebook:	
  Updates	
  per	
  day?
         – More	
  than	
  800M
   • YouTube:	
  Views/day
         – 4	
  Billion	
  views
         – 60	
  hours	
  of	
  video	
  uploaded	
  every	
  minute!
   • Social	
  Networks:	
  users	
  who	
  have	
  used	
  sites	
  for	
  spying	
  on	
  their	
  partners?
      –56%
                *Sources: Feb.2012 - from Go-Gulf.com compiled from Comscoredatamine.com, Nielsen.com, thisDigitalLife.com,
                PewInternet.org



Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?
                    • 	
  What	
  is	
  the	
  emoEon?




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?
                    • 	
  What	
  is	
  the	
  emoEon?
                    • 	
   Is	
  it	
  negaEve	
  or	
  posiEve?




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?
                    • 	
  What	
  is	
  the	
  emoEon?
                    • 	
   Is	
  it	
  negaEve	
  or	
  posiEve?
                    • 	
   What	
  is	
  the	
  health	
  of	
  my	
  brand	
  online?




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?
                    • 	
  What	
  is	
  the	
  emoEon?
                    • 	
   Is	
  it	
  negaEve	
  or	
  posiEve?
                    • 	
   What	
  is	
  the	
  health	
  of	
  my	
  brand	
  online?
                    Do	
  we	
  understand	
  context	
  and	
  content?




Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?
                    • 	
  What	
  is	
  the	
  emoEon?
                    • 	
   Is	
  it	
  negaEve	
  or	
  posiEve?
                    • 	
   What	
  is	
  the	
  health	
  of	
  my	
  brand	
  online?
                    Do	
  we	
  understand	
  context	
  and	
  content?
                    • 	
  What	
  are	
  appropriate	
  ads?



Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?
                    • 	
  What	
  is	
  the	
  emoEon?
                    • 	
   Is	
  it	
  negaEve	
  or	
  posiEve?
                    • 	
   What	
  is	
  the	
  health	
  of	
  my	
  brand	
  online?
                    Do	
  we	
  understand	
  context	
  and	
  content?
                    • 	
  What	
  are	
  appropriate	
  ads?
                    • 	
   Is	
  it	
  Ok	
  to	
  associate	
  my	
  brand	
  with	
  this	
  content?


Tuesday, November 27, 12
Turning	
  the	
  three	
  Vs	
  of	
  Big	
  Data	
  into	
  Value
                    Do	
  we	
  understand	
  what	
  each	
  individual	
  is	
  trying	
  to	
  achieve?
                    • What	
  is	
  user	
  intent?
                    • Cri,cal	
  in	
  mone,za,on,	
  adver,sing,	
  etc…
                    Do	
  we	
  understand	
  what	
  a	
  community’s	
  sen:ment	
  is?
                    • 	
  What	
  is	
  the	
  emoEon?
                    • 	
   Is	
  it	
  negaEve	
  or	
  posiEve?
                    • 	
   What	
  is	
  the	
  health	
  of	
  my	
  brand	
  online?
                    Do	
  we	
  understand	
  context	
  and	
  content?
                    • 	
  What	
  are	
  appropriate	
  ads?
                    • 	
   Is	
  it	
  Ok	
  to	
  associate	
  my	
  brand	
  with	
  this	
  content?
                    • Is	
  content	
  sad?,	
  happy?,	
  serious?,	
  informaEve?

Tuesday, November 27, 12
Case	
  Studies
                              Yahoo!	
  Big	
  Data




                                                      18
Tuesday, November 27, 12
Yahoo!	
  –	
  One	
  of	
  Largest	
  DesEnaEons	
  on	
  the	
  Web
                                                                                                                                   More people visited Yahoo!
                                                                                                                                      in the past month than:

                                                                                                                                    •   Use coupons
    80%	
  of	
  the	
  U.S.	
  Internet	
  popula:on	
  uses	
  Yahoo!	
  
                                                                                                                                    •   Vote
    –	
  Over	
  600	
  million	
  users	
  per	
  month	
  globally!
                                                                                                                                    •   Recycle
    • Global	
  network	
  of	
  content,	
  commerce,	
  media,	
  search	
  and	
  access	
  products                             •   Exercise regularly
    • 100+	
  properEes	
  including	
  mail,	
  TV,	
  news,	
  shopping,	
  finance,	
  autos,	
  travel,	
  games,	
  movies,	
   •   Have children living at
          health,	
  etc.                                                                                                               home
                                                                                                                                    •   Wear sunscreen regularly
    • 25+	
  terabytes	
  of	
  data	
  collected	
  each	
  day
          • RepresenEng	
  1000’s	
  of	
  cataloged	
  consumer	
  behaviors


      Data is used to develop content, consumer, category and campaign insights for our key content
      partners and large advertisers

                                     Sources: Mediamark Research, Spring 2004 and comScore Media Metrix, February 2005.


Tuesday, November 27, 12
Yahoo!	
  Big	
  Data	
  –	
  A	
  league	
  of	
  its	
  own…




                                 GRAND CHALLENGE PROBLEMS OF DATA PROCESSING
                               TRAVEL, CREDIT CARD PROCESSING, STOCK EXCHANGE, RETAIL, INTERNET

                                  Y! Data Challenge Exceeds others by 2 orders of magnitude



Tuesday, November 27, 12
Behavioral	
  TargeEng	
  (BT)




Tuesday, November 27, 12
Behavioral	
  TargeEng	
  (BT)




                                                 Targe:ng	
  your	
  ads	
  to	
  consumers	
  whose	
  
                                                    recent	
  behaviors	
  online	
  indicate	
  that	
  
                                               your	
  product	
  category	
  is	
  relevant	
  to	
  them


Tuesday, November 27, 12
Yahoo!	
  User	
  DNA




    • On a per consumer basis: maintain a behavioral/interests
      profile and profitability (user value and LTV) metrics
Tuesday, November 27, 12
How	
  it	
  works	
  |	
  Network	
  +	
  Interests	
  +	
  Modelling




Tuesday, November 27, 12
How	
  it	
  works	
  |	
  Network	
  +	
  Interests	
  +	
  Modelling
                            Varying Product Purchase Cycles   Analyze predictive patterns for purchase cycles in over
                                                              100 product categories




Tuesday, November 27, 12
How	
  it	
  works	
  |	
  Network	
  +	
  Interests	
  +	
  Modelling
                              Match Users to the Models   Analyze predictive patterns for purchase cycles in over
                                                          100 product categories


                                                          In each category, build models to describe behaviour
                                                          most likely to lead to an ad response (i.e. click).




Tuesday, November 27, 12
How	
  it	
  works	
  |	
  Network	
  +	
  Interests	
  +	
  Modelling
                              Rewarding Good Behaviour    Analyze predictive patterns for purchase cycles in over
                                                          100 product categories


                                                          In each category, build models to describe behaviour
                                                          most likely to lead to an ad response (i.e. click).


                                                          Score each user for fit with every category…daily.




Tuesday, November 27, 12
How	
  it	
  works	
  |	
  Network	
  +	
  Interests	
  +	
  Modelling
                             Identify Most Relevant Users   Analyze predictive patterns for purchase cycles in over
                                                            100 product categories


                                                            In each category, build models to describe behaviour
                                                            most likely to lead to an ad response (i.e. click).


                                                            Score each user for fit with every category…daily.


                                                            Target ads to users who get highest ‘relevance’ scores in
                                                            the targeting categories




Tuesday, November 27, 12
Recency	
  MaPers,	
  So	
  Does	
  Intensity

            Active now…                             …and with feeling




Tuesday, November 27, 12
DifferenEaEon	
  |	
  Category	
  specific	
  modelling




Tuesday, November 27, 12
DifferenEaEon	
  |	
  Category	
  specific	
  modelling




Tuesday, November 27, 12
DifferenEaEon	
  |	
  Category	
  specific	
  modelling




Tuesday, November 27, 12
DifferenEaEon	
  |	
  Category	
  specific	
  modelling




Tuesday, November 27, 12
Automobile	
  Purchase	
  Intender	
  Example

    • A	
  test	
  ad-­‐campaign	
  with	
  a	
  major	
  Euro	
  automobile	
  manufacturer
          – Designed	
  a	
  test	
  that	
  served	
  the	
  same	
  ad	
  creaEve	
  to	
  test	
  and	
  control	
  groups	
  on	
  Yahoo
          – Success	
  metric:	
  performing	
  specific	
  acEons	
  on	
  Jaguar	
  website

    • Test	
  results:	
  900%	
  conversion	
  lij	
  vs.	
  control	
  group
          – Purchase	
  Intenders	
  were	
  9	
  Emes	
  more	
  likely	
  to	
  configure	
  a	
  vehicle,	
  request	
  a	
  price	
  
            quote	
  or	
  locate	
  a	
  dealer	
  than	
  consumers	
  in	
  the	
  control	
  group
          – ~3x	
  higher	
  click	
  through	
  rates	
  vs.	
  control	
  group




Tuesday, November 27, 12
Mortgage	
  Intender	
  Example
                                                              Results from a client campaign on Yahoo! Network
    Example: Mortgages
                                                                                                                                                 +626%
                                                                                                                                                Conv Lift




    We found:                                                                                             +122%
                                                                                                         CTR Lift

    1,900,000 people looking                  Y! Finance Audience                           Mortgage Loans Target Mortgage Loans Target


    for mortgage loans.                           Example search terms qualified for this target:

                                                   Mortgages                 Home Loans Refinancing                                   Ditech

                                                  Example Yahoo! Pages visited:
                                                  Financing section in Real Estate
                                                  Mortgage Loans area in Finance
                                                  Real Estate section in Yellow Pages


                                           Source: Campaign Click thru Rate lift is determined by Yahoo! Internal research. Conversion is the
                                           number of qualified leads from clicks over number of impressions served. Audience size represents
                                           the audience within this behavioral interest category that has the highest propensity to
                                           engage with a brand or product and to click on an offer.
    Date: March 2006


Tuesday, November 27, 12
Experience summary at Yahoo!
              • Dealing with the largest data source in the world (25 Terabyte
                per day)
              • BT business was grown from $20M to about $500M in 3
                years of investment!
              • Building the largest database systems:
                    – World’s largest Oracle data warehouse
                    – World’s largest single DB
                    – Over 300 data mart data
                    – Analytics with thousands of KPI’s
                    – Over 5000 users of reports
                    – Largest targeting system in the world
              • Big demands for grid computing (Hadoop)
Tuesday, November 27, 12
Social	
  Network
                             Social	
  Network	
  MarkeEng
                           Understanding	
  Context	
  for	
  Ads



Tuesday, November 27, 12
Case Study: TWITTER Social Marketing?
                             Viacom’s VH1 Twitter campaign on ANVIL (the movie)
                                        (week of May 14th , 2009 – see AdAge article)

           Marketing ANVIL movie
           Very niche audience, how do you reach them?




Tuesday, November 27, 12
Case Study: TWITTER Social Marketing?
                                   Viacom’s VH1 Twitter campaign on ANVIL (the movie)
                                             (week of May 14th , 2009 – see AdAge article)

           Marketing ANVIL movie
           Very niche audience, how do you reach them?
                   Diffusion 	

                    Give 20 free DVD’s to major related artists/groups, ask them
                   To notify Twitter groups – reached over 2M people




Tuesday, November 27, 12
Case Study: TWITTER Social Marketing?
                                   Viacom’s VH1 Twitter campaign on ANVIL (the movie)
                                             (week of May 14th , 2009 – see AdAge article)

           Marketing ANVIL movie
           Very niche audience, how do you reach them?
                   Diffusion 	

                    Give 20 free DVD’s to major related artists/groups, ask them
                   To notify Twitter groups – reached over 2M people

                   Social Identity: power of word of mouth...
                   What is the Cost to VH-1?
                   Compare with traditional approach: TV commercials to promote a documentary film?




Tuesday, November 27, 12
The	
  Display	
  Ads	
  Challenge	
  Today
        Damaging to Brand                                 Irrelevant and Damaging to Brand




                                                          Mismatch




                           Completely Irrelevant




Tuesday, November 27, 12
NetSeer:	
  Intent	
  for	
  Display
  • And	
  Currently	
  Processing	
  4	
  Billion	
  Impressions	
  per	
  Day




Tuesday, November 27, 12
NetSeer:	
  Intent	
  for	
  Display
  • And	
  Currently	
  Processing	
  4	
  Billion	
  Impressions	
  per	
  Day




Tuesday, November 27, 12
NetSeer:	
  Intent	
  for	
  Display
  • And	
  Currently	
  Processing	
  4	
  Billion	
  Impressions	
  per	
  Day




Tuesday, November 27, 12
NetSeer:	
  Intent	
  for	
  Display
  • And	
  Currently	
  Processing	
  4	
  Billion	
  Impressions	
  per	
  Day




Tuesday, November 27, 12
Problem:	
  Hard	
  to	
  Understand	
  User	
  Intent
                             Contextual	
  Ad	
  served	
  by	
  Google




Tuesday, November 27, 12
Problem:	
  Hard	
  to	
  Understand	
  User	
  Intent
                             Contextual	
  Ad	
  served	
  by	
  Google   What	
  NetSeer	
  Sees:	
  




Tuesday, November 27, 12
Specialized	
  Search	
  through	
  Big	
  Data	
  AnalyEcs	
  over	
  
                                             the	
  Offers	
  Universe




Tuesday, November 27, 12
Chaos	
  for	
  Consumers	
  




                                  Consumer




Tuesday, November 27, 12
Chaos	
  for	
  Consumers	
  

                                                       Deep
                                                     Discount
                                                       Sites         Coupon
                                Flash                                 Sites
                              Sale Sites


                                                                                   Loyalty
                                                                                  Programs


                           Daily Deals
                              Sites

                                                     Consumer
                                                                               Online
                                                                               Loyalty
                                                                              Programs
                                                                 Social
                                           Search               Networks
                                           Engines




Tuesday, November 27, 12
The	
  Business	
  Model	
  behind	
  ChoozOn’s	
  Use	
  of	
  Big	
  Data




Tuesday, November 27, 12
The	
  Business	
  Model	
  behind	
  ChoozOn’s	
  Use	
  of	
  Big	
  Data

                                            Consumers
                                            • Value from brands they love
                                            • Tame the deal chaos




Tuesday, November 27, 12
The	
  Business	
  Model	
  behind	
  ChoozOn’s	
  Use	
  of	
  Big	
  Data

                                            Consumers
                                            • Value from brands they love
                                            • Tame the deal chaos




                                           Marketers
                                           • Reach targeted consumers
                                           • Build loyalty
                                           • Create brand evangelists

Tuesday, November 27, 12
The	
  Business	
  Model	
  behind	
  ChoozOn’s	
  Use	
  of	
  Big	
  Data

                                            Consumers
                                            • Value from brands they love
                                            • Tame the deal chaos




                                           Marketers
                                           • Reach targeted consumers
                                           • Build loyalty
                                           • Create brand evangelists

Tuesday, November 27, 12
Carol’s	
  Consumer	
  Network




Tuesday, November 27, 12
Carol’s	
  Consumer	
  Network
                           “Chozen”	
  Brands




Tuesday, November 27, 12
Carol’s	
  Consumer	
  Network
                           “Chozen”	
  Brands                              Interests	
  (Intent)




Tuesday, November 27, 12
Carol’s	
  Consumer	
  Network
                           “Chozen”	
  Brands                              Interests	
  (Intent)




                                                                          Shopping	
  Pals

Tuesday, November 27, 12
Carol’s	
  Consumer	
  Network
                           “Chozen”	
  Brands                               Interests	
  (Intent)




                                                                           Shopping	
  Pals
                                 Deal	
  Clubs
Tuesday, November 27, 12
What	
  Carol	
  Gets
                           Carol’s	
  Consumer	
  Network




Tuesday, November 27, 12
What	
  Carol	
  Gets
                           Carol’s	
  Consumer	
  Network                  The	
  Universe	
  of	
  Deals
                                                                 Affiliate	
  Deals
                                                                 Loyalty	
  Programs
                                                                 Daily	
  Deals
                                                                 Flash	
  Sales
                                                                 Deep	
  Discounters




                                                    Intelligent Matching




                                 A	
  Personal	
  Shopper	
  for	
  Deals
Tuesday, November 27, 12
What	
  Carol	
  Gets
                           Carol’s	
  Consumer	
  Network                  The	
  Universe	
  of	
  Deals
                                                                 Affiliate	
  Deals
                                                                 Loyalty	
  Programs                        1,600+	
  brands
                                                                 Daily	
  Deals
                                                                 Flash	
  Sales                             100,000+	
  offers
                                                                 Deep	
  Discounters




                                                    Intelligent Matching




                                 A	
  Personal	
  Shopper	
  for	
  Deals
Tuesday, November 27, 12
What	
  Carol	
  Gets
                           Carol’s	
  Consumer	
  Network                  The	
  Universe	
  of	
  Deals
                                                                 Affiliate	
  Deals
                                                                 Loyalty	
  Programs                        1,600+	
  brands
                                                                 Daily	
  Deals
                                                                 Flash	
  Sales                             100,000+	
  offers
                                                                 Deep	
  Discounters

                                                                 PLUS her                             Inbox™



                                                    Intelligent Matching




                                 A	
  Personal	
  Shopper	
  for	
  Deals
Tuesday, November 27, 12
Thank	
  You!	
  
                           TwiPer:	
  @choozon

                              nick@choozon.net
                             www.ChoozOn.com

Tuesday, November 27, 12
Tuesday, November 27, 12

Weitere ähnliche Inhalte

Was ist angesagt?

Introduction to Hadoop
Introduction to HadoopIntroduction to Hadoop
Introduction to Hadoopjoelcrabb
 
Using Hadoop as a platform for Master Data Management
Using Hadoop as a platform for Master Data ManagementUsing Hadoop as a platform for Master Data Management
Using Hadoop as a platform for Master Data ManagementDataWorks Summit
 
Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...
Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...
Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...Simplilearn
 
Big Data - Applications and Technologies Overview
Big Data - Applications and Technologies OverviewBig Data - Applications and Technologies Overview
Big Data - Applications and Technologies OverviewSivashankar Ganapathy
 
Big Data
Big DataBig Data
Big DataNGDATA
 
Data Provisioning & Optimization
Data Provisioning & OptimizationData Provisioning & Optimization
Data Provisioning & OptimizationAmbareesh Kulkarni
 
Presentation About Big Data (DBMS)
Presentation About Big Data (DBMS)Presentation About Big Data (DBMS)
Presentation About Big Data (DBMS)SiamAhmed16
 
Big Data - The 5 Vs Everyone Must Know
Big Data - The 5 Vs Everyone Must KnowBig Data - The 5 Vs Everyone Must Know
Big Data - The 5 Vs Everyone Must KnowBernard Marr
 
Big data PPT prepared by Hritika Raj (Shivalik college of engg.)
Big data PPT prepared by Hritika Raj (Shivalik college of engg.)Big data PPT prepared by Hritika Raj (Shivalik college of engg.)
Big data PPT prepared by Hritika Raj (Shivalik college of engg.)Hritika Raj
 
Snowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleSnowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleAdam Doyle
 
HADOOP TECHNOLOGY ppt
HADOOP  TECHNOLOGY pptHADOOP  TECHNOLOGY ppt
HADOOP TECHNOLOGY pptsravya raju
 
Introduction to Apache Hadoop Eco-System
Introduction to Apache Hadoop Eco-SystemIntroduction to Apache Hadoop Eco-System
Introduction to Apache Hadoop Eco-SystemMd. Hasan Basri (Angel)
 
Big Data Analytics with Hadoop
Big Data Analytics with HadoopBig Data Analytics with Hadoop
Big Data Analytics with HadoopPhilippe Julio
 
Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...
Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...
Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...Edureka!
 

Was ist angesagt? (20)

Hadoop
HadoopHadoop
Hadoop
 
Introduction to Hadoop
Introduction to HadoopIntroduction to Hadoop
Introduction to Hadoop
 
Using Hadoop as a platform for Master Data Management
Using Hadoop as a platform for Master Data ManagementUsing Hadoop as a platform for Master Data Management
Using Hadoop as a platform for Master Data Management
 
Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...
Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...
Introduction To Hadoop | What Is Hadoop And Big Data | Hadoop Tutorial For Be...
 
Big Data - Applications and Technologies Overview
Big Data - Applications and Technologies OverviewBig Data - Applications and Technologies Overview
Big Data - Applications and Technologies Overview
 
Big Data
Big DataBig Data
Big Data
 
Data Provisioning & Optimization
Data Provisioning & OptimizationData Provisioning & Optimization
Data Provisioning & Optimization
 
Big data
Big dataBig data
Big data
 
Presentation on Big Data
Presentation on Big DataPresentation on Big Data
Presentation on Big Data
 
Big Data ppt
Big Data pptBig Data ppt
Big Data ppt
 
Presentation About Big Data (DBMS)
Presentation About Big Data (DBMS)Presentation About Big Data (DBMS)
Presentation About Big Data (DBMS)
 
Big Data
Big DataBig Data
Big Data
 
Big Data - The 5 Vs Everyone Must Know
Big Data - The 5 Vs Everyone Must KnowBig Data - The 5 Vs Everyone Must Know
Big Data - The 5 Vs Everyone Must Know
 
Big data PPT prepared by Hritika Raj (Shivalik college of engg.)
Big data PPT prepared by Hritika Raj (Shivalik college of engg.)Big data PPT prepared by Hritika Raj (Shivalik college of engg.)
Big data PPT prepared by Hritika Raj (Shivalik college of engg.)
 
Snowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at ScaleSnowflake Data Science and AI/ML at Scale
Snowflake Data Science and AI/ML at Scale
 
HADOOP TECHNOLOGY ppt
HADOOP  TECHNOLOGY pptHADOOP  TECHNOLOGY ppt
HADOOP TECHNOLOGY ppt
 
Introduction to Apache Hadoop Eco-System
Introduction to Apache Hadoop Eco-SystemIntroduction to Apache Hadoop Eco-System
Introduction to Apache Hadoop Eco-System
 
Big Data Analytics with Hadoop
Big Data Analytics with HadoopBig Data Analytics with Hadoop
Big Data Analytics with Hadoop
 
Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...
Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...
Data Warehouse Tutorial For Beginners | Data Warehouse Concepts | Data Wareho...
 
Chapter 1 big data
Chapter 1 big dataChapter 1 big data
Chapter 1 big data
 

Ähnlich wie THE 3V's OF BIG DATA: VARIETY, VELOCITY, AND VOLUME from Structure:Data 2012

THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUMETHE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUMEGigaom
 
Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...
Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...
Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...BigMine
 
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data TorrentSeagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data TorrentSeeling Cheung
 
Big Data Analytics Materials, Chapter: 1
Big Data Analytics Materials, Chapter: 1Big Data Analytics Materials, Chapter: 1
Big Data Analytics Materials, Chapter: 1RUHULAMINHAZARIKA
 
Big Data, NoSQL, NewSQL & The Future of Data Management
Big Data, NoSQL, NewSQL & The Future of Data ManagementBig Data, NoSQL, NewSQL & The Future of Data Management
Big Data, NoSQL, NewSQL & The Future of Data ManagementTony Bain
 
Anexinet Big Data Solutions
Anexinet Big Data SolutionsAnexinet Big Data Solutions
Anexinet Big Data SolutionsMark Kromer
 
SQL vs NoSQL: Big Data Adoption & Success in the Enterprise
SQL vs NoSQL: Big Data Adoption & Success in the EnterpriseSQL vs NoSQL: Big Data Adoption & Success in the Enterprise
SQL vs NoSQL: Big Data Adoption & Success in the EnterpriseAnita Luthra
 
Data mining - GDi Techno Solutions
Data mining - GDi Techno SolutionsData mining - GDi Techno Solutions
Data mining - GDi Techno SolutionsGDi Techno Solutions
 
big data processing.pptx
big data processing.pptxbig data processing.pptx
big data processing.pptxssuser96aab9
 
Introduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQLIntroduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQLTushar Shende
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big DataIMC Institute
 
Lunch & Learn Intro to Big Data
Lunch & Learn Intro to Big DataLunch & Learn Intro to Big Data
Lunch & Learn Intro to Big DataMelissa Hornbostel
 
Qiagram Slides 2011 05
Qiagram Slides 2011 05Qiagram Slides 2011 05
Qiagram Slides 2011 05bhughes26
 

Ähnlich wie THE 3V's OF BIG DATA: VARIETY, VELOCITY, AND VOLUME from Structure:Data 2012 (20)

THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUMETHE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME
 
Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...
Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...
Big Data Analytics: Applications and Opportunities in On-line Predictive Mode...
 
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data TorrentSeagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
 
Big Data Analytics Materials, Chapter: 1
Big Data Analytics Materials, Chapter: 1Big Data Analytics Materials, Chapter: 1
Big Data Analytics Materials, Chapter: 1
 
Big data.pptx
Big data.pptxBig data.pptx
Big data.pptx
 
Big data ppt
Big data pptBig data ppt
Big data ppt
 
Dw 07032018-dr pl pradhan
Dw 07032018-dr pl pradhanDw 07032018-dr pl pradhan
Dw 07032018-dr pl pradhan
 
Big Data, NoSQL, NewSQL & The Future of Data Management
Big Data, NoSQL, NewSQL & The Future of Data ManagementBig Data, NoSQL, NewSQL & The Future of Data Management
Big Data, NoSQL, NewSQL & The Future of Data Management
 
Anexinet Big Data Solutions
Anexinet Big Data SolutionsAnexinet Big Data Solutions
Anexinet Big Data Solutions
 
SQL vs NoSQL: Big Data Adoption & Success in the Enterprise
SQL vs NoSQL: Big Data Adoption & Success in the EnterpriseSQL vs NoSQL: Big Data Adoption & Success in the Enterprise
SQL vs NoSQL: Big Data Adoption & Success in the Enterprise
 
Data mining - GDi Techno Solutions
Data mining - GDi Techno SolutionsData mining - GDi Techno Solutions
Data mining - GDi Techno Solutions
 
big data processing.pptx
big data processing.pptxbig data processing.pptx
big data processing.pptx
 
Introduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQLIntroduction to Bigdata and NoSQL
Introduction to Bigdata and NoSQL
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big Data
 
Lunch & Learn Intro to Big Data
Lunch & Learn Intro to Big DataLunch & Learn Intro to Big Data
Lunch & Learn Intro to Big Data
 
Big Data Boom
Big Data BoomBig Data Boom
Big Data Boom
 
Big data
Big dataBig data
Big data
 
Qiagram Slides 2011 05
Qiagram Slides 2011 05Qiagram Slides 2011 05
Qiagram Slides 2011 05
 
Qiagram
QiagramQiagram
Qiagram
 
Qiagram
QiagramQiagram
Qiagram
 

Mehr von Gigaom

Structure 2014 - The strategic value of the cloud - Joe Weinman
Structure 2014 - The strategic value of the cloud - Joe WeinmanStructure 2014 - The strategic value of the cloud - Joe Weinman
Structure 2014 - The strategic value of the cloud - Joe WeinmanGigaom
 
Structure 2014 - The right and wrong way to scale - Rackspace
Structure 2014 - The right and wrong way to scale - RackspaceStructure 2014 - The right and wrong way to scale - Rackspace
Structure 2014 - The right and wrong way to scale - RackspaceGigaom
 
Structure 2014 - The future of cloud computing survey results
Structure 2014 - The future of cloud computing survey resultsStructure 2014 - The future of cloud computing survey results
Structure 2014 - The future of cloud computing survey resultsGigaom
 
Structure 2014 - Launchpad Competition
Structure 2014 - Launchpad CompetitionStructure 2014 - Launchpad Competition
Structure 2014 - Launchpad CompetitionGigaom
 
Structure 2014 - Disrupting the data center - Intel sponsor workshop
Structure 2014 - Disrupting the data center - Intel sponsor workshopStructure 2014 - Disrupting the data center - Intel sponsor workshop
Structure 2014 - Disrupting the data center - Intel sponsor workshopGigaom
 
Structure 2014 - Cloud trends - Battery
Structure 2014 - Cloud trends - BatteryStructure 2014 - Cloud trends - Battery
Structure 2014 - Cloud trends - BatteryGigaom
 
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...Gigaom
 
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...Gigaom
 
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit BendovStructure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit BendovGigaom
 
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...Gigaom
 
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA, Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA, Gigaom
 
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari GesherStructure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari GesherGigaom
 
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris HaddadStructure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris HaddadGigaom
 
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...Gigaom
 
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrathStructure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrathGigaom
 
Structure Data 2014: IS VIDEO BIG DATA?, Steve Russell
Structure Data 2014: IS VIDEO BIG DATA?, Steve RussellStructure Data 2014: IS VIDEO BIG DATA?, Steve Russell
Structure Data 2014: IS VIDEO BIG DATA?, Steve RussellGigaom
 
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan WaiteStructure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan WaiteGigaom
 
How Data is Remaking E-commerce - from Roadmap 2013
How Data is Remaking E-commerce - from Roadmap 2013How Data is Remaking E-commerce - from Roadmap 2013
How Data is Remaking E-commerce - from Roadmap 2013Gigaom
 
25 Favorite Experiences in Tech - from Roadmap 2013
25 Favorite Experiences in Tech - from Roadmap 201325 Favorite Experiences in Tech - from Roadmap 2013
25 Favorite Experiences in Tech - from Roadmap 2013Gigaom
 
How Moore’s Law is Influencing Design - from Roadmap 2013
How Moore’s Law is Influencing Design - from Roadmap 2013How Moore’s Law is Influencing Design - from Roadmap 2013
How Moore’s Law is Influencing Design - from Roadmap 2013Gigaom
 

Mehr von Gigaom (20)

Structure 2014 - The strategic value of the cloud - Joe Weinman
Structure 2014 - The strategic value of the cloud - Joe WeinmanStructure 2014 - The strategic value of the cloud - Joe Weinman
Structure 2014 - The strategic value of the cloud - Joe Weinman
 
Structure 2014 - The right and wrong way to scale - Rackspace
Structure 2014 - The right and wrong way to scale - RackspaceStructure 2014 - The right and wrong way to scale - Rackspace
Structure 2014 - The right and wrong way to scale - Rackspace
 
Structure 2014 - The future of cloud computing survey results
Structure 2014 - The future of cloud computing survey resultsStructure 2014 - The future of cloud computing survey results
Structure 2014 - The future of cloud computing survey results
 
Structure 2014 - Launchpad Competition
Structure 2014 - Launchpad CompetitionStructure 2014 - Launchpad Competition
Structure 2014 - Launchpad Competition
 
Structure 2014 - Disrupting the data center - Intel sponsor workshop
Structure 2014 - Disrupting the data center - Intel sponsor workshopStructure 2014 - Disrupting the data center - Intel sponsor workshop
Structure 2014 - Disrupting the data center - Intel sponsor workshop
 
Structure 2014 - Cloud trends - Battery
Structure 2014 - Cloud trends - BatteryStructure 2014 - Cloud trends - Battery
Structure 2014 - Cloud trends - Battery
 
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
Structure Data 2014: HOW MICRODATA CAN SAY A LOT ABOUT MACROECONOMICS, David ...
 
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
Structure Data 2014: QLIK SPONSOR WORKSHOP: ANALYTICS THE WAY NATURE INTENDED...
 
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit BendovStructure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
Structure Data 2014: FIVE MYTHS ABOUT BIG DATA, Amit Bendov
 
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
Structure Data 2014: AMID BILLIONS OF METRICS, YOUR SOFTWARE IS TRYING TO TEL...
 
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA, Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
Structure Data 2014: SISENSE SPONSOR WORKSHOP: ON BEER, CHIPS AND DATA,
 
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari GesherStructure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
Structure Data 2014: INVERTING 80/20: BEYOND BESPOKE BIG DATA, Ari Gesher
 
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris HaddadStructure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
Structure Data 2014: TRACKING A SOCCER GAME WITH BIG DATA, Chris Haddad
 
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
Structure Data 2014: TECH AGAINST HUMAN TRAFFICKING AND ILLICIT NETWORKS, Jus...
 
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrathStructure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
Structure Data 2014: DATA DRIVEN DESIGN AT FORMULA ONE SPEED, Geoff McGrath
 
Structure Data 2014: IS VIDEO BIG DATA?, Steve Russell
Structure Data 2014: IS VIDEO BIG DATA?, Steve RussellStructure Data 2014: IS VIDEO BIG DATA?, Steve Russell
Structure Data 2014: IS VIDEO BIG DATA?, Steve Russell
 
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan WaiteStructure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
Structure Data 2014: BIG DATA ANALYTICS RE-INVENTED, Ryan Waite
 
How Data is Remaking E-commerce - from Roadmap 2013
How Data is Remaking E-commerce - from Roadmap 2013How Data is Remaking E-commerce - from Roadmap 2013
How Data is Remaking E-commerce - from Roadmap 2013
 
25 Favorite Experiences in Tech - from Roadmap 2013
25 Favorite Experiences in Tech - from Roadmap 201325 Favorite Experiences in Tech - from Roadmap 2013
25 Favorite Experiences in Tech - from Roadmap 2013
 
How Moore’s Law is Influencing Design - from Roadmap 2013
How Moore’s Law is Influencing Design - from Roadmap 2013How Moore’s Law is Influencing Design - from Roadmap 2013
How Moore’s Law is Influencing Design - from Roadmap 2013
 

THE 3V's OF BIG DATA: VARIETY, VELOCITY, AND VOLUME from Structure:Data 2012

  • 1. THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME SPEAKER: Nick Weir CEO ChoozON Tuesday, November 27, 12
  • 2. Mining  Big  Data  and  the  3V’s: The  Business  and  Science  of  Big  Data Nick  Weir,  Ph.D.   CEO ChoozOn  Corpora,on Twi$er:  @usamaf March  21,  2012 GigaOM  Structure:  Data  Conference New  York,  NY Tuesday, November 27, 12
  • 3. Outline • Big  Data  all  around  us • The  role  of  Data  Mining  and  PredicEve  AnalyEcs • On-­‐line  data  and  facts • Case  studies  from  mulEple  verEcals: 1. Yahoo!  Big  Data 2. Social  Network  Data 3. ChoozOn  Big  Data  from  offers Tuesday, November 27, 12
  • 4. What  MaPers  in  the  Age  of  Big  Data?   1.Being  able  to  exploit  all  the  data  that  is  available   • not  just  what  you've  got  available   • what  you  can  acquire  and  use  to  enhance  your  acEons 2.  ProliferaEng  analyEcs  throughout  your  organizaEon • make  every  part  of  your  business  smarter 3.  Driving  significant  business  value   • embedding  analyEcs  into  every  area  of  your  business  can  help  you  drive   top  line  revenues  and/or  boPom  line  cost  efficiencies   Tuesday, November 27, 12
  • 5. Why  Big  Data? Tuesday, November 27, 12
  • 6. Why  Big  Data? A  new  term,  with  associated  “Data  Scien:st”  posi:ons: Tuesday, November 27, 12
  • 7. Why  Big  Data? A  new  term,  with  associated  “Data  Scien:st”  posi:ons: • Big  Data:  is  a  mix  of  structured,  semi-­‐structured,  and  unstructured   data: –Typically  breaks  barriers  for  tradiEonal  RDB  storage –Typically  breaks  limits  of  indexing  by  “rows” –Typically  requires  intensive  pre-­‐processing  before  each  query  to  extract   “some  structure”  –  usually  using  Map-­‐Reduce  type  operaEons Tuesday, November 27, 12
  • 8. Why  Big  Data? A  new  term,  with  associated  “Data  Scien:st”  posi:ons: • Big  Data:  is  a  mix  of  structured,  semi-­‐structured,  and  unstructured   data: –Typically  breaks  barriers  for  tradiEonal  RDB  storage –Typically  breaks  limits  of  indexing  by  “rows” –Typically  requires  intensive  pre-­‐processing  before  each  query  to  extract   “some  structure”  –  usually  using  Map-­‐Reduce  type  operaEons • Above  leads  to  “messy”  situaEons  with  no  standard  recipes  or   architecture:  hence  the  need  for  “data  scienEsts”   –Conduct  “Data  ExpediEons”   –Discovery  and  learning  on  the  spot Tuesday, November 27, 12
  • 9. What  Makes  Data  “Big  Data”? Tuesday, November 27, 12
  • 10. What  Makes  Data  “Big  Data”? • Big  Data  is  Characterized  by  the  3-­‐V’s: –Volume:  larger  than  “normal”  –  challenging  to  load  and  process • Expensive  to  do  ETL • Expensive  to  figure  out  how  to  index  and  retrieve • MulEple  dimensions  that  are  “key” –Velocity:  Rate  of  arrival  poses  real-­‐,me  constraints  on  what  are  typically  “batch   ETL”  opera,ons • If  you  fall  behind  catching  up  is  extremely  expensive  (replicate  very  expensive  systems) • Must  keep  up  with  rate  and  service  queries  on-­‐the-­‐fly –Variety:  Mix  of  data  types  and  varying  degrees  of  structure • Non-­‐standard  schema • Lots  of  BLOB’s  and  CLOB’s • DB  queries  don’t  know  what  to  do  with  semi-­‐structured  and  unstructured  data. Tuesday, November 27, 12
  • 11. The  DisEncEon  between  “Data”  and  “Big  Data”  is  fast   disappearing • Most  real  data  sets  nowadays  come  with  a  serious  mix  of  semi-­‐structured  and   unstructured  components: – Images – Video – Text  descripEons  and  news,  blogs,  etc… – User  and  customer  commentary – ReacEons  on  social  media:  e.g.  TwiPer  is  a  mix  of  data  anyway • Using  standard  transforms,  enEty  extracEon,  and  new  generaEon  tools  to  transform   unstructured  raw  data  into  semi-­‐structured  analyzable  data   • Hadoop  vs.  Not  Hadoop  -­‐    when  to  use  what  kind  of  techniques  requiring  Map-­‐Reduce   and  grid  compuEng Tuesday, November 27, 12
  • 12. Text  Data:  The  Big  Driver • While  we  speak  of  “big  data”  and  the  “Variety”  in  3-­‐V’s • Reality:  biggest  driver  of  growth  of  Big  Data  has  been  text  data • In  fact  Map-­‐Reduce  became  popularized  by  Google  to  address  the   problem  of  processing  large  amounts  of  text  data:   – Indexing  a  full  copy  of  the  web – Frequent  re-­‐indexing – Many  operaEons  with  each  being  a  simple  operaEon  but  done  at  large  scale • Most  work  on  analysis  of  “images”  and  “video”  data  has  really  been   reduced  to  analysis  of  surrounding  text Tuesday, November 27, 12
  • 13. To  Hadoop  or  not  to  Hadoop? When  to  use  techniques  requiring  Map-­‐Reduce  and  grid  compu?ng? • Typically  organizaEons  try  to  use  Map-­‐Reduce  for  everything  to  do  with  Big  Data – Inefficient  and  ojen  irraEonal – Certain  operaEons  require  specialized  storage • UpdaEng  segment  memberships  over  large  numbers  of  users • Defining  new  segments  on  user  or  usage  data • Map-­‐Reduce  is  useful  when  a  very  simple  operaEon  is  to  be  applied  on  a  large  body  of   unstructured  data – Typically  this  is  during  enEty  and  aPribute  extracEon – SEll  need  Big  Data  analysis  post-­‐Hadoop • Map-­‐Reduce  is  not  efficient  or  effecEve  for  tasks  involving  deeper  staEsEcal  modeling –  Good  for  gathering  counts  and  simple  (sufficient)  staEsEcs • E.g.  how  many  Emes  a  keyword  occurs,  quick  aggregaEon  of  simple  facts  in  unstructured  data,  esEmates  of  variances,   density,  etc… – Mostly  pre-­‐processing  for  Data  Mining Tuesday, November 27, 12
  • 14. Hadoop  Use  Cases  by  Data  Type Tuesday, November 27, 12
  • 17. Big  Data  Applica:ons  and  Uses 100%  Capture IT  Log  &  Security   Forensics  &  AnalyEcs Find  New  Signal Predict  Events Product  Planning Automated  Device  Data Failure  Analysis AnalyEcs ProacEve  Fixes RecommendaEon AdverEsing SegmentaEon AnalyEcs Social  Media Hadoop  +    MPP  +  EDW Big  Data Cost  ReducEon Ad  Hoc  Insight Warehouse  AnalyEcs PredicEve  AnalyEcs Tuesday, November 27, 12
  • 18. So  this  Internet  thing  is  going  to  be  big! Big  Opportunity,  Big  Data,  Big  Challenges! Tuesday, November 27, 12
  • 19. Stats  about  on-­‐line  usage Tuesday, November 27, 12
  • 20. Stats  about  on-­‐line  usage • How  many  people  are  on-­‐line  today?   – 2.1  Billion  (per  Comscore  esEmates) – 30%  of  world  PopulaEon Tuesday, November 27, 12
  • 21. Stats  about  on-­‐line  usage • How  many  people  are  on-­‐line  today?   – 2.1  Billion  (per  Comscore  esEmates) – 30%  of  world  PopulaEon • How  much  Eme  is  spent  on-­‐line  per  month  by  the  whole  world? – 4M  person-­‐years  per  month Tuesday, November 27, 12
  • 22. Stats  about  on-­‐line  usage • How  many  people  are  on-­‐line  today?   – 2.1  Billion  (per  Comscore  esEmates) – 30%  of  world  PopulaEon • How  much  Eme  is  spent  on-­‐line  per  month  by  the  whole  world? – 4M  person-­‐years  per  month • How  many  hours  per  month  per  Internet  User? – 16  hours  (global  average) – 32  hours  (U.S.  Average) Tuesday, November 27, 12
  • 23. Stats  about  on-­‐line  usage • How  many  people  are  on-­‐line  today?   – 2.1  Billion  (per  Comscore  esEmates) – 30%  of  world  PopulaEon • How  much  Eme  is  spent  on-­‐line  per  month  by  the  whole  world? – 4M  person-­‐years  per  month • How  many  hours  per  month  per  Internet  User? – 16  hours  (global  average) – 32  hours  (U.S.  Average) *Sources: Feb.2012 - from Go-Gulf.com compiled from Comscoredatamine.com, Nielsen.com, thisDigitalLife.com, PewInternet.org Tuesday, November 27, 12
  • 25. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? Tuesday, November 27, 12
  • 26. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion Tuesday, November 27, 12
  • 27. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? Tuesday, November 27, 12
  • 28. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? – More  than  250M Tuesday, November 27, 12
  • 29. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? – More  than  250M • Facebook:  Updates  per  day? Tuesday, November 27, 12
  • 30. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? – More  than  250M • Facebook:  Updates  per  day? – More  than  800M Tuesday, November 27, 12
  • 31. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? – More  than  250M • Facebook:  Updates  per  day? – More  than  800M • YouTube:  Views/day Tuesday, November 27, 12
  • 32. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? – More  than  250M • Facebook:  Updates  per  day? – More  than  800M • YouTube:  Views/day – 4  Billion  views – 60  hours  of  video  uploaded  every  minute! Tuesday, November 27, 12
  • 33. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? – More  than  250M • Facebook:  Updates  per  day? – More  than  800M • YouTube:  Views/day – 4  Billion  views – 60  hours  of  video  uploaded  every  minute! • Social  Networks:  users  who  have  used  sites  for  spying  on  their  partners? –56% Tuesday, November 27, 12
  • 34. Volume,  Velocity,  Variety • Google:    How  many  queries  per  day? – More  than  1  Billion • Twi$er:  How  many  Tweets/day? – More  than  250M • Facebook:  Updates  per  day? – More  than  800M • YouTube:  Views/day – 4  Billion  views – 60  hours  of  video  uploaded  every  minute! • Social  Networks:  users  who  have  used  sites  for  spying  on  their  partners? –56% *Sources: Feb.2012 - from Go-Gulf.com compiled from Comscoredatamine.com, Nielsen.com, thisDigitalLife.com, PewInternet.org Tuesday, November 27, 12
  • 35. Turning  the  three  Vs  of  Big  Data  into  Value Tuesday, November 27, 12
  • 36. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? Tuesday, November 27, 12
  • 37. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? Tuesday, November 27, 12
  • 38. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Tuesday, November 27, 12
  • 39. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? Tuesday, November 27, 12
  • 40. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? •  What  is  the  emoEon? Tuesday, November 27, 12
  • 41. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? •  What  is  the  emoEon? •   Is  it  negaEve  or  posiEve? Tuesday, November 27, 12
  • 42. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? •  What  is  the  emoEon? •   Is  it  negaEve  or  posiEve? •   What  is  the  health  of  my  brand  online? Tuesday, November 27, 12
  • 43. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? •  What  is  the  emoEon? •   Is  it  negaEve  or  posiEve? •   What  is  the  health  of  my  brand  online? Do  we  understand  context  and  content? Tuesday, November 27, 12
  • 44. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? •  What  is  the  emoEon? •   Is  it  negaEve  or  posiEve? •   What  is  the  health  of  my  brand  online? Do  we  understand  context  and  content? •  What  are  appropriate  ads? Tuesday, November 27, 12
  • 45. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? •  What  is  the  emoEon? •   Is  it  negaEve  or  posiEve? •   What  is  the  health  of  my  brand  online? Do  we  understand  context  and  content? •  What  are  appropriate  ads? •   Is  it  Ok  to  associate  my  brand  with  this  content? Tuesday, November 27, 12
  • 46. Turning  the  three  Vs  of  Big  Data  into  Value Do  we  understand  what  each  individual  is  trying  to  achieve? • What  is  user  intent? • Cri,cal  in  mone,za,on,  adver,sing,  etc… Do  we  understand  what  a  community’s  sen:ment  is? •  What  is  the  emoEon? •   Is  it  negaEve  or  posiEve? •   What  is  the  health  of  my  brand  online? Do  we  understand  context  and  content? •  What  are  appropriate  ads? •   Is  it  Ok  to  associate  my  brand  with  this  content? • Is  content  sad?,  happy?,  serious?,  informaEve? Tuesday, November 27, 12
  • 47. Case  Studies Yahoo!  Big  Data 18 Tuesday, November 27, 12
  • 48. Yahoo!  –  One  of  Largest  DesEnaEons  on  the  Web More people visited Yahoo! in the past month than: • Use coupons 80%  of  the  U.S.  Internet  popula:on  uses  Yahoo!   • Vote –  Over  600  million  users  per  month  globally! • Recycle • Global  network  of  content,  commerce,  media,  search  and  access  products • Exercise regularly • 100+  properEes  including  mail,  TV,  news,  shopping,  finance,  autos,  travel,  games,  movies,   • Have children living at health,  etc. home • Wear sunscreen regularly • 25+  terabytes  of  data  collected  each  day • RepresenEng  1000’s  of  cataloged  consumer  behaviors Data is used to develop content, consumer, category and campaign insights for our key content partners and large advertisers Sources: Mediamark Research, Spring 2004 and comScore Media Metrix, February 2005. Tuesday, November 27, 12
  • 49. Yahoo!  Big  Data  –  A  league  of  its  own… GRAND CHALLENGE PROBLEMS OF DATA PROCESSING TRAVEL, CREDIT CARD PROCESSING, STOCK EXCHANGE, RETAIL, INTERNET Y! Data Challenge Exceeds others by 2 orders of magnitude Tuesday, November 27, 12
  • 51. Behavioral  TargeEng  (BT) Targe:ng  your  ads  to  consumers  whose   recent  behaviors  online  indicate  that   your  product  category  is  relevant  to  them Tuesday, November 27, 12
  • 52. Yahoo!  User  DNA • On a per consumer basis: maintain a behavioral/interests profile and profitability (user value and LTV) metrics Tuesday, November 27, 12
  • 53. How  it  works  |  Network  +  Interests  +  Modelling Tuesday, November 27, 12
  • 54. How  it  works  |  Network  +  Interests  +  Modelling Varying Product Purchase Cycles Analyze predictive patterns for purchase cycles in over 100 product categories Tuesday, November 27, 12
  • 55. How  it  works  |  Network  +  Interests  +  Modelling Match Users to the Models Analyze predictive patterns for purchase cycles in over 100 product categories In each category, build models to describe behaviour most likely to lead to an ad response (i.e. click). Tuesday, November 27, 12
  • 56. How  it  works  |  Network  +  Interests  +  Modelling Rewarding Good Behaviour Analyze predictive patterns for purchase cycles in over 100 product categories In each category, build models to describe behaviour most likely to lead to an ad response (i.e. click). Score each user for fit with every category…daily. Tuesday, November 27, 12
  • 57. How  it  works  |  Network  +  Interests  +  Modelling Identify Most Relevant Users Analyze predictive patterns for purchase cycles in over 100 product categories In each category, build models to describe behaviour most likely to lead to an ad response (i.e. click). Score each user for fit with every category…daily. Target ads to users who get highest ‘relevance’ scores in the targeting categories Tuesday, November 27, 12
  • 58. Recency  MaPers,  So  Does  Intensity Active now… …and with feeling Tuesday, November 27, 12
  • 59. DifferenEaEon  |  Category  specific  modelling Tuesday, November 27, 12
  • 60. DifferenEaEon  |  Category  specific  modelling Tuesday, November 27, 12
  • 61. DifferenEaEon  |  Category  specific  modelling Tuesday, November 27, 12
  • 62. DifferenEaEon  |  Category  specific  modelling Tuesday, November 27, 12
  • 63. Automobile  Purchase  Intender  Example • A  test  ad-­‐campaign  with  a  major  Euro  automobile  manufacturer – Designed  a  test  that  served  the  same  ad  creaEve  to  test  and  control  groups  on  Yahoo – Success  metric:  performing  specific  acEons  on  Jaguar  website • Test  results:  900%  conversion  lij  vs.  control  group – Purchase  Intenders  were  9  Emes  more  likely  to  configure  a  vehicle,  request  a  price   quote  or  locate  a  dealer  than  consumers  in  the  control  group – ~3x  higher  click  through  rates  vs.  control  group Tuesday, November 27, 12
  • 64. Mortgage  Intender  Example Results from a client campaign on Yahoo! Network Example: Mortgages +626% Conv Lift We found: +122% CTR Lift 1,900,000 people looking Y! Finance Audience Mortgage Loans Target Mortgage Loans Target for mortgage loans. Example search terms qualified for this target: Mortgages Home Loans Refinancing Ditech Example Yahoo! Pages visited: Financing section in Real Estate Mortgage Loans area in Finance Real Estate section in Yellow Pages Source: Campaign Click thru Rate lift is determined by Yahoo! Internal research. Conversion is the number of qualified leads from clicks over number of impressions served. Audience size represents the audience within this behavioral interest category that has the highest propensity to engage with a brand or product and to click on an offer. Date: March 2006 Tuesday, November 27, 12
  • 65. Experience summary at Yahoo! • Dealing with the largest data source in the world (25 Terabyte per day) • BT business was grown from $20M to about $500M in 3 years of investment! • Building the largest database systems: – World’s largest Oracle data warehouse – World’s largest single DB – Over 300 data mart data – Analytics with thousands of KPI’s – Over 5000 users of reports – Largest targeting system in the world • Big demands for grid computing (Hadoop) Tuesday, November 27, 12
  • 66. Social  Network Social  Network  MarkeEng Understanding  Context  for  Ads Tuesday, November 27, 12
  • 67. Case Study: TWITTER Social Marketing? Viacom’s VH1 Twitter campaign on ANVIL (the movie) (week of May 14th , 2009 – see AdAge article) Marketing ANVIL movie Very niche audience, how do you reach them? Tuesday, November 27, 12
  • 68. Case Study: TWITTER Social Marketing? Viacom’s VH1 Twitter campaign on ANVIL (the movie) (week of May 14th , 2009 – see AdAge article) Marketing ANVIL movie Very niche audience, how do you reach them? Diffusion Give 20 free DVD’s to major related artists/groups, ask them To notify Twitter groups – reached over 2M people Tuesday, November 27, 12
  • 69. Case Study: TWITTER Social Marketing? Viacom’s VH1 Twitter campaign on ANVIL (the movie) (week of May 14th , 2009 – see AdAge article) Marketing ANVIL movie Very niche audience, how do you reach them? Diffusion Give 20 free DVD’s to major related artists/groups, ask them To notify Twitter groups – reached over 2M people Social Identity: power of word of mouth... What is the Cost to VH-1? Compare with traditional approach: TV commercials to promote a documentary film? Tuesday, November 27, 12
  • 70. The  Display  Ads  Challenge  Today Damaging to Brand Irrelevant and Damaging to Brand Mismatch Completely Irrelevant Tuesday, November 27, 12
  • 71. NetSeer:  Intent  for  Display • And  Currently  Processing  4  Billion  Impressions  per  Day Tuesday, November 27, 12
  • 72. NetSeer:  Intent  for  Display • And  Currently  Processing  4  Billion  Impressions  per  Day Tuesday, November 27, 12
  • 73. NetSeer:  Intent  for  Display • And  Currently  Processing  4  Billion  Impressions  per  Day Tuesday, November 27, 12
  • 74. NetSeer:  Intent  for  Display • And  Currently  Processing  4  Billion  Impressions  per  Day Tuesday, November 27, 12
  • 75. Problem:  Hard  to  Understand  User  Intent Contextual  Ad  served  by  Google Tuesday, November 27, 12
  • 76. Problem:  Hard  to  Understand  User  Intent Contextual  Ad  served  by  Google What  NetSeer  Sees:   Tuesday, November 27, 12
  • 77. Specialized  Search  through  Big  Data  AnalyEcs  over   the  Offers  Universe Tuesday, November 27, 12
  • 78. Chaos  for  Consumers   Consumer Tuesday, November 27, 12
  • 79. Chaos  for  Consumers   Deep Discount Sites Coupon Flash Sites Sale Sites Loyalty Programs Daily Deals Sites Consumer Online Loyalty Programs Social Search Networks Engines Tuesday, November 27, 12
  • 80. The  Business  Model  behind  ChoozOn’s  Use  of  Big  Data Tuesday, November 27, 12
  • 81. The  Business  Model  behind  ChoozOn’s  Use  of  Big  Data Consumers • Value from brands they love • Tame the deal chaos Tuesday, November 27, 12
  • 82. The  Business  Model  behind  ChoozOn’s  Use  of  Big  Data Consumers • Value from brands they love • Tame the deal chaos Marketers • Reach targeted consumers • Build loyalty • Create brand evangelists Tuesday, November 27, 12
  • 83. The  Business  Model  behind  ChoozOn’s  Use  of  Big  Data Consumers • Value from brands they love • Tame the deal chaos Marketers • Reach targeted consumers • Build loyalty • Create brand evangelists Tuesday, November 27, 12
  • 85. Carol’s  Consumer  Network “Chozen”  Brands Tuesday, November 27, 12
  • 86. Carol’s  Consumer  Network “Chozen”  Brands Interests  (Intent) Tuesday, November 27, 12
  • 87. Carol’s  Consumer  Network “Chozen”  Brands Interests  (Intent) Shopping  Pals Tuesday, November 27, 12
  • 88. Carol’s  Consumer  Network “Chozen”  Brands Interests  (Intent) Shopping  Pals Deal  Clubs Tuesday, November 27, 12
  • 89. What  Carol  Gets Carol’s  Consumer  Network Tuesday, November 27, 12
  • 90. What  Carol  Gets Carol’s  Consumer  Network The  Universe  of  Deals Affiliate  Deals Loyalty  Programs Daily  Deals Flash  Sales Deep  Discounters Intelligent Matching A  Personal  Shopper  for  Deals Tuesday, November 27, 12
  • 91. What  Carol  Gets Carol’s  Consumer  Network The  Universe  of  Deals Affiliate  Deals Loyalty  Programs 1,600+  brands Daily  Deals Flash  Sales 100,000+  offers Deep  Discounters Intelligent Matching A  Personal  Shopper  for  Deals Tuesday, November 27, 12
  • 92. What  Carol  Gets Carol’s  Consumer  Network The  Universe  of  Deals Affiliate  Deals Loyalty  Programs 1,600+  brands Daily  Deals Flash  Sales 100,000+  offers Deep  Discounters PLUS her Inbox™ Intelligent Matching A  Personal  Shopper  for  Deals Tuesday, November 27, 12
  • 93. Thank  You!   TwiPer:  @choozon nick@choozon.net www.ChoozOn.com Tuesday, November 27, 12