SlideShare ist ein Scribd-Unternehmen logo
1 von 49
Downloaden Sie, um offline zu lesen
Statistical Challenges in Display Advertising
     Deepak Agarwal
     Director, LinkedIn Relevance Science Labs
     ISBIS 2012
     Bangkok, Thailand, 20th June, 2012
DISCLAIMER



 ―The views expressed in this presentation
are mine and in no way represents the
official position of LinkedIn‖




                  STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Agenda

 Background on Advertising

 Background on Display Advertising
   – Guaranteed Delivery : Inventory sold in futures market
   – Spot Market --- Ad-exchange, Real-time bidder (RTB)


 Statistical Challenges with examples




                            STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
The two basic forms of advertising

1. Brand advertising
  – creates a distinct favorable image


2. Direct-marketing
  –   Advertising that strives to solicit a "direct
      response‖:

      buy, subscribe, vote, donate, etc,                now or
      soon
                                                                     4
                        STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Brand advertising …




                                                                   5
                      STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Sometimes both Brand and Performance




                                                                6
                   STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Web Advertising
There are lots of ads on the web …
100s of billions of advertising dollars
 spent online per year (e-marketer)


                                      7
Online advertising: 6000 ft. Overview




                                                                                  Advertisers
            Ads                Pick
                               ads
           Content                         Ad Network
User
                                         Examples:
                                         Yahoo, Google,
                                         MSN, RightMedia,
           Content                       …
           Provider
                     STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Web Advertising: Comes in different flavors


 Sponsored (―Paid‖ ) Search
  – Small text links in response to query to a search engine


 Display Advertising
  – Graphical, banner, rich media; appears in several contexts like
    visiting a webpage, checking e-mails, on a social network,….

  – Goals of such advertising campaigns differ
       Brand Awareness
       Performance (users are targeted to take some action, soon)
          – More akin to direct marketing in offline world


                             STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Paid Search: Advertise Text Links




                  STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Display Advertising: Examples




                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Display Advertising: Examples




                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
LinkedIn company follow ad




                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Brand Ad on Facebook




           STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Paid Search Ads versus Display Ads


Paid Search                             Display
 Context (Query) important              Reaching desired audience

 Small text links                       Graphical, banner, Rich media
                                              – Text, logos, videos,..
 Performance based                      Hybrid
    – Clicks, conversions                     – Brand, performance


 Advertisers can cherry-pick            Bulk buy by marketers
  instances                                   – But things evolving
                                                      Ad exchanges, Real-time
                                                       bidder (RTB)



                                STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Display Advertising Models

 Futures Market (Guaranteed Delivery)
  – Brand Awareness (e.g.
    Gillette, Coke, McDonalds, GM,..)


 Spot Market (Non-guaranteed)
  – Marketers create targeted campaigns
      Ad-exchanges have made this process efficient
         – Connects buyers and sellers in a stock-market style market

      Several portals like LinkedIn and Facebook have self-serve
       systems to book such campaigns



                            STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Guaranteed Delivery (Futures Market)

 Revenue Model: Cost per ad impression(CPM)
   Ads are bought in bulk targeted to users based on
    demographics and other behavioral features
       GM ads on LinkedIn shown to “males above 55”
        Mortgage ad shown to “everybody on Y! ”


  Slots booked in advance and guaranteed
     – “e.g. 2M targeted ad impressions Jan next year”
     – Prices significantly higher than spot market
        – Higher quality inventory delivered to maintain mark-up




                         STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Measuring effectiveness of brand advertising


 "Half the money I spend on advertising is wasted; the trouble is, I don't know
  which half." - John Wanamaker

 Typically
    – Number of visits and engagement on advertiser website
    – Increase in number of searches for specific keywords
    – Increase in offline sales in the long-run

 How?
    – Randomized design (treatment = ad exposure, control = no exposure)
    – Sample surveys
    – Covariate shift (Propensity score matching)

 Several statistical challenges (experimental design, causal inference
  from observational data, survey methodology)


                                   STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Example of an opportunity in this area




                      STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Guaranteed delivery


  Fundamental Problem: Guarantee impressions (with overlapping
   inventory)
                                                       1. Predict Supply
   Young                           US                  2. Incorporate/Predict Demand
                                                       3. Find the optimal allocation
           4       2           1
                                                             • subject to supply and
                   3                                           demand constraints
               2           2


                       1                          si
     LI
  Homepage                 Female                                    xij
                                                                                               dj



                                        STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Example

                                    Supply Pools

Young                 US            US, Y, nF
                                    Supply =                              Demand
    4         2       1
                                    2
              3
          2                         Price = 1                                 US & Y
                  2
                                                                              (2)
         1                           US, Y, F
LI
           Female                    Supply =
Homepage
                                     3
                                     Price = 5
        Supply Pools



                                         How should we distribute
                                         impressions from the supply pools to
                                         satisfy this demand?
                           STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Example (Cherry-picking)

 Cherry-picking:                          Supply Pools
  Fulfill demands at least cost
                                           US, Y, nF
                                           Supply =                              Demand
                                                                        (2)
                                           2
                                           Price = 1                                 US & Y
                                                                                     (2)
                                            US, Y, F
                                            Supply =
                                            3
                                            Price = 5



                                                How should we distribute
                                                impressions from the supply pools to
                                                satisfy this demand?
                                  STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Example (Fairness)

 Cherry-picking:                          Supply Pools
  Fulfill demands at least cost
                                           US, Y, nF
 Fairness:
                                           Supply =                              Demand
  Equitable distribution of                                             (1)
                                           2
  available supply pools
                                           Cost = 1                                  US & Y
                                                                       (1)           (2)
                                            US, Y, F
                                            Supply =
                                            3
                                            Cost = 5

 Agarwal and
  Tomlin, INFORMS, 2010
 Ghosh et al, EC, 2011                         How should we distribute
                                                impressions from the supply pools to
                                                satisfy this demand?
                                  STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
The optimization problem
 Maximize Value of remnant inventory (to be sold in spot market)
   – Subject to ―fairness‖ constraints (to maintain high quality of
     inventory in the guaranteed market)
   – Subject to supply and demand constraints



 Can be solved efficiently through a flow program

 Key statistical input: Supply forecasts




                                                                           24
                              STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Various component of a
Guaranteed Delivery system
OFFLINE                                      Field Sales
     COMPONENTS                                   Team, sells
                                                             Advertisers
                                                   Products
 Supply
                                                  (segments)
forecasts     Admission
               Control
              should the new                   Contracts signed,
  Demand      contract request                 Negotiations involved
forecasts &     be admitted?
               (solve VIA LP)
  booked
 inventory
                                                             Pricing
                                                             Engine

                     STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
ONLINE SERVING




 Stochastic Supply
                                      Allocation                                 Opportunity
                        Near Real
 Contract Statistics      Time                            On Line Ad
                                         Plan              Serving
                       Optimization
Stochastic Demand                     (from LP)                                    Ads




                                  STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
High dimensional Forecasting

 Supply forecasts important input required both at booking time
  (admission control) and serving time
 Problem: Given historical time series data in a high dimensional
  space (trillions of combinations), forecast number of visits for an
  arbitrary query for a future time horizon
    – E.g.: Male visits from Bangkok on LinkedIn next year in January


 Challenging statistical problem
    – Curse of dimensionality & massive data
    – arbitrary query subset
    – latency constraints

         Forecasting High-dimensional data, Agarwal et al, SIGMOD, 2011



                               STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Spot Market
Unified Marketplace (Ad exchange)

 Publishers, Ad-networks, advertisers participate together in a singe
  exchange
                          Advertisers
                      Sports Accessories                         Online Education
Car Insurance
                submit ads to the network

                            Intermediaries

                       display ads for the network
www.cars.com                                                           www.elearners.com
                        www.sportsauthority.com

                           Publishers
 Clearing house for publishers, better ROI for advertisers, better
  liquidity, buying and selling is easier


                                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Overview: The Open Exchange

                                               Bids $0.75 via Network…
                         Bids $0.50
          Bids $0.60



                                                                                       Ad.com
AdSense




                                                                                   Bids $0.65—WINS!



            Has ad
            impression
            to sell --
                                      … which becomes
            AUCTIONS
                                      $0.45 bid



 Transparency and value
                                        STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Unified scale: Expected CPM

 Campaigns are CPC, CPA, CPM

 They may all participate in an auction together

 Converting to a common denomination
  – Requires absolute estimates of click-through rates
    (CTR) and conversion rates.

  – Challenging statistical problem


                       STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Recall problem scenario on Ad-exchange
              Response rates
              (click, conversion,                                   Bids
 conversion   ad-view)
                                            Auction
                 Statistical
                 model




                                                                                    Advertisers
                                    Select argmax f(bid, rate)
                 Click
                                     Pick
               Ads                  best ads

              Page                                    Ad
User                                                Network


              Publisher  STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Statistical Issues in Conducting Auctions

 f(bid, rate) (e.g. f = bid*rate)
    – Response rates (Click-rate, conversion rate) to be estimated
 High dimensional regression problem

                    F(y | o = (i, c, u), j)

    Opportunity=(publisher, context, user)                   ad


 Response obtained via interaction among few heavy-tailed
  categorical variables (opportunity and ad)
    – Total levels for categorical variables : millions and changes over time
    – Response rate: very small (e.g. 1 in 10k or less)


                                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Data for Response Rate Estimation

 Covariates
  – User Xu : Declared, Inferred (e.g. based on tracking, could
    have significant measurement error) (xud, xuf)


  – Publisher Xi: Characteristics of publisher page
       (e.g. Business news page? Related to Medicine industry? Other
        covariates based on NLP of landing page)


  – Context Xc: location where ad was shown,device, etc.

  – Ad Xj: advertiser type, campaign keywords, NLP on ad
    landing page



                            STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Building a good predictive model

 We can build f(Xu, Xi, Xc, Xj ) to predict CTR
   – Interactions important, high-dimensional regression problem
   – Methods used (e.g. logistic with Lasso, Ridge)
         Billions of observations, hundreds of millions of covariates
          (sparse)


 Is this enough? Not quite
   – Covariates not enough to capture interactions, modeling
     residual interactions at resolution of ads/campaign important

   – Variable dimension: New ads/campaigns routinely introduced,
     old ones disappear (runs out of budget)



                               STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Factor Model to reduce dimension of
parameters




Model Fitting based on an MCEM algorithm
Scales up in a distributed computing environment
More details: Agarwal et al, WWW 2012
                                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Exploiting hierarchical structure
Model Setup
                  baseline

Po,j =       f( Xo                  xj            ) λij
                                                      residual
                    i                  j


   E = ∑( f(xi, xu,xc xj) (Expected clicks)
     ij    u,c)




   Sij ~ Poisson(Eij λij)

                        STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Hierarchical Smoothing of residuals

 Assuming two hierarchies (Publisher and advertiser)
                                                       Advertiser
Pub type
                                                        Account-id

    Pub                                              campaign
                      cell z = (i,j)                 Ad

               (   Sz, Ez, λz)
                                                                              Advertiser
                              Pub type
                                                                              Account-id

                                Pub                                         campaign
                                                       z                    Ad
                                               (Sz, Ez, λz)


                               STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Spike and Slab prior on random effects
 Prior on node states: IID Spike and Slab prior




    – Encourage parsimonious solutions
         Several cell states have residual of 1

    – Agarwal and Kota, KDD 2010, 2011




                             STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Random projections (Langford et al, ICML 2008)

 Project all features (covariates as well as
  ad, publisher, campaign ids) to a lower dimension
  subspace through sparse random projections
   – Preserves inner-products between covariate vectors
     approximately


 Learn logistic using stochastic gradient descent on
  massive amounts of data

 Open source software available (Vowpal Wabbit)




                           STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Computation at serve time
 At serve time (when a user visits a website), thousands of qualifying
  ads have to be scored to select the top-k within a few milliseconds

 Accurate but computationally expensive models may not satisfy
  latency requirements
    – Parsimony along with accuracy is important

 Typical solution used: two-phase approach
    – Phase 1: simpler but fast to compute model to narrow down the
      candidates
    – Phase 2: more accurate but more expensive model to select top-k

 Important to keep this aspect in mind when building models
    – Model approximation: Langford et al, NIPS 08, Agarwal et al, WSDM
      2011


                              STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Need uncertainty estimates

 Goal is to maximize revenue
   – Unnecessary to build a model that is accurate
     everywhere, more important to be accurate for things that
     matter!

   – E.g. Not much gain in improving accuracy for low ranked ads



 Sequential design problem (explore/exploit)
   – Spend more experimental budget on ads that appear to be
     potentially good (even if the estimated mean is low due to small
     sample size)



                            STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Explore/Exploit Problem
      (Robbins, Gittins, Whittle, Lai, Berry, Auer, ….)

      There is positive utility in showing ads that currently
       have low mean but high uncertainty
      E.g. Consider 2 ads (same bids)
                      – Goal: Select most popular
                      – CTR1 ~ (mean=.01,var=.1), CTR2~ (mean=.05,var~0)


                                    Ad 2
Probability density




                                                   If we only take a single decision,
                                                        give 100% visits to Ad 2
                        Ad 1
                                                   If we take multiple decisions in the future,
                                                        explore Ad 1 since true CTR1
                                                        may be larger.
                                           CTR


                                               STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Heuristics used in practice

 For a given opportunity, compute priority for
  each ad independently and rank them
   – Priority quantifies future ad potential in the face of uncertainty

 Upper confidence bound policy (UCB)
   – Mean + uncertainty-estimate
       mean + k* sd(estimator)


 Thompson sampling (1930s)
   – randomization by drawing samples from the posterior
        Simple when working in a Bayesian framework



                              STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
Advanced advertising Eco-System
 New technologies
   – Real-time bidder: change bid dynamically, cherry-pick users
           – Track users based on cookie information
           – New intermediaries: sell user data (BlueKai,….)
           – Many sites ―pixelated‖, they are ―watching you‖


   – Demand side platforms: single unified platform to buy
     inventories on multiple ad-exchanges

   – Optimal bidding strategies (around 10 companies, many more
     brewing up)




                              STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
To Summarize
 Display advertising is an evolving and multi-billion dollar industry
  that supports a large swath of internet eco-system
 Plenty of opportunities for statistics
    –   High dimensional forecasting that feeds into optimization
    –   Measuring brand effectiveness
    –   Estimating rates of rare events in high dimensions
    –   Sequential designs (explore/exploit) requires uncertainty estimates
    –   Constructing user-profiles based on tracking data
    –   Targeting users to maximize performance
    –   Optimal bidding strategies in real-time bidding systems

 New challenges
    – Mobile ads, Social ads
 At LinkedIn
    – Job Ads, Company follows, Hiring solutions


                                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
This is our time, let us take the leap
and become data entrepreneurs!




                 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK

Weitere ähnliche Inhalte

Andere mochten auch

How Market Mix Modeling Can Impact Cross-Channel Budget and Business Planning
How Market Mix Modeling Can Impact Cross-Channel Budget and Business PlanningHow Market Mix Modeling Can Impact Cross-Channel Budget and Business Planning
How Market Mix Modeling Can Impact Cross-Channel Budget and Business PlanningVivastream
 
Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...
Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...
Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...iMedia Connection
 
Innovations in Market Mix Modelling
Innovations in Market Mix Modelling Innovations in Market Mix Modelling
Innovations in Market Mix Modelling Absolutdata Analytics
 
Multi Channel Attribution - Driving Marketing Spend Planning In The Big Data Age
Multi Channel Attribution - Driving Marketing Spend Planning In The Big Data AgeMulti Channel Attribution - Driving Marketing Spend Planning In The Big Data Age
Multi Channel Attribution - Driving Marketing Spend Planning In The Big Data AgeAbsolutdata Analytics
 
Digital Marketing Strategies for Financial Services
Digital Marketing Strategies for Financial ServicesDigital Marketing Strategies for Financial Services
Digital Marketing Strategies for Financial ServicesBackbase
 
Advertising concepts & principles
Advertising concepts & principlesAdvertising concepts & principles
Advertising concepts & principlesmanjushachute
 
Online Advertising
Online AdvertisingOnline Advertising
Online Advertisingamalrains
 
Albun para nilsen
Albun para nilsenAlbun para nilsen
Albun para nilsenmilena5439
 
1er trimestre informàtica
1er trimestre informàtica1er trimestre informàtica
1er trimestre informàticaMaite Bernaus
 
חוברת קורס
חוברת קורסחוברת קורס
חוברת קורסStas Segin
 
Digital Virgo Corporate Presentation
Digital Virgo Corporate PresentationDigital Virgo Corporate Presentation
Digital Virgo Corporate Presentationsdesferet
 
STEP Inspire: Non-Profit Assignment
STEP Inspire: Non-Profit AssignmentSTEP Inspire: Non-Profit Assignment
STEP Inspire: Non-Profit Assignmentmdc5070
 
Week 2 presentation
Week 2 presentationWeek 2 presentation
Week 2 presentationRyan Dippre
 
中心运营及管理计划
中心运营及管理计划中心运营及管理计划
中心运营及管理计划yongnianlou
 
Excess hair growth
Excess hair growthExcess hair growth
Excess hair growthhpinn
 

Andere mochten auch (19)

How Market Mix Modeling Can Impact Cross-Channel Budget and Business Planning
How Market Mix Modeling Can Impact Cross-Channel Budget and Business PlanningHow Market Mix Modeling Can Impact Cross-Channel Budget and Business Planning
How Market Mix Modeling Can Impact Cross-Channel Budget and Business Planning
 
Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...
Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...
Insight Address: "Cross-Media Marketing Synergies: Spreading Dollars Across C...
 
Innovations in Market Mix Modelling
Innovations in Market Mix Modelling Innovations in Market Mix Modelling
Innovations in Market Mix Modelling
 
Multi Channel Attribution - Driving Marketing Spend Planning In The Big Data Age
Multi Channel Attribution - Driving Marketing Spend Planning In The Big Data AgeMulti Channel Attribution - Driving Marketing Spend Planning In The Big Data Age
Multi Channel Attribution - Driving Marketing Spend Planning In The Big Data Age
 
Digital Marketing Strategies for Financial Services
Digital Marketing Strategies for Financial ServicesDigital Marketing Strategies for Financial Services
Digital Marketing Strategies for Financial Services
 
Advertising concepts & principles
Advertising concepts & principlesAdvertising concepts & principles
Advertising concepts & principles
 
Online Advertising
Online AdvertisingOnline Advertising
Online Advertising
 
Albun para nilsen
Albun para nilsenAlbun para nilsen
Albun para nilsen
 
leren lezen gr 3, thema Verzamelen
leren lezen gr 3, thema Verzamelenleren lezen gr 3, thema Verzamelen
leren lezen gr 3, thema Verzamelen
 
1er trimestre informàtica
1er trimestre informàtica1er trimestre informàtica
1er trimestre informàtica
 
72 pat2
72 pat272 pat2
72 pat2
 
חוברת קורס
חוברת קורסחוברת קורס
חוברת קורס
 
Digital Virgo Corporate Presentation
Digital Virgo Corporate PresentationDigital Virgo Corporate Presentation
Digital Virgo Corporate Presentation
 
Rising juniors
Rising juniorsRising juniors
Rising juniors
 
STEP Inspire: Non-Profit Assignment
STEP Inspire: Non-Profit AssignmentSTEP Inspire: Non-Profit Assignment
STEP Inspire: Non-Profit Assignment
 
Week 2 presentation
Week 2 presentationWeek 2 presentation
Week 2 presentation
 
中心运营及管理计划
中心运营及管理计划中心运营及管理计划
中心运营及管理计划
 
Excess hair growth
Excess hair growthExcess hair growth
Excess hair growth
 
outsourcing_TnD
outsourcing_TnDoutsourcing_TnD
outsourcing_TnD
 

Ähnlich wie Statistical Challenges in Display Advertising

42629 lecture 11 pt 3
42629 lecture 11 pt 342629 lecture 11 pt 3
42629 lecture 11 pt 3Tom Howard
 
The role of marketing
The role of marketingThe role of marketing
The role of marketingKhánh Nhân
 
5 Trends in Economic Development You Can't Ignore
5 Trends in Economic Development You Can't Ignore5 Trends in Economic Development You Can't Ignore
5 Trends in Economic Development You Can't IgnoreGIS Planning
 
Module 4 strategy & planning final
Module 4 strategy & planning finalModule 4 strategy & planning final
Module 4 strategy & planning finalVivastream
 
Content Measurement: Quantifying Content Contribution - Content2Conversion Co...
Content Measurement: Quantifying Content Contribution - Content2Conversion Co...Content Measurement: Quantifying Content Contribution - Content2Conversion Co...
Content Measurement: Quantifying Content Contribution - Content2Conversion Co...G3 Communications
 
Brand Asset Valuation
Brand Asset ValuationBrand Asset Valuation
Brand Asset ValuationChappy_02
 
Venture Capital Investment Pitch Deck Template - Leave Behind
Venture Capital Investment Pitch Deck Template - Leave BehindVenture Capital Investment Pitch Deck Template - Leave Behind
Venture Capital Investment Pitch Deck Template - Leave BehindJace Grebski
 
10 Tips on Multi-media Marketing for Small Businesses
10 Tips on Multi-media Marketing for Small Businesses10 Tips on Multi-media Marketing for Small Businesses
10 Tips on Multi-media Marketing for Small BusinessesRhonaB
 
Presentation given at Kinexions12
Presentation given at Kinexions12Presentation given at Kinexions12
Presentation given at Kinexions12Lora Cecere
 
LinkedIn Recruitment Ads
LinkedIn Recruitment AdsLinkedIn Recruitment Ads
LinkedIn Recruitment Adsshkumar
 
Unit - 3_Strategic Management (18MBA25)_ External Analysis
Unit - 3_Strategic Management (18MBA25)_ External AnalysisUnit - 3_Strategic Management (18MBA25)_ External Analysis
Unit - 3_Strategic Management (18MBA25)_ External AnalysisVijay K S
 
Endorphin Platform - Social Scoring Platform.
Endorphin Platform - Social Scoring Platform.Endorphin Platform - Social Scoring Platform.
Endorphin Platform - Social Scoring Platform.Artem Zavyalov
 
2012 URISA Track, Creating Jobs with GIS, Wayne Kocina
2012 URISA Track, Creating Jobs with GIS, Wayne Kocina2012 URISA Track, Creating Jobs with GIS, Wayne Kocina
2012 URISA Track, Creating Jobs with GIS, Wayne KocinaGIS in the Rockies
 
Analogue Dollars vs. Digital Pennies
Analogue Dollars vs. Digital PenniesAnalogue Dollars vs. Digital Pennies
Analogue Dollars vs. Digital PenniesAchillesMedia
 
Paid search competitive analysis
Paid search competitive analysisPaid search competitive analysis
Paid search competitive analysisAdlucent
 
Presentation in chicago s&opie event version4
Presentation in chicago s&opie event version4Presentation in chicago s&opie event version4
Presentation in chicago s&opie event version4Lora Cecere
 
Towni combined pitch-032813
Towni combined pitch-032813Towni combined pitch-032813
Towni combined pitch-032813John Shields
 

Ähnlich wie Statistical Challenges in Display Advertising (20)

42629 lecture 11 pt 3
42629 lecture 11 pt 342629 lecture 11 pt 3
42629 lecture 11 pt 3
 
The role of marketing
The role of marketingThe role of marketing
The role of marketing
 
5 Trends in Economic Development You Can't Ignore
5 Trends in Economic Development You Can't Ignore5 Trends in Economic Development You Can't Ignore
5 Trends in Economic Development You Can't Ignore
 
Module 4 strategy & planning final
Module 4 strategy & planning finalModule 4 strategy & planning final
Module 4 strategy & planning final
 
Content Measurement: Quantifying Content Contribution - Content2Conversion Co...
Content Measurement: Quantifying Content Contribution - Content2Conversion Co...Content Measurement: Quantifying Content Contribution - Content2Conversion Co...
Content Measurement: Quantifying Content Contribution - Content2Conversion Co...
 
Brand Asset Valuation
Brand Asset ValuationBrand Asset Valuation
Brand Asset Valuation
 
Venture Capital Investment Pitch Deck Template - Leave Behind
Venture Capital Investment Pitch Deck Template - Leave BehindVenture Capital Investment Pitch Deck Template - Leave Behind
Venture Capital Investment Pitch Deck Template - Leave Behind
 
Social Targeting M6D
Social Targeting M6DSocial Targeting M6D
Social Targeting M6D
 
10 Tips on Multi-media Marketing for Small Businesses
10 Tips on Multi-media Marketing for Small Businesses10 Tips on Multi-media Marketing for Small Businesses
10 Tips on Multi-media Marketing for Small Businesses
 
Presentation given at Kinexions12
Presentation given at Kinexions12Presentation given at Kinexions12
Presentation given at Kinexions12
 
LinkedIn Recruitment Ads
LinkedIn Recruitment AdsLinkedIn Recruitment Ads
LinkedIn Recruitment Ads
 
Unit - 3_Strategic Management (18MBA25)_ External Analysis
Unit - 3_Strategic Management (18MBA25)_ External AnalysisUnit - 3_Strategic Management (18MBA25)_ External Analysis
Unit - 3_Strategic Management (18MBA25)_ External Analysis
 
Endorphin Platform - Social Scoring Platform.
Endorphin Platform - Social Scoring Platform.Endorphin Platform - Social Scoring Platform.
Endorphin Platform - Social Scoring Platform.
 
2012 URISA Track, Creating Jobs with GIS, Wayne Kocina
2012 URISA Track, Creating Jobs with GIS, Wayne Kocina2012 URISA Track, Creating Jobs with GIS, Wayne Kocina
2012 URISA Track, Creating Jobs with GIS, Wayne Kocina
 
Analogue Dollars vs. Digital Pennies
Analogue Dollars vs. Digital PenniesAnalogue Dollars vs. Digital Pennies
Analogue Dollars vs. Digital Pennies
 
Paid search competitive analysis
Paid search competitive analysisPaid search competitive analysis
Paid search competitive analysis
 
Presentation in chicago s&opie event version4
Presentation in chicago s&opie event version4Presentation in chicago s&opie event version4
Presentation in chicago s&opie event version4
 
Competitive intelligence, How to get it...
Competitive intelligence, How to get it...Competitive intelligence, How to get it...
Competitive intelligence, How to get it...
 
Towni combined pitch-032813
Towni combined pitch-032813Towni combined pitch-032813
Towni combined pitch-032813
 
4ps
4ps4ps
4ps
 

Mehr von Deepak Agarwal

Aiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversionAiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversionDeepak Agarwal
 
Recsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and DeepakRecsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and DeepakDeepak Agarwal
 
Statistical Models for Massive Web Data
Statistical Models for Massive Web DataStatistical Models for Massive Web Data
Statistical Models for Massive Web DataDeepak Agarwal
 

Mehr von Deepak Agarwal (7)

Aiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversionAiinpractice2017deepaklongversion
Aiinpractice2017deepaklongversion
 
Recsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and DeepakRecsys2016 Tutorial by Xavier and Deepak
Recsys2016 Tutorial by Xavier and Deepak
 
kdd2015
kdd2015kdd2015
kdd2015
 
Enar short course
Enar short courseEnar short course
Enar short course
 
ENAR short course
ENAR short courseENAR short course
ENAR short course
 
Statistical Models for Massive Web Data
Statistical Models for Massive Web DataStatistical Models for Massive Web Data
Statistical Models for Massive Web Data
 
Bdpanel
BdpanelBdpanel
Bdpanel
 

Kürzlich hochgeladen

Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 

Kürzlich hochgeladen (20)

Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 

Statistical Challenges in Display Advertising

  • 1. Statistical Challenges in Display Advertising Deepak Agarwal Director, LinkedIn Relevance Science Labs ISBIS 2012 Bangkok, Thailand, 20th June, 2012
  • 2. DISCLAIMER ―The views expressed in this presentation are mine and in no way represents the official position of LinkedIn‖ STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 3. Agenda  Background on Advertising  Background on Display Advertising – Guaranteed Delivery : Inventory sold in futures market – Spot Market --- Ad-exchange, Real-time bidder (RTB)  Statistical Challenges with examples STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 4. The two basic forms of advertising 1. Brand advertising – creates a distinct favorable image 2. Direct-marketing – Advertising that strives to solicit a "direct response‖: buy, subscribe, vote, donate, etc, now or soon 4 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 5. Brand advertising … 5 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 6. Sometimes both Brand and Performance 6 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 7. Web Advertising There are lots of ads on the web … 100s of billions of advertising dollars spent online per year (e-marketer) 7
  • 8. Online advertising: 6000 ft. Overview Advertisers Ads Pick ads Content Ad Network User Examples: Yahoo, Google, MSN, RightMedia, Content … Provider STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 9. Web Advertising: Comes in different flavors  Sponsored (―Paid‖ ) Search – Small text links in response to query to a search engine  Display Advertising – Graphical, banner, rich media; appears in several contexts like visiting a webpage, checking e-mails, on a social network,…. – Goals of such advertising campaigns differ  Brand Awareness  Performance (users are targeted to take some action, soon) – More akin to direct marketing in offline world STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 10. Paid Search: Advertise Text Links STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 11. Display Advertising: Examples STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 12. Display Advertising: Examples STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 13. LinkedIn company follow ad STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 14. Brand Ad on Facebook STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 15. Paid Search Ads versus Display Ads Paid Search Display  Context (Query) important  Reaching desired audience  Small text links  Graphical, banner, Rich media – Text, logos, videos,..  Performance based  Hybrid – Clicks, conversions – Brand, performance  Advertisers can cherry-pick  Bulk buy by marketers instances – But things evolving  Ad exchanges, Real-time bidder (RTB) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 16. Display Advertising Models  Futures Market (Guaranteed Delivery) – Brand Awareness (e.g. Gillette, Coke, McDonalds, GM,..)  Spot Market (Non-guaranteed) – Marketers create targeted campaigns  Ad-exchanges have made this process efficient – Connects buyers and sellers in a stock-market style market  Several portals like LinkedIn and Facebook have self-serve systems to book such campaigns STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 17. Guaranteed Delivery (Futures Market)  Revenue Model: Cost per ad impression(CPM) Ads are bought in bulk targeted to users based on demographics and other behavioral features GM ads on LinkedIn shown to “males above 55” Mortgage ad shown to “everybody on Y! ” Slots booked in advance and guaranteed – “e.g. 2M targeted ad impressions Jan next year” – Prices significantly higher than spot market – Higher quality inventory delivered to maintain mark-up STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 18. Measuring effectiveness of brand advertising  "Half the money I spend on advertising is wasted; the trouble is, I don't know which half." - John Wanamaker  Typically – Number of visits and engagement on advertiser website – Increase in number of searches for specific keywords – Increase in offline sales in the long-run  How? – Randomized design (treatment = ad exposure, control = no exposure) – Sample surveys – Covariate shift (Propensity score matching)  Several statistical challenges (experimental design, causal inference from observational data, survey methodology) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 19. Example of an opportunity in this area STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 20. Guaranteed delivery  Fundamental Problem: Guarantee impressions (with overlapping inventory) 1. Predict Supply Young US 2. Incorporate/Predict Demand 3. Find the optimal allocation 4 2 1 • subject to supply and 3 demand constraints 2 2 1 si LI Homepage Female xij dj STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 21. Example Supply Pools Young US US, Y, nF Supply = Demand 4 2 1 2 3 2 Price = 1 US & Y 2 (2) 1 US, Y, F LI Female Supply = Homepage 3 Price = 5 Supply Pools How should we distribute impressions from the supply pools to satisfy this demand? STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 22. Example (Cherry-picking)  Cherry-picking: Supply Pools Fulfill demands at least cost US, Y, nF Supply = Demand (2) 2 Price = 1 US & Y (2) US, Y, F Supply = 3 Price = 5 How should we distribute impressions from the supply pools to satisfy this demand? STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 23. Example (Fairness)  Cherry-picking: Supply Pools Fulfill demands at least cost US, Y, nF  Fairness: Supply = Demand Equitable distribution of (1) 2 available supply pools Cost = 1 US & Y (1) (2) US, Y, F Supply = 3 Cost = 5  Agarwal and Tomlin, INFORMS, 2010  Ghosh et al, EC, 2011 How should we distribute impressions from the supply pools to satisfy this demand? STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 24. The optimization problem  Maximize Value of remnant inventory (to be sold in spot market) – Subject to ―fairness‖ constraints (to maintain high quality of inventory in the guaranteed market) – Subject to supply and demand constraints  Can be solved efficiently through a flow program  Key statistical input: Supply forecasts 24 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 25. Various component of a Guaranteed Delivery system
  • 26. OFFLINE Field Sales COMPONENTS Team, sells Advertisers Products Supply (segments) forecasts Admission Control should the new Contracts signed, Demand contract request Negotiations involved forecasts & be admitted? (solve VIA LP) booked inventory Pricing Engine STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 27. ONLINE SERVING Stochastic Supply Allocation Opportunity Near Real Contract Statistics Time On Line Ad Plan Serving Optimization Stochastic Demand (from LP) Ads STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 28. High dimensional Forecasting  Supply forecasts important input required both at booking time (admission control) and serving time  Problem: Given historical time series data in a high dimensional space (trillions of combinations), forecast number of visits for an arbitrary query for a future time horizon – E.g.: Male visits from Bangkok on LinkedIn next year in January  Challenging statistical problem – Curse of dimensionality & massive data – arbitrary query subset – latency constraints  Forecasting High-dimensional data, Agarwal et al, SIGMOD, 2011 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 30. Unified Marketplace (Ad exchange)  Publishers, Ad-networks, advertisers participate together in a singe exchange Advertisers Sports Accessories Online Education Car Insurance submit ads to the network Intermediaries display ads for the network www.cars.com www.elearners.com www.sportsauthority.com Publishers  Clearing house for publishers, better ROI for advertisers, better liquidity, buying and selling is easier STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 31. Overview: The Open Exchange Bids $0.75 via Network… Bids $0.50 Bids $0.60 Ad.com AdSense Bids $0.65—WINS! Has ad impression to sell -- … which becomes AUCTIONS $0.45 bid Transparency and value STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 32. Unified scale: Expected CPM  Campaigns are CPC, CPA, CPM  They may all participate in an auction together  Converting to a common denomination – Requires absolute estimates of click-through rates (CTR) and conversion rates. – Challenging statistical problem STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 33. Recall problem scenario on Ad-exchange Response rates (click, conversion, Bids conversion ad-view) Auction Statistical model Advertisers Select argmax f(bid, rate) Click Pick Ads best ads Page Ad User Network Publisher STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 34. Statistical Issues in Conducting Auctions  f(bid, rate) (e.g. f = bid*rate) – Response rates (Click-rate, conversion rate) to be estimated  High dimensional regression problem F(y | o = (i, c, u), j) Opportunity=(publisher, context, user) ad  Response obtained via interaction among few heavy-tailed categorical variables (opportunity and ad) – Total levels for categorical variables : millions and changes over time – Response rate: very small (e.g. 1 in 10k or less) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 35. Data for Response Rate Estimation  Covariates – User Xu : Declared, Inferred (e.g. based on tracking, could have significant measurement error) (xud, xuf) – Publisher Xi: Characteristics of publisher page  (e.g. Business news page? Related to Medicine industry? Other covariates based on NLP of landing page) – Context Xc: location where ad was shown,device, etc. – Ad Xj: advertiser type, campaign keywords, NLP on ad landing page STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 36. Building a good predictive model  We can build f(Xu, Xi, Xc, Xj ) to predict CTR – Interactions important, high-dimensional regression problem – Methods used (e.g. logistic with Lasso, Ridge)  Billions of observations, hundreds of millions of covariates (sparse)  Is this enough? Not quite – Covariates not enough to capture interactions, modeling residual interactions at resolution of ads/campaign important – Variable dimension: New ads/campaigns routinely introduced, old ones disappear (runs out of budget) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 37. Factor Model to reduce dimension of parameters Model Fitting based on an MCEM algorithm Scales up in a distributed computing environment More details: Agarwal et al, WWW 2012 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 39. Model Setup baseline Po,j = f( Xo xj ) λij residual i j E = ∑( f(xi, xu,xc xj) (Expected clicks) ij u,c) Sij ~ Poisson(Eij λij) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 40. Hierarchical Smoothing of residuals  Assuming two hierarchies (Publisher and advertiser) Advertiser Pub type Account-id Pub campaign cell z = (i,j) Ad ( Sz, Ez, λz) Advertiser Pub type Account-id Pub campaign z Ad (Sz, Ez, λz) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 41. Spike and Slab prior on random effects  Prior on node states: IID Spike and Slab prior – Encourage parsimonious solutions  Several cell states have residual of 1 – Agarwal and Kota, KDD 2010, 2011 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 42. Random projections (Langford et al, ICML 2008)  Project all features (covariates as well as ad, publisher, campaign ids) to a lower dimension subspace through sparse random projections – Preserves inner-products between covariate vectors approximately  Learn logistic using stochastic gradient descent on massive amounts of data  Open source software available (Vowpal Wabbit) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 43. Computation at serve time  At serve time (when a user visits a website), thousands of qualifying ads have to be scored to select the top-k within a few milliseconds  Accurate but computationally expensive models may not satisfy latency requirements – Parsimony along with accuracy is important  Typical solution used: two-phase approach – Phase 1: simpler but fast to compute model to narrow down the candidates – Phase 2: more accurate but more expensive model to select top-k  Important to keep this aspect in mind when building models – Model approximation: Langford et al, NIPS 08, Agarwal et al, WSDM 2011 STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 44. Need uncertainty estimates  Goal is to maximize revenue – Unnecessary to build a model that is accurate everywhere, more important to be accurate for things that matter! – E.g. Not much gain in improving accuracy for low ranked ads  Sequential design problem (explore/exploit) – Spend more experimental budget on ads that appear to be potentially good (even if the estimated mean is low due to small sample size) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 45. Explore/Exploit Problem (Robbins, Gittins, Whittle, Lai, Berry, Auer, ….)  There is positive utility in showing ads that currently have low mean but high uncertainty  E.g. Consider 2 ads (same bids) – Goal: Select most popular – CTR1 ~ (mean=.01,var=.1), CTR2~ (mean=.05,var~0) Ad 2 Probability density If we only take a single decision, give 100% visits to Ad 2 Ad 1 If we take multiple decisions in the future, explore Ad 1 since true CTR1 may be larger. CTR STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 46. Heuristics used in practice  For a given opportunity, compute priority for each ad independently and rank them – Priority quantifies future ad potential in the face of uncertainty  Upper confidence bound policy (UCB) – Mean + uncertainty-estimate  mean + k* sd(estimator)  Thompson sampling (1930s) – randomization by drawing samples from the posterior  Simple when working in a Bayesian framework STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 47. Advanced advertising Eco-System  New technologies – Real-time bidder: change bid dynamically, cherry-pick users – Track users based on cookie information – New intermediaries: sell user data (BlueKai,….) – Many sites ―pixelated‖, they are ―watching you‖ – Demand side platforms: single unified platform to buy inventories on multiple ad-exchanges – Optimal bidding strategies (around 10 companies, many more brewing up) STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 48. To Summarize  Display advertising is an evolving and multi-billion dollar industry that supports a large swath of internet eco-system  Plenty of opportunities for statistics – High dimensional forecasting that feeds into optimization – Measuring brand effectiveness – Estimating rates of rare events in high dimensions – Sequential designs (explore/exploit) requires uncertainty estimates – Constructing user-profiles based on tracking data – Targeting users to maximize performance – Optimal bidding strategies in real-time bidding systems  New challenges – Mobile ads, Social ads  At LinkedIn – Job Ads, Company follows, Hiring solutions STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK
  • 49. This is our time, let us take the leap and become data entrepreneurs! STATISTICAL CHALLENGES IN DISPLAY ADVERTISING, ISBIS2012, BANGKOK