Weitere ähnliche Inhalte
Ähnlich wie All Possible Worlds iab mx_public (20)
Kürzlich hochgeladen (20)
All Possible Worlds iab mx_public
- 2. Agenda
• Strategy for Digital Audience Measurement
• Current Methodology and Enhancements
• Impact of Mobile Devices and Tablets
• Fragmentation of Platforms and Consumption
• Towards a Digital Measurement Ecosystem
• User Panels, Site-Centric Tag Measurement, Network-Centric Logs,
Dynamic Empanelment
• Establishing Next Generation Best Practices
• Probabilistic Reference Point and Validation
© comScore, Inc. Proprietary. 2
- 8. The scope of the challenge for Digital Media Measurement
© comScore, Inc. Proprietary. 8
- 9. Digital Requirements
REQUIRE Depth & Representivity
– Endless scope of possible behaviors
– Engagement profiles dominated by heaviest users
– Must capture micro audiences
REQUIRE Integration within broad Media Research environment
– Media ecosystem is bigger than just Digital
– Methods & Practices must synch with those used in other media
REQUIRE full view into the digital consumer across all platforms
– Web-only view no longer represents the digital consumer
REQUIRE ability to integrate methodologies
– Methods that work on one platform may fail on another
– Must integrate ‘best-of-breed’ methodologies for complete measurement
© comScore, Inc. Proprietary. 9
- 10. Key UDM Lessons
Panel Requirements Staggeringly high
– Key segments difficult to represent: Work, Tech Professionals
– Systemic under-measurement of certain behaviors: news, developer networks
– .. efforts to date to build a “perfect panel” have fallen short
Publishers willing to provide Site-centric measurement
– Better measurement is valued. Outweighs cost of tagging burden.
– Ability to ‘close-the-loop’ to reconcile sources reduces transaction costs
Some Ecosystem elements best measured by tag-based collection
– Ad networks, Video & Mobile
– Tags allow for rich event attribution (Ad Net reach, YouTube, Hulu)
– Coverage across all platforms
UDM must evolve in response to feedback
– Simple assumptions must yield to more robust methods (cookie/person est.)
– UDM highlights key gaps in panel methodology (large deltas, apples/oranges)
– Must integrate Site-centric measurements into panel methodology
© comScore, Inc. Proprietary. 10
- 11. Digital Measurement Outlook
Drive next-generation Panel Methodology
– Leverage Site-census measurements
Integrate Digital Measurement into broader media ecosystem
– Adopt industry standards: enumeration, demography, sampling
– Deliver transparency through public audit reporting
– Integrate digital into Agency/Planning infrastructure & 3rd party tools
Move Digital beyond the Web
– Mobile matters. So do Tablets… whether we can build panels or not
– It is our responsibility to deliver measurement
Develop Cross-Platform integration techniques
– Anchor methods to site-centric observations across platforms
– Accommodate platform-specific methods to deliver a Unified view
© comScore, Inc. Proprietary. 11
- 13. Building Blocks of Unified Digital Measurement
Establishment Surveys
Population Targets
Enumeration
Unified Digital Measurement Online Recruitment
UVs and Page Views Panel
Unification
Recruitment
comScore
Unified Digital
Measurement
Passive Meter Data Collection
Traffic Data Session Assignment
Dictionary
Site Hierarchy Allocation Collection
Site Centric tagging
Panel Bias
Correction
Weighting, Preventing Bias,
© comScore, Inc. Proprietary. 13
- 14. Enumeration
Enumeration
Panel
Unification
Recruitment
comScore
Unified Digital
Measurement
Traffic Data
Allocation Collection
Bias
Correction
© comScore, Inc. Proprietary. 14
- 15. Defining the Universe: The Establishment Survey
Accurate estimates of the size and composition of the target populations
are critical to audience measurement.
High quality establishment survey enables comScore to accurately
estimate the target population sizes for in-market age and gender
demographics, penetration, and frequency of use in each market.
comScore currently uses proprietary surveys, high quality government
sources (eg, Eurostat), or the local ‘gold standard’ survey.
For expanded demographics and to avoid mobile-only issues, comScore
has agreed to partner or license gold standard surveys like the NRS in the
UK, PMB Print Measurement Bureau in Canada, The Readership Works
by Ipsos in AU, and The EGM survey in Spain.
comScore currently measures & reports persons age 6+ who accessed the
Internet from either a home or a work computer in the past 30 days.
© comScore, Inc. Proprietary. 15
- 16. Panel Recruitment
Enumeration
Panel
Unification
Recruitment
comScore
Unified Digital
Measurement
Traffic Data
Allocation Collection
Bias
Correction
© comScore, Inc. Proprietary. 16
- 17. Recruitment in the Digital Age
Given the issues of:
– Scalability (the need for panel sizes in the hundreds of thousands to address the long
tail of digital media and fragmentation)
– Declining efficacy of RDD recruiting
comScore recruits panelists online, in two ways:
– The Affiliate Program (“banners-plus-Permission Research”)
– Third Party Application Providers (TAP)
Pioneers of large, non-probability panels
– Panel Dashboard, Jackknife Replication
© comScore, Inc. Proprietary. 17
- 18. Online Recruitment from Two Approaches…
Affiliate Program Third-Party Application Provider (TAP)
Affiliate network comprised of web comScore partners with application
entities which meet comScore's providers who offer visitors a “quid
quality criteria pro quo”
On these sites, panelists are The web user is offered something
recruited via banner ads free (software, applications, utilities
etc.) in exchange for exposure to
Appeals are targeted through a
our panel solicitation
broad array of smaller web entities
The user can generally get the free
Respondents are directed to our
app without joining the panel
online intake entity
www.permissionresearch.com New “Value Proposition” (in Affiliate
and TAP): Trees for Knowledge
© comScore, Inc. Proprietary. 18
- 20. TAP Recruiting Experience
User is on the Internet
searching for software
applications, or some
other download
This user is looking for
“photo animation
software”
© comScore, Inc. Proprietary. 20
- 21. TAP
User gets desired search result and clicks through to site
© comScore, Inc. Proprietary. 21
- 22. TAP – Partner Site
Once on the site, user
decides to initiate the free
download
© comScore, Inc. Proprietary. 22
- 24. TAP – Installation Flow
Accept the License
Agreement of the value
proposition software (this
is the partner’s
agreement)
© comScore, Inc. Proprietary. 24
- 25. TAP – User Acceptance
User is also asked to join
the comScore panel.
This is a separate step
from the partner’s
software acceptance
User must take a positive
action (not pre-selected)
© comScore, Inc. Proprietary. 25
- 26. TAP – Partner-side Demo Collection
Take a short survey where
we collect basic demos
© comScore, Inc. Proprietary. 26
- 27. Value Propositions from TAP Recruitment
A sampling of the many value propositions offered….
PC Utilities and Productivity Desktop Personalization and
Digital Media Applications Games and Entertainment
Tools Appearance
•Alarm Clock •Screensavers •Audio/Video Converters •Arcade Games
•Application Launcher •Icons •Audio Editor •Board Games
•Checklist Software •Images •Audio Extractor •Fantasy Games
•Download Accelerator •Smileys •CD and DVD Burners & •Global Radio & TV Stations
•Download Manager •Emoticons Rippers •Lyric finder
•Media Streamer •Video Converter for •Media Players •Music Downloads
•PC Access Control Screensavers •Media Downloaders •Photo Album
Software •Icon File converter •Media Search Tools •Photo Morphing Software
•PC Customizable Shutdown •Wallpapers / Themes •Media Library/Organizer •Quest Games
•PC DVR •Screen Pen •Ringtones
•TV Optimization Software •Sports Games
•Windows Application/PC •Strategy Games
Lock •Online Games
•File Protection •War Games
•PC Power Saver
•Online Disk Storage
•History Cleaner
•Internet Usage and
Performance Stats
•Unit Converter/Metric
Converter
•Currency
Calculator/Converter
•Customizable Binary
Converter
© comScore, Inc. Proprietary. 27
- 28. Trees for Knowledge
In 2008 comScore launched a new recruitment appeal, for both
Permission Research and TAP panelists
– Under our Trees for Knowledge program, comScore has partnered with
Trees for the Future to plant a tree for new members that join the panel
– comScore’s initial donation will support the planting of one million trees in
developing communities throughout the world
Creative made available to affiliates to promote campaign
Permission Research offer: plant a tree at installation; then one tree for
each month the panelist stays in panel
© comScore, Inc. Proprietary. 28
28
- 29. Passive Data Collection
Enumeration
Panel
Unification
Recruitment
comScore
Unified Digital
Measurement
Traffic Data
Allocation Collection
Bias
Correction
© comScore, Inc. Proprietary. 29
- 30. comScore’s Privacy Protected Proprietary Collection Software with
Explicit Panelist Permission
comScore’s cProxy meter allows us to “see” user activity at the device or screen
side (user experience as opposed to site-centric) without using cookies
■ comScore captures:
– URL
– Engagement (active versus passive)
– Keystroke and mouse activity and intensity
– Information passing between the user and the Internet entity
– Ads delivered, whether clicked or not
– Application installation and usage (Excel, Word etc.)
■ Data capture tiers
– HTTP: comScore can capture just about anything over the HTTP protocol (including HTTP and
HTTPS)
– Proprietary technology for real time measurement of advanced protocols like streaming
– Measurement of AOL proprietary and IM engagement
© comScore, Inc. Proprietary. 30
- 32. Who is Using the Device?
In the digital age, tracking the behavior of devices is easier
than ever before.
Moving from devices (which don’t buy products or view
advertisements) to people still poses unique challenges.
In panel measurement, the ideal is passive observation over
an extended time period.
One approach: use “who are you?” pop-ups.
– Requiring respondents to self-identify each user session
(similar to traditional media construct used in TV People Meter)
– Intrusive
– Fatiguing
– Biasing
Intrusiveness drives panel turnover to unmanageable levels.
© comScore, Inc. Proprietary. 32
- 33. Who is using the device? – Passive Biometric Observation
comScore develops a unique biometric signature for each panelist.
– Session Assignment Technology (SAT): a proprietary, patented technology
that creates biometric “fingerprints” for each person using the device.
Family Roster: Mouse Activity:
Email, Age, Sex… Double Click Speed,
Unique Profile of Each
Mouse Movement
User in a Household
Sharing a Computer
Online Accounts: Keyboard Activity:
Cadence tracking of
100+ keywords
© comScore, Inc. Proprietary. 33
- 34. Bias Correction
Enumeration
Panel
Unification
Recruitment
comScore
Unified Digital
Measurement
Traffic Data
Allocation Collection
Bias
Correction
© comScore, Inc. Proprietary. 34
- 35. Preventing Panel Recruitment Bias
At the recruitment stage, this is handled by 3 strategies:
– Hundreds of diverse incentives
– Over 40 different partners, no access panelists or ‘take a survey’ partners
– Multiple advertising links to partners provide huge reach over each campaign
Provides extensive and broad reach across online universe to minimize
potential recruitment bias.
Additionally, both demographic AND behavioural weighting (based on
census information); the latter removes recruitment bias.
Ultimately, the controls in UDM are census based rather than sample based,
and therefore they overcome any recruitment bias and sampling error of
either a calibration panel or probability sample.
© comScore, Inc. Proprietary. 35
- 36. The Recruitment Dashboard: Online Recruitment Balancing
Recruitment targeting allows more control over marginal recruitment via
development of partner dashboard.
The dashboard uses reports to track partner yield by demographic group,
over time
Current Month Index Last Month Index 2 Months Ago Index Partner1
Target In Tab vs UE In Tab vs UE In Tab vs UE Target In Tab % UE
All_2-5 2.3 2.8 2.7 All_2-5 50 0.8% 2.0%
All_6-11 4.4 5.3 4.9 All_6-11 50 0.8% 8.1%
All_12-14 0 0.0% 5.7%
All_12-14 15.1 43.9 42.7
All_15-17 0 0.0% 6.0%
All_15-17 114.7 154.4 151.6 All_18-24 1,600 26.7% 13.9%
All_18-24 203.4 193.4 188.0 All_25-34 2,000 33.3% 18.0%
All_25-34 122.4 118.0 117.7 All_35-44 1,800 30.0% 18.1%
All_35-44 91.1 92.0 91.6 All_45-49 500 8.3% 8.8%
All_50-54 0 0.0% 6.7%
All_45-49 81.5 79.7 80.8
All_55-64 0 0.0% 7.6%
All_50-54 89.4 78.5 82.3 All_65+ 0 0.0% 5.0%
All_55-64 96.3 82.5 84.4
Total 6,000 100.0% 100.0%
All_65+ 89.8 83.0 90.5
© comScore, Inc. Proprietary. 36
- 37. Traffic Allocation
Enumeration
Panel
Unification
Recruitment
comScore
Unified Digital
Measurement
Traffic Data
Allocation Collection
Bias
Correction
© comScore, Inc. Proprietary. 37
- 38. Web visitation and traffic allocated based on 6 levels of site hierarchy
© comScore, Inc. Proprietary. 38
- 39. Client Focus Dictionary (CFD) – Media Entities
Traffic allocated across
– 118 content categories and sub-categories
– Six levels of hierarchy supporting a “parent/child”
relationship with intelligent grouping of properties
– Ad Networks providing potential and actual reach
– All platforms including internet and mobile
Building block for each level in the hierarchy is a
URL pattern
– Example: %yahoo.com%sports%cricket%
Allows calculation of key inputs to Unified
Digital Measurement at as granular a level as a
URL pattern resulting in
– Accurate estimates of crucial metrics like cookie
deletion, audience overlap etc
– Bias elimination created by grouping different
types of sites into one broad bucket
© comScore, Inc. Proprietary. 39
- 40. Unification
Enumeration
Panel
Unification
Recruitment
comScore
Unified Digital
Measurement
Traffic Data
Allocation Collection
Bias
Correction
© comScore, Inc. Proprietary. 40
- 41. Fundamental Problem of Site Centric Measurement:
No Unique User ID
Web Analytics Approximation
Unique User = Cookie ID (if Cookies can be set) or
IP Address + User Agent
More Problems…
• Cookies are deleted, and when they are, the same user can be counted multiple
times
• IP Addresses change all the time causing inflation of user counts
• In any case, a UU is an attempt to identify a machine (or a browser), which can
represent multiple people or a fraction of the usage of a single person
• Some 56% of all machines had a session on more than one browser, and since
each browser sets its own cookie on a machine, they can’t de-dup.
© comScore, Inc. Proprietary. 41
- 42. The Impact of Cookie Deletion:
Repeat Visitors are counted Multiple Times
The site reads THREE distinct cookies, which means
this ONE visitor is counted THREE times.
© comScore, Inc. Proprietary. 42
- 43. Unified Unique Visitors
UDM combines cookies at census level with panel insights to report unique
visitors for the combined home and work audience.
Methodology:
– Collect 3rd party cookies
– Filter cookie data to include:
User generated calls by excluding bot and spider traffic (IAB lists)
Country specific traffic by excluding traffic outside reportable market
Home, work, mobile and commercial shared use audience
– Use panel to understand cookies per person (explained in the next slide)
Based on entity-specific user behavior
– Some visitors to a entity may not receive a comScore cookie due to non-
tagged portion of the entity
comScore accounts for these visitors using the panel.
– Visitors are then de-duplicated across home and work for both unique visitors
with and without cookies.
© comScore, Inc. Proprietary. 43
- 44. Cookies per Person
Advertisers demand people measurement, not cookies or browsers
Cookies are not people due to duplication at the cookie level caused by
several factors including:
– Users deleting or blocking cookies
– Multiple browsers per device This is what differentiates UDM
from other ‘hybrid’ methodologies;
– Multiple devices per location consistent with Global Industry
– Multiple users on the same device recommendations
comScore calculates cookies per person each month at the site level by
country using panel observations
– This factor is applied to cookies observed for the entity to remove duplication and
derive Unique Persons
Cookies per person may be seen as a function of:
– Site usage (see above)
– Visitation frequency
– Usage intensity
© comScore, Inc. Proprietary. 44
- 46. UDM Traffic collection
We collect traffic (tag counts and cookies) from various
locations, devices and content types that are tagged
• Home
• Work
• Libraries
Locations • Universities
• Internet Cafes
• Other
• PC
• Mac
Devices • Mobile phones
• Tablets
• Other
• PC based web
Content • Mobile web
Types • Apps
• Other
© comScore, Inc. Proprietary. 46
- 47. Total Universe Page Views
Segment Traffic by Total Universe
Location / Device Page Views
Mobile Phones
Shared
Machines
Work
Home
Filtered
Census Traffic
© comScore, Inc. Proprietary. 47
- 48. Traffic Allocation
Segment Traffic by Total Universe
Location / Device Unique Visitors
Home Work
Home +
Work
Mobile Other
Multiple
Shared
Filtered Web Apps Schools use
machines
Census Traffic
Libraries
& Other
© comScore, Inc. Proprietary. 48
- 51. Device Fragmentation
Connected Devices per HH
60%
% of Households 50%
40%
PC Devices
30%
All Devices
20%
10%
0%
1 2 3 4+
# of Devices in HH
Mobile Devices & Tablets are Driving trend in Multi-device Access
© comScore, Inc. Proprietary. 51
- 52. Device Fragmentation
More than 50% Connected Devices per HH
of Internet 60%
Households still
have a Single % of Households 50%
PC
40%
PC Devices
30%
.. But Nearly All Devices
40% of 20%
Internet
Households 10%
have more
than 3 0%
Connected 1 2 3 4+
Devices # of Devices in HH
Mobile Devices & Tablets are Driving trend in Multi-device Access
© comScore, Inc. Proprietary. 52
- 53. Behaviors are different across devices.
PC behaviors do not represent activities across all
devices
© comScore, Inc. Proprietary. 53
- 54. iPhone & iPad User Incremental Usage vs. PC
iPhone & iPad Users: The Platform is
Incremental Reach over PC
the Message:
60.0% 56.8%
50.0%
40.0% Behaviours vary
28.9%
30.0% 26.8%
23.6% 21.7% by platform
20.0% 12.5%
8.7%
10.0% 6.9%
0.2%
0.0%
Information
e-mail
Maps
Business/
Networking
Total Internet
Retail
Sports
Navigation
Finance
Search/
News/
Social
Incremental
2.0x 9.2x 1.6x 1.2x 1.9x 1.3x 2.8x 2.5x 1.4x
Duration
Source: comScore custom research; experimental iPhone/PC Overlap panel
© comScore, Inc. Proprietary. 54
- 55. Trend toward a mutli-platform digital world is just
beginning
© comScore, Inc. Proprietary. 55
- 56. Platform Explosion is just getting started
40%
US Smartphone Penetration: 33.5%
35%
30%
Smartphone Population
25%
Growing Strong
20%
50%+ Annual Growth
15%
10%
US iPad Penetration: 4.4%
5%
0% Nov-2010
Feb-2011
Feb-2010
Mar-2010
Mar-2011
May-2010
Jul-2010
Aug-2010
Sep-2010
Dec-2010
Apr-2010
Oct-2010
Apr-2011
Jan-2010
Jun-2010
Jan-2011
Jun-2011
May-2011
Source: comScore Mobilens US June 2011
© comScore, Inc. Proprietary. 56
- 57. Platform Explosion is just getting started
100%
90%
66% of
80%
Americans
70% DO NOT yet
60% have a
Smartphones
50%
40%
US Smartphone Penetration: 33.5%
30%
20%
US iPad Penetration: 4.4% 95% of
10%
Americans
0% Nov-2010 DO NOT yet
Feb-2011
Feb-2010
Mar-2010
Mar-2011
May-2010
Jul-2010
Aug-2010
Sep-2010
Dec-2010
Apr-2010
Oct-2010
Apr-2011
Jan-2010
Jun-2010
Jan-2011
Jun-2011
May-2011
have an iPad
Source: comScore Mobilens US June 2011
© comScore, Inc. Proprietary. 57
- 58. Unified Digital Measurement™ (UDM)
Global PERSON Global DEVICE
Measurement Measurement
PANEL CENSUS
Unified Digital Measurement (UDM)
Patent-Pending Methodology
Adopted by 80% of Top 100 US Media Properties & 60% in U.K.
© comScore, Inc. Proprietary. 58 V0910
- 60. Technical Measurement Review
Platform Network Publisher Format Demo
Coverage Coverage Coverage Coverage Availability
Metered Panel Poor Very Good Moderate Moderate Very Good
Publisher Tags Very Good Very Good Poor Moderate Poor
Network Census Good Poor Very Good Good Poor
© comScore, Inc. Proprietary. 60
- 61. Technical Measurement Review
Platform Network Publisher Format Demo
Coverage Coverage Coverage Coverage Availability
Metered Panel Poor Very Good Moderate Moderate Very Good
Publisher Tags Very Good Very Good Poor Moderate Poor
Network Census Good Poor Very Good Good Poor
Site Tags provide foundation for full cross platform view.
Drives de-duplication methodology
© comScore, Inc. Proprietary. 61
- 62. Technical Measurement Review
Platform Network Publisher Format Demo
Coverage Coverage Coverage Coverage Availability
Metered Panel Poor Very Good Moderate Moderate Very Good
Publisher Tags Very Good Very Good Poor Moderate Poor
Network Census Good Poor Very Good Good Poor
Panels provide foundation for person-centric
measurement & demographics
© comScore, Inc. Proprietary. 62
- 63. Technical Measurement Review
Platform Network Publisher Format Demo
Coverage Coverage Coverage Coverage Availability
Metered Panel Poor Very Good Moderate Moderate Very Good
Site Tags Very Good Very Good Poor Moderate Poor
Network Census Good Poor Very Good Good Poor
Publisher coverage remains a challenge. Site tags will
remain most accurate measurement
© comScore, Inc. Proprietary. 63
- 65. A Next Generation Digital Measurement System
Industry Oversight Define Universe & Control
& Audits by Independent for Demographic Bias
Agents
High Quality
Enumeration
Survey
Active Panel
Census Panel
Calibration Partner Control
Control Dashboard
Panel
Behavioral In-tab Monthly Composition
Bias Sample
Probability Enhanced
Validation Sample
Panel Selection
Probability Jackknife
Validation Replication
Panel Study
Panel Stability & Calculate
Margin of Error
© comScore, Inc. Proprietary. 65
- 66. Probability Validation Panel: Concept & Explanation
• Non-probability sampling in digital user panels
• Sampling challenges
• Statistical efficiency and effective sample size
• Jackknife replicates, probability and validation
• Empirical method to examine variance in sample data
• Provides standard error, design effect, stability over sample sources
• Assumes EDF is good estimator of PDF, very large samples,…
• PVP triangulates on bias and reliability questions
• Census as a source for behavioral weights
• Panelist cookie consumption versus universe cookie consumption
© comScore, Inc. Proprietary. 66
- 67. Standard Error Study using Jackknife replication
Jackknife replication, related to the more general Bootstrap, is used to
assess the variability of a statistic by examining the variation within the
sample data (vs parametric assumptions), especially with complex
sampling (Efron & Tibshirani, 1994).
Variance of standard comScore key measures estimators (for monthly,
annual, including top 100 properties & associated entities):
– Unique Visitors
– Page Views
– Duration(Minutes)
Purpose is to understand the variance of the sample design, controlling for
comScore’s methodology and processes for any monthly delivery.
Results of the study will be utilized to create a general functional form for
estimating margin of error for any reportable data.
© comScore, Inc. Proprietary. 67
- 68. Unique Visitor replication results: Smoothed polynomial showing
consistency of UV estimates month over month
© comScore, Inc. Proprietary. 68
- 69. Page View replication results: Smoothed polynomial showing
consistency of PV estimates month over month
© comScore, Inc. Proprietary. 69
- 70. Jackknife Validation continued…
Replication study shows consistency of estimates across properties of
various sizes and categories, and across time periods.
Provides empirical data to develop a polynomial function for estimating
margin of error for any reportable data.
Addresses key concern/question about variance estimates from non-
probability samples and their stability over time, given the fluctuation of
sample sources (partners, affiliate sites).
Also powerful to validate bias correction using device census calibration –
there are important assumptions, however:
– But Jackknife assumes EDF is representative of the PDF. While this is
mitigated by large sample sizes, it requires validation…hence, PVP.
© comScore, Inc. Proprietary. 70
- 71. Summary
• Digital Measurement Challenges are unprecedented
• Fragmentation of devices, consumption, attention
• Multi-source and mixed methods are required
• Measurement across platforms and locations
• Site tagging, OS and IP information plus user panels
• Duplication issue across multiple access points
• Differences in patterns of duplication across content types, and demos
• Industry standards must be set; industry participation is crucial
• Probabilistic reference for multi-source measurement
• Guide to industry accepted methodology and best practices
© comScore, Inc. Proprietary. 71