This document discusses the importance of monitoring and measurement in public relations and communications. It provides an overview of Katie Delahaye Paine's company, Paine Publishing, which helps communications professionals define and measure success. It discusses the need to move beyond traditional metrics like impressions and earned media value to more meaningful metrics that measure outcomes and impact on business goals. It outlines industry standards and principles for social media measurement developed by the Barcelona Principles project. Finally, it provides a case study of how the Atlantic City Alliance used monitoring and metrics to improve the tone and quality of media coverage about Atlantic City and correlate that coverage with increases in website traffic and changes in public perceptions.

1. Measure What Matters: Monitoring and Measurement
in the New Media World
Katie Delahaye Paine, CEO
Paine Publishing
Jan. 22, 2015
www.painepublishing.com
measurementqueen@gmail.com | @queenofmetrics

2. Paine Publishing:
Providing communications’
professionals the knowledge and
information they need to navigate
their path to perfect measurement
 Newsletters
 Training Courses
 Consulting
 painepublishing.com
Katie Delahaye Paine:
Helping communications
professionals define and
measure success for 25 years.
 Founder of:
 The Delahaye Group
 KDPaine & Partners
 Paine Publishing
 Author of:
 Measuring the Networked Non-Profit
 Measure What Matters
 Measuring Public Relationships

3. Don’t Measure What’s Easy
3
Cartoon by Rob Cottingham

4. Measure What Matters
25/02/2015News Group International 4
Cartoon by Rob Cottingham

5. Welcome to the
“I want what I want, not what
you send me” era
aka: “The Relationship Era” ®

6. Measuring Attribution in an era of all media everywhere
Web Analytics
eCommerce
iCloud &
Google+
Map My Run/Social
Sharing
CRM/Convio

7. Conversations
$$$$$$$$$$$$$$$$$$$$$$$$$$
Savings, shorter cycles, more
renewals, better ideas,
Product
Support HRIR SalesCIR&D
Mktg
Research
Prod.
MktgPR//Mkt
7
Social media isn’t media, it is your business

8. We need a new Attribution Model
ROI
Other
Paid
Marketing
Digital/
Online
Media
Buy
Print/TV
Media
Buy
ROI
Media
Relations
/Social
Media/PR
Digital/
Online
Media
Buy
Print/TV
Media
Buy
Media
Relations
/Social
Media/PR

9. What has changed?
 Collapse of mass media
 Growth of media everywhere
 Intolerance for messaging
 The Barcelona Principles

10. The Barcelona Principles, The Conclave, and Industry Standards
1. Importance of goal setting & measurement
2. Measuring the effect on outcomes is preferred to
measuring outputs
3. The effect on business results can & should be
measured where possible
4. Media measurement requires quantity & quality
5. Earned Media Value/AVEs are not the value of public
relations
6. Social media can & should be measured
7. Transparency & replicability are paramount to sound
measurement
All standards are available on
http://www.painepublishing.com/
standards-central
Conclave Members

11. Social Media Measurement Standards have been published for :
Content Sourcing & Methods1
Reach & Impressions2
Engagement3
Influence & Relevance4
Opinion & Advocacy5
Impact & Value6

12. Eyeball
counting
HITS Outcomes
MSM Online
Social
Media
Impressions are not awareness…
Where’s the “So what?”

13. Likes are not engagement
AdvocacyCommitmentTrial/Consideration
FollowersLikesImpressions

14. • Influence happens when you are persuaded to change behavior or opinion that
would otherwise not have changed.
• Influence can be attributed to either an outlet or an individual.
• Influence happens online and off line and both should be considered
• Should include some combination of the following five elements:
• Reach
• Engagement around individual
• Relevance to topic
• Frequency of posts around the topic
• Audience impact as measured by the ability to get the target audience to
change behavior or opinion.
• If an individual scores a 0 on one element, they have no influence
• Influence cannot be expressed in a single score or algorithm
#4: Influence & Relevance

15. #5 Standards for Measuring Tone/Sentiment
 Whatever process is defined and applied, it must be used
consistently throughout any analysis.
 Sentiment reliability varies by vendor and approach – therefore
coding definitions, consistency and transparency are critical.
 Opinions, recommendations and other qualitative measures are
typically more valuable than raw sentiment and increasingly
measurable:
 Opinions (“It’s a good product”)
 Recommendations (“try it” or “avoid it”)
 Feeling/Emotions (“That product makes me feel happy”)
 Intended action (“I’m going to buy that product tomorrow”)

16. • Impact and value will always be dependent on client objectives
• Need to define outcomes in advance – will likely span multiple business
goals, especially for social (crosses disciplines)
• “ROI” should be strictly limited to measurable financial impact; “total
value” can be used for financial and non-financial impact combination
• Value can be calculated in positive returns (sales, reputation, etc.) or
avoided negative returns (risk mitigated, costs avoided)
• Key performance indicators and balanced scorecards are helpful to
connect social media impact to business results/language
#6: Impact & Value

17. Good Relationships Are More Cost Effective
 Type I love Zappos into Google, and you find
1.19 million references
 Type Citibank and you get 21,000 references.
Citibank spends 100 times more a year on
advertising than Zappos.
 Cost per delegate acquired:
 Obama: $6,024
 Clinton: $147,058
 Romney: $2,389,464

18. ROI
 ROI is:
 Cost savings
 Greater efficiency
 Lower legal costs
 Faster time to implementation
 Lower cost per impact vs. other
marketing channels
 Lower cost per message
exposure vs. other channels
 ROI is not:
 Impressions
 Earned media value or AVE
 Number of shares, views or
likes

19. ACA Measurement Program Parameters
 Goal: Move to Industry Standards Metrics and away
from media value and ad value equivalency
 Measure quality not just quantity
 Began in January 2013
 Established a benchmark: January-March 2013
 Customized “Quality Score” that indexes message
content, tone, and media importance on a scale of
+10 to -10
 Correlated with web traffic to measure outcomes
 Correlated with longitudinal research to determine
impact on preference and intent to visit
 Data used continuously in strategic decision making

20. Definitions of “Success”
 Workshop defined the criteria
 All criteria linked back to the goals:
High-quality media coverage
Intent to visit
Improved reputation
Visits/non-gaming revenue

21. Reality: Negative coverage of AC is constant & growing
0
100
200
300
400
500
600
700
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
Tone of Coverage about Atlantic City Over Time
Negative Neutral Positive
Miss America

22. Solution: ACA Programs generate predominantly positive
coverage
0
20
40
60
80
100
120
140
160
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
Sentiment of ACA Coverage Over Time
Negative Neutral Positive

23. Without ACA events, OCS scores for Atlantic City would
have been significantly lower
4
3.25
2.75
2.99
3.65
2.96
3.36 3.24
2.34 2.37 2.43
1.30
-1.24
0.37
-0.05
0.28 0.28
-1.56
4.91
3.92
2.99
3.58
4.14 4.1 4.27 4.12 4.29
2.78
2.56
1.53
-0.29
0.61
0.20
1.44
1.77
0.63
-2
-1
0
1
2
3
4
5
6
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June
AverageOCSScore Average OCS Score Over Time
Atlantic City OCS without ACA Atlantic City OCS with ACA
The red line represents coverage of Atlantic City minus all mentions of ACA and its
programs

24. ACA has made a demonstrable difference in the quality of
media coverage
 If an ACA or one of its
programs was mentioned
in a media story, it was:
 More likely to contain
an endorsement
 More likely to be
positive
 More likely to contain
key messages
 Less likely to be
negative 0.22
0.00
0.62
1.42
4.61
0.08
0.08
0.01
-0.61
-0.66
Endorsements
Positive Visual
Negative Visual
Tone
Message
Communication
Average of OCS
Differences in Quality of
Coverage with and without
ACA
Atlantic City Atlantic City Alliance

25. OCS scores for Atlantic City significantly improved thanks
to ACA events
36
278
458
334
537
432
342
225
995
329 324
247
139
419
260
476
538
925
4.89
3.92
2.99
3.58
4.14 4.10
4.27 4.12
4.29
2.78
2.56
1.55
-0.29
0.61
0.20
1.44
1.77
0.63
-1.00
0.00
1.00
2.00
3.00
4.00
5.00
6.00
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June
0
200
400
600
800
1000
1200
OCSScore
Numberofitems
AC Mentions vs. OCS Over Time
AC Mentions AC OCS
Dead whale, non-
gaming promotions,
Caesar’s CEO quote,
Lady Antibellum
Revel Casino
value, Electric
Jitney test,
Fake Chips, Phil
Ivey suit, Blake
Shelton
Showboat closing,
Revel facing 2nd
bankruptcy

26. ACA maintains high OCS score with
proactive program coverage Lady Antebellum,
Non-Gambling Fun,
Hello Summer,
GMA segment
14
44 31
93 87 75 75
40
95
28
7
27 17 18 12
107
122
254
5.5
7.18
6.3
5.02
6.24 6.42
7.42
7.21
6.54
5.27
4.25
5.08
6.47
5.94
5.33
4.75
6.87
6.41
0
1
2
3
4
5
6
7
8
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June
0
50
100
150
200
250
300
OCSScore
Numberofmentions
ACA Mentions vs. OCS Over Time
ACA Mentions ACA OCS
Miss America
Statue, ACA
new campaign,
Blake Shelton,
World Poker
Championship

27. 14
39
23
36
133
235
8 7 4 2 10 158 11 1 2 9 2
24
70
21
73
88
110
13
104
74
138
87
48
Jan Feb Mar Apr May Jun
Qtr1 Qtr2
2014
Atlantic City has something for everyone
Atlantic City is a clean and safe place to visit
Atlantic City is a year-round destination spot
Atlantic City is making a comeback
Atlantic City offers more than just gaming
Communication of the “Making a Comeback” & “Something for Everyone” messages
increased as ACA announced new programs

28. Quality of coverage has improved in all markets
4.00
4.25
8.00
3.62
-1.00
4.00
6.50
4.65
5.07
5.88
6.50
5.70
7.17
5.13 5.13
4.67
7.31
7.02
6.25
6.89
5.00
5.29
6.40
7.55
9.00
Jan Feb Mar Apr May Jun
New Markets Yield High Quality Results
Houston Chicago National Baltimore Philadelphia New York DC

29. Free entertainment generated the highest OCS scores in Q2
5.26
5.73
6.36
6.54
6.71
7.15
7.31
7.36
7.50
7.67
8.00
8.10
8.68
9.30
Miss America
Meet AC
DO AC
July 4th Fireworks
Air Show
Blake Shelton
Miss'd America Pageant
Sand Blast
Hello Summer
Boardwalk Hall Light Show
Challenge Triathlon
Sand Sculpting World Cup
Lady Antebellum Concert
Free Entertainment
Top Programs By OCS

30. When ACA programs received media coverage, traffic followed
0
50
100
150
200
250
300
0
20000
40000
60000
80000
100000
120000
140000
Relationship between ACA Program Mentions and Site Visits
Site Visits Program Mentions

31. ACA programs drive higher OCS scores, which correlate
highly with web visits
0.41
0.44
0.47
AC items
ACA Items
ACA OCS Scores
Correlations between Web Visits and PR Metrics
Pearson r. value

32. PR is having a positive impact on preference & perceptions
 People who say they have recently heard news about Atlantic
City are very likely to perceive AC as fun – a key driver of
preference
 Respondents who remember seeing news reports about
Atlantic City are also very likely to associate key messages
and positioning statements such as “place I am excited to go
to” and “fun place to hang out with friends.”
 The same respondents are also more likely to recommend
Atlantic City to friends.

33. 6 steps to standards-compliant
measurement
Step 1: Define your goal(s)
What outcomes is this strategy or tactic going to achieve?
What are your measurable objectives?
Step 2: Define the parameters
Who are you are trying to reach? How do your efforts
connect with those audiences to achieve the goal.
Step 3: Define your benchmarks
Who or what are you going to compare your results to?
Step 4: Define your metrics
What are the indicators to judge your progress?
Step 5: Select your data collection tool(s)
Step 6: Analyze your data.
Turn it into action, measure again
Six Steps
to Success
1
2
3
4
5
6

34. Definitions
 Monitoring – process by which data are systematically and regularly
collected about a program over time.
 Measurement – a way of giving an activity a precise dimension,
generally by comparison to some standard; usually done in a
quantifiable or numerical manner; see also: data, scale
 Metrics – an agreed upon number or percentage that indicates
progress towards a goal
 Evaluation – a form of research that determines the relative
effectiveness of a public relations campaign or program by measuring
program*
*IPR Dictionary of Measurement Terms, Dr. Donald Stacks www.instituteforpr.org

35. Why do we communicate?
Outtakes
(Intermediary Effects)
• Awareness
• Knowledge/Education
• Understanding
Outcomes
(Target Audience Action)
• Revenue
• Leads
• Engagement
• Advocacy
ActivitiesHow does what you do
contribute to the bottom line?

36. Step 1: Define the goals
 What return is expected? – Define in terms of the
mission.
 Define your champagne moment. If you are
celebrating complete 100% success a year from
now, what is different about the organization?
36

37. Goals & Suggested Metrics
Goals
Goal 1: Meet
sales targets
Goal 2: Reduce
risk/threats
Goal 3: Increase
market share in
new market
Communications
Contribution
• Expand the
marketable
universe
• Reduce sales
cycle
• Increase trust
• Increase
advocacy
• Expand the
marketable
universe
Metrics
• % increase in
desirable share
of voice
• % increase in
awareness
• % decrease in
undesirable
voice
• % increase in
trust scores
• % increase in
desirable voice
in new market
• % increase in
awareness of
brand in new
market

38. Step 2: Understand the parameters
What are management’s priorities?
Who are you are trying to reach?
How do your efforts connect with
those audiences to achieve the goal?
What influences their decisions?
What’s important to them?
What makes them act?
38
Goal: Get the cat to stop howling
Options: Local? Cheap? Convenient?
Strategy: Buy cat food

39. Step 3: Establish benchmarks
 Past Performance Over Time
 Think 3
 Whatever keeps your
C-suite up at night

40. Step 4: Pick your Kick-Butt Index
The Perfect KBI
 Is actionable
 Is there when you need it
 Continuously improves your processes
& gets you where you want to go
 You become what you measure, so
pick your KBI carefully
40

41. Defining High Quality Coverage
The OCS Score
Desirable Criteria Score Undesirable Score Score
Positive sentiment 1 Negative sentiment -2
Contains one or more
positive message(s)
3
Contains one or more
negative messages(s)
-3
Event/program is mentioned 2
No event/program is
mentioned
0
Appears in Tier 1 Media 2 Negative mention in Tier 1 -1
Third party endorsement 1 Recommends competition -2
Contains a desirable visual 1 Contains undesirable visual -2
Total Score 10 Total -10

42. Step 5: Pick the right measurement tools
 If you want to measure messaging, positioning,
themes, sentiment:
Content analysis
 If you want to measure awareness, perception,
relationships, preference:
Survey research
 If you want to measure engagement, action,
purchase:
Web analytics
 If you want predictions and correlations
you need two out of three

43. Step 5 continued: Selecting a measurement tool
43
Objective KBI Tool
Increase message
communications
Increase percent of items containing one or
more message(s)
CyberAlert, Prime Research
Increase
awareness/preference
% of audience preferring your brand
to the competition
Survey Monkey, Survata
Engage
marketplace
% increase in engagement on website and/or
social sites
Simply Measured, Unmetric,
Google Analytics, Site Catalyst,
Network Analysis
Become a thought
leader on a specific
issue/market/topic
% increase in desirable share of voice on
issue/market/top
Little Bird, Trackr, CyberAlert

44. Testing the accuracy of coding
48.94%
68.31%
88.64%
0.00% 20.00% 40.00% 60.00% 80.00% 100.00%
SDL
UberVu
NetBase
% Agreement with human coding
SDL UberVu NetBase
33.12%
58.00%
7.84%
11.95%
26.53%
13.00%
0.32%
9.46%
40.35%
30.00%
92.11%
78.00%
SDL
Beyond
NetBase
UberVu
% positive %negative %neutral

45. Everyone agrees on “Neutral”…
But, not the case with “Negative” & “Positive”
100.00%
62.32%
14.82%
72.36%
0.00%
37.68%
85.18%
27.64%
Mixed
Negative
Neutral
Positive
Degree to which agreement occurred
No Yes

46. Step 6: Be data informed, not data driven
 Rank order results from worst to best
 Ask “So What?” at least 3 times
 Put your data into an overall framework
consistent with C-Suite expectations
 Find your “Data Geek”
 Compare to last month, last quarter, 13-month
average

47. Photo Event
High Message Content
Resource Use
Low
High
Medium
Veryhigh
Level of Engagement
VeryHigh
Medium
High
Low
High Resources
No Message
Low Resources
Webinar
Status update
Link
Ultimate
Road Trip
Google + Chat
Media Day
Corporate Video
Advocacy vs. resource use
Resource Use
Low
High
Medium
Veryhigh
VeryHigh
Medium
High
Low
High ResourcesLow Resources
Webinar
Status update
Link
Ultimate
Road Trip
Google + Chat
Media Day
Corporate Video

48. Real-time metrics focus on outcomes

49. 10 Ways to find the money for measurement
1. Don’t call it measurement – It’s Research
2. Consolidate – Social? IR? Agencies? Research?
3. Crowdsource – Who else has tools or surveys?
4. Spread the cost over 2 fiscal years
5. Reach out to universities
6. Take advantage of free stuff
7. Monitor only what matters
8. Narrow your Top Tier media list to what you CAN afford
9. Google Analytics
10. Facebook Insights
11. Twitter Analytics

50. Remember these points:
It’s not about the media, it’s about the business and the customer.1
It’s not about Big Data, it’s about how you use it.2
You need to be data informed, not data-driven.3
It’s not how loud you’re shouting, it’s about relationships.4
Standards are a reality, not an excuse to hide behind.5

51. Thank You!
 For more information on measurement, read my blog:
http://www.painepublishing.com/blog
 For a copy of this presentation or to subscribe to our newsletter, give me
your card or email me at measurementqueen@gmail.com
 Follow me on Twitter: @queenofmetrics
 “Like” Paine Publishing’s Facebook account
 Call me at 1-603-682-0735