The document outlines an agenda for a presentation on measuring communication, providing conceptual frameworks and examples of how to measure communication outputs, outcomes, and stakeholder behaviors and assessments at various stages of the communication planning process. The presentation also provides examples of qualitative and quantitative tools that can be used to gather and analyze measurement data.

1. Measuring Communication -
Simple Approaches That Can
Change Your Image
Presentation to the
Federal Communicators Network
Jeffrey Brooke, ABC
Principal Consultant
Organizational Communication & Change Management
Jbrooke@MITRE.org
P r e s en tati on a p p r ove d f o r P u b l i c Re l e a se :12 -46 11 . Di s t r i bu tio n Un l i mi t ed
*ABC – Accredited in Business Communication by the International Association of Business Communicators
© 2012 The MITRE Corporation. All rights reserved.

2. |2|
Agenda
 Logistics
– 30 minute presentation
– Q&A – via text box
– Presentation will be forwarded
 Illustration - a typical communicator dilemma
 What to measure
– Conceptual framework for measurement
 How to measure
– Other examples and tools

3. |3|
Communication measurement
is a way of thinking
 Measurement is about:
– Gathering data
– To tell a story
– That allows people to make decisions
 Not about:
– Complicated math

4. |4|
Start with internal communication
 Focusing on internal communication today
 An organization’s brand/reputation, comes from the inside out.
Employees treat customers as management treats employees.
– Effective communication improves employee engagement
– Engaged employees deliver better for the customer
 Here’s an example.

5. |5|
Scenario - Call center at “the Bureau”

6. |6|
Call center agent, Jill, answers question and
completes service applications for citizens

7. |7|
Goal: communication with Jill will align her
performance with the Bureau’s goals
Desired
Communication
interaction with
citizens

8. |8|
Head of call centers needs quality improved
(response accuracy)

9. |9|
His e-mails, speeches, newsletters have stressed
quality – but haven’t produced change

10. | 10 |
He wants your help
getting his quality message through
“What other communication should I do?
I thought maybe a special newsletter each month focused
just on quality, like industry best practices.
Then maybe a contest between units and then spotlight
the quality superstars in the newsletter each month.
What do you think?”

11. | 11 |
Stepping back for a moment…
The communication flow – classic model
Communication Flow
Sender Message Channel Receiver Results
Noise
Feedback

12. | 12 |
How does the basic model work in our
large and complex organizations?

13. | 13 |
Organizational roles that tie
to steps in the model
Communication Flow
Sender Message Channel Receiver Results
Noise
Feedback
Communicator Stakeholder Perspective Client Program Owner
Perspective (e.g. employees) Perspective

14. | 14 |
Planned communication in an organization
Steps that operationalize the model
Communicator Stakeholder Perspective Client Program Owner
Perspective (e.g. employees) Perspective
Plan Deploy Frame of Communication Stakeholder Results
Communication Reference Behaviors
Activities Outcomes
Communication Flow

15. | 15 |
Planning & measures are intertwined
Planning starts at the end, works backward
Communicator Stakeholder Perspective Client Program Owner
Perspective (e.g. employees) Perspective
Plan Deploy Stakeholder Communication Stakeholder Results
Communication Assessment Behaviors
Activities Outcomes
(Frame of (Knowledge &
Reference) Attitude)
Planning Flow

16. | 16 |
Communication planning
Explore measures for all steps, not just the end
Communicator Stakeholder Perspective Client Program Owner
Perspective (e.g. employees) Perspective
Plan Deploy Stakeholder Communication Stakeholder Results
Assessment Outcomes Behaviors
(Frame of (Knowledge &
Reference) Attitude)
Planning Flow
Measures Measures Measures Measures Measures Measures

17. | 17 |
How to Measure Communication
Overview of Key Tools
 Qualitative
– Interviews
– Focus Groups (and Structured Group Interviews)
– Polls
 Quantitative
– Surveys
– Counting

18. | 18 |
Qualitative Data Gathering
Top tools Best for Examples
understanding
• Focus groups • Depth—by probing • What people know
• Interviews
• Polls • To understand the • How people feel
range of issues on a about a topic
topic
• Frame of reference
• Story-telling nature of on a topic
dialogue can uncover
forces around
change—enablers
and barriers to
change

19. | 19 |
Quantitative Data Gathering
Top tools Best for Examples
Surveys Prediction • Comparison over
• Breadth—how widely time, and between
shared is an opinion organizations
• Precise, for tracking • Correlation analysis
changes over time. for prediction
Counting Outputs • Number of articles
• Event attendance
• Percentage of
articles dedicated to
each strategy

20. | 20 |
Communication planning
1. Results
Stakeholder Perspective Client Program Owner
Communicator Perspective
(e.g. employees) Perspective
Results
Plan What to measure How to measures
Questions to ask: Examples: Examples:
Outcomes: • Customer satisfaction • Survey
• What needs to change and why? • Budget requirements • Financial statements
• How will you know if you’ve • Congressional • GAO/committee
accomplished your goal? requirement feedback
Outputs: • Number • Counting
• What changes in outputs are needed • Quality • Sampling
to achieve the goals? • Cost • Cost accounting
• How will you measure progress?

21. | 21 |
Results
Outcomes and Outputs
Will be measured by the program owner
 Outcome: Higher satisfaction from
– GAO/Congress
– Citizens
 Output: Higher quality calls

22. | 22 |
Communication planning
2. Targeted stakeholder behavior
Stakeholder Perspective Client Program Owner
Communicator Perspective
(e.g. employees) Perspective
Stakeholder
Behaviors
Plan What to measure How to measure
Questions to ask Examples Examples
• Who needs to do what differently to Actual behaviors Business process
produce the output changes? • Applying new training measures
• How will you measure the behaviors? • Using new procedures
• What will change to enable the new
behaviors? Behavioral indicators Survey questions

23. | 23 |
Behaviors
Not clear
Ron assumes the agents have leeway.
Sees his communication as motivation.

24. | 24 |
What to tell Ron?
One approach
 “Yes! I can help you with a quality-focused newsletter and other
ways to get out your message.”
“AND—
– how do you feel about first letting me do some feedback sessions
to get a sense of what will and won’t work. I might find better
options for you to consider.”

25. | 25 |
Communication planning
3. Stakeholder assessment – change readiness
Stakeholder Perspective Client Program Owner
Communicator Perspective
(e.g. employees) Perspective
Stakeholder
Assessment
1. Frame of
Reference for
the change
Plan What to measure How to measure
Questions to ask the program owner Examples Examples
Client hypothesis about reaction to the Validate with the Focus group
change: stakeholder: Interview
• What changes will enable the new • Will you be able to do
behaviors or remove barriers? this, in light of the
• What is the stakeholder’s frame of planned support?
reference on what you are asking
them to do differently?
• What might they see as enabling or
blocking them?

26. | 26 |
Conduct stakeholder assessment
Focus groups at call centers

27. | 27 |
Finding:
Quality measured quarterly, at the office level

28. | 28 |
Finding: Quantity Metric–
Quantity measured every 30 minutes, at the group level
Every 30 minutes, Group Level

29. | 29 |
Frame of reference determines
what Jill will notice, how she will interpret it
Communication Frame of
Reference
(Enablers & Barriers) Resulting
interaction
with citizens

30. | 30 |
Stakeholder assessment:
Focus groups determine Jill’s frame of reference
Elements of frame of
reference
• Leadership Frame of
• Mission & Vision Reference
• Culture
• Mgt. practices (Enablers & Resulting
Barriers) interaction
• Policies &
procedures with citizens
• Work unit climate
• Individual needs
• Task & individual
skills
(See Burke-Litwin model)

31. | 31 |
What to tell Ron?
 What would you tell Ron?
 What would you recommend
to him as a next step?
 Might this change how Ron
sees you?

32. | 32 |
Communication planning
3. Stakeholder assessment – communication fitness
Stakeholder Perspective Client Program Owner
Communicator Perspective
(e.g. employees) Perspective
Stakeholder “Fitness” measures.
Assessment What do they think of
2. Frame of
Reference on our stuff? Does it “fit”
communication
their needs?”
Plan What to measure How to measure
During deployment: “Fitness” of communication During deployment
How to assess the effectiveness of each
communication activity during Examples: • Survey
deployment. - Clarity • Poll
- Right channels • Focus group
Past experience: - Amount
Usefulness of communication channels - Timing Past experience
and approaches with this stakeholder. - Exposure • Compilation of
previous data
• Communication audit

33. | 33 |
Example:
Ongoing assessment of "fitness" of messages
 Weekly survey of employees and managers
 Hottest topic of the week
 Statistically reliable sample

34. | 34 |
Example "Fitness" measures
Message penetration and usefulness

35. | 35 |
Example "Fitness" measures
Timeliness, satisfaction, importance, amount
Q4. To what extent do you feel
this communication was timely? Timely Not timely
(Running average for "Timely" is
82%)
Q5. Overall, how satisfied were
you with this communication? Satisfied Dissatisfied
(Running average for "Satisfied" is
86%)
Q6. How important is the topic of
"new system procedures" to you? Important Unimportant
(Running average for "Important" is
65%)
Q7. How would you rate the
amount of communication you've About right Too much Too little
received on this topic? (Running
average for "About right" is 68%)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

36. | 36 |
Communication planning
4. Communication outcomes
Stakeholder Perspective Client Program Owner
Communicator Perspective
(e.g. employees) Perspective
Communication
Outcomes
Knowledge &
Attitude
Plan What to measure How to measure
What knowledge and attitude will Examples Examples
produce the targeted behaviors, as seen
through the stakeholder’s frame of • Knowledge and attitudes • Survey
reference? (perceived) • Poll
• Actual knowledge • Focus group

37. | 37 |
Example knowledge measure

38. | 38 |
Example measure of feelings
To what extent do you feel the new Performance Plan
system will help the agency reach its goals?
40%
38%
35%
31%
30%
25% 25%
25% 23%
20%
20% Employees
17%
Sups
15%
13%
10% 9%
5%
0%
0%
A great deal Somewhat A little None I’m not sure yet.

39. | 39 |
Example: Measuring knowledge and feelings
for course correction
 Situation:
– Executive Committee interactive video town hall
 Topic:
– Transformation update – two years into effort
 Update on each of five change strategies
 Scripts weighted heavily on “why this strategy is important”
 Measurement Plan
– Pre-event random survey – two questions for each strategy
 Know: “How important do you feel this is to the agency transformation?”
 Feel: “How effectively do you think this strategy is being implemented?”
– Recommend possible script changes
– Post-event repeat survey

40. | 40 |
Measuring knowledge and feelings
Pre-event survey (two days prior)
Several strategies showed:
• good scores for importance
(feeling)
• lower scores for
implementation (knowledge)
Comments indicated lack of
knowledge about early
implementation efforts and
results.
Updated script for these
strategies to weight more heavily
on pilots and early
implementation efforts.

41. | 41 |
Measuring knowledge and feelings
Post-event survey (one hour after event)
Scripts adjustments were
successful.

42. | 42 |
Communication planning
5. Deploy communication activities
Stakeholder Perspective Client Program Owner
Communicator Perspective
(e.g. employees) Perspective
Deploy
communication
activities
Plan What to measure How to measure
What types of communication will best Communication Outputs Examples
convey the targeted knowledge and
attitudes? Examples: • Counting
• # of articles and events
• % of content supporting
each business strategy
• % of stakeholders who
provide input on a change
• # of awards for a desired
behavior
• # informal leader
conversations

43. | 43 |
Targeting value change with
Primary communication activities
messages
Content, such as intranet
pages, publications and formal
presentations.
Interaction, such as input to
decisions, dialogue with
managers, and collaboration
with colleagues.
Leadership example, such as
how leaders signal what they
value through promotions,
recognition, budget, their time
and informal comments.

44. | 44 |
Output measures
Content analysis – weekly data collection
% of total
editorial
Themes Totals space E-Bulletin Managers Bulletin Magazine Webbie
Total Length 66586 37702 15582 7289 6013
Vision 222 0% 0 0% 222 1% 0 0% 0
Mission 239 0% 239 1% 0 0% 0 0% 0
Values 12026 18% 2249 6% 498 3% 6165 85% 336
Value: Integrity 208 0% 0 0% 0 0% 0 0% 0
Value: Teamwork 10712 16% 2053 5% 498 3% 6165 85% 336
Value: Commitment 1423 2% 196 1% 0 0% 0 0% 0
Value: Dependability 389 1% 0 0% 0 0% 0 0% 0
Goals 39169 59% 16872 45% 11046 71% 2330 32% 1911
Goal 1: Develop a flexible
digital… 3421 5% 1092 3% 0 0% 1284 18% 0
Goal 2: Prepare and
equip… 6601 10% 3694 10% 1076 7% 1046 14% 0
Goal 3: Continue to
transition the… 784 1% 567 2% 0 0% 0 0% 0
Goal 4: To serve the
public's needs by… 892 1% 342 1% 352 2% 0 0% 23
Goal 5: Develop an
integrated enterprise
approach to… 32496 49% 14603 39% 10694 69% 0 0% 1888
Quotes 16629 25% 5112 14% 3973 25% 7090 97% 315
Photos/Images 497 55 9 24 25
188 63 10 60
Total articles or services 786

45. Example output measure: Percentage of total editorial space,
by category, for all six channels
50%
45%
40%
35%
30% 28% 27%
25%
25%
20% 17%
14%
15%
10%
6%
5%
1% 0%
0%
Vision Mission Values Goals Strategic Administrative Recognition Quotes
Priorities

46. Example output measure: Percentage of total editorial space,
by category, for E-Bulletin
50%
45%
40%
35%
30%
25%
25% 23%
19%
20%
15% 13%
10% 8%
5% 3%
1% 1%
0%
Vision Mission Values Goals Strategic Administrative Recognition Quotes
Priorities

47. Example output measure: Percentage of total editorial
space, deep dive in “goals” coverage
50%
45%
40%
35%
30% 28%
25%
25%
20%
15%
10%
5%
5% 3% 2% 2%
0%
Goals Total Goal 1: Develop Goal 2: Prepare Goal 3: Continue Goal 4: To serve Goal 5: Develop
a… and equip… to transition the the public's an enterprise
… needs by… approach to…

48. | 48 |
Communication Measures
Integrated with Communication Planning
Communicator Stakeholder Client Program Owner
Perspective Perspective Perspective
5. 3. 4. 2. 1.
Communi-
cation Tactics to Stakeholder Communication Stakeholder Results
Deploy Assessment Outcomes Behaviors
plan
elements (Frame of
Reference)
Measures Communication Communication Knowledge Specific Outcomes
Outputs Fitness and behaviors and outputs
for each
element Feelings needed
Changes that
will enable the
behaviors
Change
Readiness

49. | 49 |
Does anyone really care?
Measures can move your “brand” from overhead, toward operations.
Communications Executive Council survey: 2012 vs. 2007 data
 Responsibilities of communication function
– #1 increase: Social Media 80% (from zero)
– #2 increase: Measurement 82% (from 64%)
 Top communication strategic priorities for 2013
– #1: Employee engagement
– #2: Improving leadership communication
– #3: Building company presence in social media
 Strategies to improve communication function effectiveness
– #1 Improved consultative partnership with business leaders
– #2 Deeper business acumen on the team
CEC 2012 report: Taking Your Communications Budgeting and Planning to the Next Level

50. | 50 |
What next?
 Deeper topics
– Design and use tools for valid and reliable data
– Communication audits
 Broader topics
– Looking at measures across organizations and time
– What communication function can do to add value
 Feedback on today’s presentation
 Reach out to me and FCN community
 Q&A