This presentation will share the critical components senior learning leaders are looking for in executive reporting as well as the key elements business leaders expect to see when reviewing metrics from learning organizations. The session will share components of a major executive reporting initiative and offer practical and credible advice to link learning to business outcomes. Learning objectives: Review elements that comprise balanced executive reporting metrics. Discuss core attributes on a learning leader report. Present key components of a business manager report for learning. Offer practical and credible examples to link learning to business impact.

1. Reporting Talent Development
Metrics to Executives
You can listen to today’s webinar using your computer’s speakers
or you may dial into the teleconference.
If you would like to join the teleconference,
please dial 1.408.600.3600 and enter access code 927 649 233 #.
You will be on hold until the seminar begins.
#CLOwebinar

2. Reporting Talent Development
Metrics to Executives
Speaker: Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors Inc.
Moderator Daniel Margolis
Managing Editor
Chief Learning Officer magazine
#CLOwebinar

3. Tools You Can Use
• Q&A
– Click on the Q&A panel (?)
in the bottom right corner
– Type in your question in the
space provided
– Click on “Send.”
#CLOwebinar

4. Tools You Can Use
• Polling
– The poll will appear on the
right side of your screen
– Select the best option for
each question
– Click on “Submit”
#CLOwebinar

5. Frequently Asked Questions
Will I receive a copy of the webinar recording?
YES
Will I receive a copy of the slides?
YES
Please allow up to 2 business days to receive these materials
#CLOwebinar

6. Reporting Talent Development
Metrics to Executives
Daniel Margolis
Managing Editor
Chief Learning Officer magazine
#CLOwebinar

7. Reporting Talent Development
Metrics to Executives
Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors Inc.
#CLOwebinar

8. Reporting Talent Development Metrics to Executives
Jeffrey Berk
Chief Operating Officer

9. Agenda
 Review elements that comprise balanced executive
reporting metrics
Discuss core attributes on a learning leader report
Present key components of a business manager report for
learning
Offer practical yet credible examples to link learning to
business impact
February 7, 2012 © 2011, KnowledgeAdvisors 8

10. About KnowledgeAdvisors
KnowledgeAdvisors will improve learning effectiveness within your
organization.
For organizations that utilize learning
and development to drive business
outcomes
KnowledgeAdvisors offers a learning
measurement solution …
That improves the effectiveness and
business impact of learning.
February 7, 2012 © 2011, KnowledgeAdvisors 9

11. About KnowledgeAdvisors
February 7, 2012 © 2011, KnowledgeAdvisors 10

12. Sample Customers
February 7, 2012 © 2011, KnowledgeAdvisors 11

13. Key Metrics
February 7, 2012 © 2011, KnowledgeAdvisors 12

14. Reporting What’s Important
Outcome Metrics - Business Outcomes
 Employee Contribution Margin
 Productivity
 Employee Retention
 Employee Engagement
 Profitability
Effectiveness Metrics – Program Performance
 Delivery Quality
 Knowledge Gain
 Application to Job
 Business Alignment
 Value
Efficiency Metrics – Cost and Activity
 # of Participants
 % of Employees Trained
 Total L&D Investment
 L&D Investment per Participant
 Courses Utilized
 Classes Conducted
 Instructors Utilized
 Locations Used
February 7, 2012 © 2011, KnowledgeAdvisors 13

15. Strategic, Visible, Costly Programs
Leadership Q1 2010 Q2 2010 Q3 2010 Q4 2010 Q1 2011
Leadership Development
Training`
•Leader Retention Effectiveness 76% 80% 77% 75% 76%
•Employee Engagement Rating
Efficiency Rating 79% 78% 76% 78% 79%
•Employee Retention
Business 74% 74% 70% 70% 71%
•Leader Performance Ratings Outcome Rating
•Leader Fiscal Management
Onboarding Program
•Time to Performance
•Performance Rating
•Employee Engagement
•Participant Confidence
•Competency Level
Sales Training
•Growth Rate
•Profit Margin
•Win Rate
•Time to Close
•Deal Size
February 7, 2012 © 2009 KnowledgeAdvisors
14

16. Talent Development Reporting Principles
 Executive initiative
championed by
KnowledgeAdvisors and
supported by leading
organizations
 Standard framework to
measure learning: efficiency,
effectiveness and outcome
measures
 Common terminology and
metrics
 Inspired by GAAP
http://www.knowledgeadvisors.com/tdrp
© 2011 KnowledgeAdvisors Page: 15

17. Talent Development Reporting Process
 Business Outcomes
– Organizational goals
– L&D’s impact on those goals
 Effectiveness
– Levels 1-5, timeliness
 Efficiency
– Costs
– Volume (participants, programs, classes, hours)
– Ratios
– Utilization rates
– Program and vendor management
February 7, 2012 © 2011, KnowledgeAdvisors 16

18. Talent Development Reporting Principles
Business Executive Learning Executive
Summary Report Reports
Executive
Reports L&D Summary L&D Program L&D Operational
Report Report Report
Business Learning Learning
Statements Outcomes Effectiveness Efficiency
Guiding Principles
Statement Statement Statement
Data Sets Business
Effectiveness Efficiency
Outcomes
Extract, convert and calculate
Standard Measures
Data Learning Other Sources
Sources Evaluation
Financial Data Management (e.g HRIS, ERP,
System
System CRM)
7-Feb-12 © 2011 KnowledgeAdvisors Page: 17

19. Dashboards, Scorecards, Statements
February 7, 2012 © 2011, KnowledgeAdvisors 18

20. L&D Summary Report
February 7, 2012 © 2011, KnowledgeAdvisors 19

21. L&D Executive Report
 Narrative
 Text and Charts
 Multiple Views
 Audience
 L&D Executives
 L&D Management
 Discuss it
 90 minutes
 Quarterly meeting
February 7, 2012 © 2011, KnowledgeAdvisors 20

22. Business Executive Report
 Narrative
 Text and Charts
 Multiple Views
 Audience
 Business Executives
 Line Management
 Discuss it
 90 minutes
 Quarterly meeting
February 7, 2012 © 2011, KnowledgeAdvisors 21

23. Dashboard Case Study
Background: Commercial
insurance company
Business Issue: Visual, concise
place for learning metrics
Analytics Solution: Leveraged
dashboards to build custom, robust
solution
Analytics Results: Real-time
access to impact, cost, cycle-time,
satisfaction, quality and volume
dashboards.
February 7, 2012 © 2011, KnowledgeAdvisors 22

24. Executive Tab
February 7, 2012 © 2011, KnowledgeAdvisors 23

25. Quality Tab
What we learn
 Ensure performance is consistent over time and compare most recent
quarter’s performance to the norm
 Overall survey scores, quarterly fluctuation, scores by curriculum area
 Instructor ratings, identify areas to improve
 Overall instructor & course scores against benchmarks
 Top/bottom rated courses based on courseware scores and associated
comments
February 7, 2012 © 2011, KnowledgeAdvisors 24

26. Volume Tab
What we learn
 Total hours delivered,
number of completions
 Breakout of volume by
business / function and
learning method
(eLearning, Instructor Led)
 Trends in volume over time
 Compare trends in volume
by usage rates / trends of
licensed course libraries,
curriculum areas (are we
buying more than we need?
Are we maximizing our
investment?)
February 7, 2012 © 2011, KnowledgeAdvisors 25

27. Six Approaches to Align Learning to Impact
High Once you determine Requires actual results
Investment Business Impact what data and advanced
analysis is appropriate, analysis techniques
select a method:
Actual
Results Causal
Typically Correlations Modeling
employs Business
surveys Impact
Advanced Templates
Business
Results
Human Evaluation
Capital
Approach
Smart
Sheets
Low Complex
Investment
February 7, 2012 © 2011, KnowledgeAdvisors 26
Simple

28. Smile Sheet to Smart Sheet
February 7, 2012 © 2011, KnowledgeAdvisors 27

29. Smart Sheet in Practice
“We need our measurement tools
 Review programs and process now more than ever; if
that aren’t providing we don’t have access to this data,
how will we be able to make
impact to the intelligent decisions on what to cut?”
business - KnowledgeAdvisors Client
February 7, 2012 © 2011, KnowledgeAdvisors 28

30. Business Results Evaluation Approach
February 7, 2012 © 2011, KnowledgeAdvisors 29

31. Business Impact Template in Practice
Background: Large CPG based in
Canada, sales training
Business Issue: Visible sales
program, requested to link between
it and sales results
Analytics Solution: Used process
to evaluate sales change and
impact training had on sales
Analytics Results: 7% increase in
sales, 20% of the increase was due
to training, this was a $74,000
increase per sales person when the
training was $5000 per person
February 7, 2012 © 2011, KnowledgeAdvisors 30

32. Actual Results Correlations
This is a method where actual results can be input or imported at an
aggregate, periodic frequency. It is then correlated in time periods
against training evaluation data for impact indicators.
The example below is from sales data and sale training.
February 7, 2012 © 2011, KnowledgeAdvisors 31

33. Measurement Success Stories
Executive Program for Government Entity:
An executive leadership program generated more
than double the impact of typical leadership programs
equating to nearly 70% on-the-job application and a 2
to 1 benefit to cost ratio.
Strategic Sales Training
A strategic sales training program generated $74,000
in increased sales per sales person for a $5,000 per
person investment. This data was the catalyst that
convinced management to expand the program to
other sales territories.
Front-Line Staff New Hire Program
An on-boarding program for front-line staff brought
an additional 15,000 hours of increased productivity to
the business as a result of the increased speed to
competency. This analysis helped save a new hire
program that was questioned by management.
February 7, 2012 © 2011, KnowledgeAdvisors 32

34. Business Relevant Additional Examples
 Simple Survey
 Asked high performers to rate their high
impact factors
 92% effective supply chain
 89% informal learning tools
 73% formal learning programs
 32% technology
 14% compensation
 Clearly learning contributes to high
performance Source: Dr. John Sullivan, Presentation at Analytics Symposium
February 7, 2012 © 2011, KnowledgeAdvisors 33

35. Business Relevant Additional Examples
 Basic Correlation
 Correlate customer service scores with use of
learning tools and programs
 Identified high training use CSR (customer
service reps) based on tools used, programs
completed and grouped into high and low use
 Compared CSR customer satisfaction score
with learning use
 Results: Top 10% customer satisfaction were
also in the high learning group, bottom 10%
were in the low learning group. Source: Dr. John Sullivan, Presentation at Analytics Symposium
February 7, 2012 © 2011, KnowledgeAdvisors 34

36. Business Relevant Additional Examples
 Before vs. After
 Hypothesis was that a learning support
tool could improve innovation
 Before the learning tool, 15% of
employees contributed innovation ideas
 After the learning tool, 24% of employees
contributed innovation ideas
 Its easy to see the ‘After’ contribution
Source: Dr. John Sullivan, Presentation at Analytics Symposium
February 7, 2012 © 2011, KnowledgeAdvisors 35

37. Business Relevant Additional Examples
 Control Group (side by side)
 Determine if learning projects influenced
employee retention
 Employees with no learning projects
averaged a turnover rate of 21%
 Employees with 2 or more learning
projects averaged a turnover rate of 2%
 Clearly the control group vs. learning
group shows a difference
Source: Dr. John Sullivan, Presentation at Analytics Symposium
February 7, 2012 © 2011, KnowledgeAdvisors 36

38. Business Relevant Additional Examples
 Take it in, Take it out
 Sales team use of social media learning
tool to see impact on sales results
 Tool put In, sales went up 34%
 Tool taken Out, sales went down 29%
 Essentially, without the tool sales returned
to where they were originally. Clearly the
tool made a difference
Source: Dr. John Sullivan, Presentation at Analytics Symposium
February 7, 2012 © 2011, KnowledgeAdvisors 37

39. Measurement Sustainability
• Strong leadership for driving measurement
1 with governance to support it. • Roles and accountabilities for
metrics are defined with
• Culture supports and drives
2 consistent expectations on
what should be measured.
need for standards and
8 common talent development 2
processes and measurement Roles
• Strong desire to use the data for
7 3 3 • Broad base of skills in
Data
decision making Data
Utilization
Skills / analytics and
Behaviors
• Maturity in how data is reported Utilization 3 measurement and strong
1
and used to improve end to end capabilities in end-to-end
7 talent processes
Leadership
measurement methods
6 4
• Standardization for a consistent Technology Processes
and single point of analytics with 5
6 integration to and from feeder Technolog Standards,
• Established and consistent
Processes processes, technologies and
systems y Tools
4 communications for
measurement
Standards,
• Standards have been created for data collection
Tools
• Data integrity exists
5
February 7, 2012
2/7/2012 © 2010 KnowledgeAdvisors
© 2011, KnowledgeAdvisors 38

40. Events:
Analytics Symposium March 5-7 New Orleans (Executive Reporting)
http://www.knowledgeadvisors.com/events/symposium/
Learning Analytics 2 Day Workshop April 25-26 Chicago
http://www.knowledgeadvisors.com/events/workshops/learning-analytics-2-day-
workshop/
Contact Information
For further information on the information contact:
Jeffrey Berk
Chief Operating Officer
+1 312 676-4411
jberk@knowledgeadvisors.com
www.knowledgeadvisors.com

41. Join Our Next CLO Webinar
Driving Your Organization’s Culture by Building
Learning Relationships for Mutual Benefit
Tuesday, March 22, 2012
CLO Webinars start at 2 p.m. Eastern / 11 a.m. Pacific
Register at www.clomedia.com/events
Join the CLO Network: http://network.clomedia.com/
#CLOwebinar