L&D leaders from Crowe LLP – a public accounting, consulting, and technology firm – will share best practices on how they’re moving away from smile sheets and completion metrics to tracking behavior change and performance on the job. View the presentation to learn: - Case example by Crowe LLP on how they’re piloting innovative ways to track employee behavior change and performance post-learning - New Udemy for Business data insights on how learning impacts key business outcomes like employee engagement, productivity, growth, and retention - How to win over key stakeholders when rolling out a new learning metrics approach - Tips and advice on how to get started with measuring the ROI of learning

2. Udemy for Business
Proving the ROI
of Learning
November 15, 2018| 10:00am PST

3. Speaker
Cara Brennan Allamano
Senior Vice President
Human Resources
Udemy
Bob Wagner
Learning Program Leader
Crowe LLP
Julie McCorkle
Learning Project Specialist
Crowe LLP

4. Agenda
1 How learning impacts key business outcomes
2 Crowe LLP: From volume metrics to behavior
change
3 Crowe LLP: Lessons learned
4 Q&A

5. How learning impacts
key business outcomes

6. State of the ROI of learning study
Demographics of survey respondents

7. Most companies feel they are doing a good job
measuring the ROI of learning
How well do companies feel they are measuring the ROI of learning at their organization?

8. However, most companies are still only measuring
satisfaction and completion rates
ROI metrics used by surveyed companies

9. Online learning can boost productivity
More high productivity companies use online learning as their primary L&D resource

10. Employees at high-engagement companies spend
more time learning
Engagement and hours of learning

11. High-engagement companies also spend more
budget on learning
Companies that spend on average $2000-$2500 on learning per employee annually

12. Employees at high-growth companies learn more
Growth and hours of learning

13. • Our survey found no strong link
between employee tenure &
retention and learning
• L&D can’t be offered in isolation
without actual growth
opportunities
Learning alone doesn’t impact employee retention

14. Crowe LLP: From volume
metrics to behavior
change

15. • Crowe LLP is a global public accounting,
consulting, and technology firm with
4000+ employees
• National Association of State Boards of
Accountancy (NASBA) registered
• Previously metrics oriented on ticking the
box on compliance and smile sheets
About Crowe LLP and our training context

16. • Migration: Shifting from
instructor-led training to online
learning
• Technology: Identifying tech-
enabled strategies to support
migration
• Metrics: Tracking the ROI of our
new approach
Crowe University: Reinventing our learning strategy

17. • Use data to inform decision-
making
• Identify learning that does not
have intended impact or “scrap
learning”
• Leverage technology to
demonstrate ROI
Making our learning approach and metrics
to be impact-focused

18. • Pilot Metrics That Matter (MTM)
methodology and tool
• Ask learner + manager 60-90 days
after training: Are you/they
performing any differently in the
workplace?
Shift away from tracking volume metrics to
on-the-job behavior change

19. • Are people increasing the quality
of the services they deliver?
• Are client satisfaction scores
increasing?
• Is employee engagement
increasing?
Metrics we tracked on the MTM pilot project

20. • MTM calculations relate learner-perceived
performance improvement directly to training
• Causation: Survey specifically asks - what
percentage of improvement do you attribute to
learning?
• MTM benchmarks based on their large data set
How we calculate performance improvement
as a result of learning

21. • Previously, we spent time manually
manipulating data
• MTM helped us automate low-value tasks
like data collection, organization, and
reporting
• We can now spend more time on data
insights and sharing with stakeholders
We also automated low-value data collection
tasks with MTM

22. • Training hours per employee
• Training expense per employee
• FTE Satisfaction with training
• External vendor expense
• L&D cost per FTE
Previous ROI metrics focused on volume

23. • Application: How can we increase
application of new skills on the job?
• Results: To what degree will a
learning program improve a specific
business outcome?
• Value: What will be the return on
the learning investment?
But what do our business leaders really care about?

24. Crowe University: Performance-driven metrics

25. Reported Benefit to Cost Ratio: 5.90
For every dollar spent, approximately $6 was
returned in benefit (Benchmark is 3.57)
Our MTM pilot reports focus more on outcomes

26. Crowe LLP: Lessons learned

27. • We had the support of key business leaders on
shifting away from compliance to behavior
change
• MTM process is a significant change
• Number and type of questions changed
significantly – we could have better prepared
our learners and their performance managers
• Some viewed it as taking too much of their
time
Lessons learned: Change the mindset of
stakeholders at every level

28. • Engage program level managers so they can
help drive the process and understand the
data
• Ask managers what they are trying to
accomplish to help you better leverage the
rich data
• Explain the process and value for learners so
they understand the new measurement
process
Lessons learned: Communicate to managers and
learners about new measurement process

29. • MTM data is self-reported by learners and
managers
• Not integrated with other company feedback
systems (e.g. clients, engagement, and
retention data)
• Long term view: integrate all data into one
dashboard
Lessons learned: Data is not all integrated
into one dashboard

30. • Engage your change management
function
• Solicit willing business partners to
embark on the journey with you
• Don’t be afraid to ask business impact
questions of your stakeholders
Final tips: How to get started

31. Q&A
Cara Brennan Allamano
SVP of HR
Udemy
Bob Wagner
Learning Professional
Crowe LLP
Julie McCorkle
Learning Project Manager
Crowe LLP

32. Thank you