1. An Inch or a Mile?Proven Practices in Measuring Learning ImpactNovember, 2010 KedarKulkarni

2. ESI & KA Strategic Partnership © ESI International, Inc. 2010.

3. ESI © ESI International, Inc. 2010. US Offices Arlington/Washington D.C. Chicago New York San Francisco ESI Partners Japan Poland South Korea Global Offices Australia Canada China Germany Hong Kong India New Zealand Singapore South Africa Spain Sweden Taiwan United Arab Emirates United Kingdom

4. KnowledgeAdvisors & Metrics that Matter® © ESI International, Inc. 2010. KnowledgeAdvisors is ahuman capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions. Experts in metrics that matter

5. Agenda Current market trends Top five challenges for measuring impact Six Proven Practices in Measuring Learning Impact A Note on Informal Learning and Measurement Sustainability Project Management Training—Imagine the Impact to Your Projects (Case study) © ESI International, Inc. 2010.

6. Current Market Trends More focus on demonstrating business outcomes Open to collaborating with partners and suppliers to build and execute measurement plans Focus on transparency and project reporting More focus on how to establish “Current” State Examples: Knowledge / Competencies / Organizational / Portfolio / Project Assessment Tool © ESI International, Inc. 2010.

8. Level 2: Learning

9. Level 3: Transfer

10. Level 4: Results (ROI)Effective Assistance (Coaching/Counseling) by Manager Manager (Sender) Manager (Developer) Assess Learn Reinforce/Validate Pre-course preparation and assessment Intensive learning through the course (Classroom/e-Training) Post-course implementation and assessment Learning and Development by Him/Herself Person (Trainee) Person (Developed)

11. Scrap Learning/Manager Support © ESI International, Inc. 2010. Typical Training without Strong Manager Support COSTS ? % DID NOT TRY IT AT ALL ? % TRIED IT TO SOME EXTENT BUT GAVE UP ? % TRIED IT AND GOT POSITIVE RESULTS UNREALIZED VALUE RETURN The preferred method of optimizing training COSTS ? % DID NOT TRY IT AT ALL ? % TRIED IT TO SOME EXTENT BUT GAVE UP 85%+ TRIED IT AND GOT POSITIVE RESULTS RETURN Source: Robert Brinkerhoff

13. When is learning applied to the job?

14. What are the barriers to impact?

15. What can be done to reduce scrap?Source: KnowledgeAdvisors

16. Current Market Trends— Measuring Learning to Impact Project Performance © ESI International, Inc. 2010.

17. A New Paradigm © ESI International, Inc. 2010. Build on Existing Data Collection Points ROI VOI

18. Determine where you need to go But no matter where you want to go, your must…. © ESI International

19. Add Value— Understand the value Identify value metrics Establish a baseline for each metric—pragmatically Identify leading indicators for each metric Measure continuously and manage your stakeholders © ESI International, Inc. 2010. Influence vs. Interest

20. Top Five Challenges Lack of resources No executive support Lack of qualified people to determine and track metrics Afraid of the results The need has not been there © ESI International, Inc. 2010. www.esi-intl.com/valuesurvey

21. Agenda Current market trends Top five challenges for measuring impact Six Proven Practices in Measuring Learning Impact A Note on Informal Learning and Measurement Sustainability Project Management Training—Imagine the Impact to Your Projects (Case study) © ESI International, Inc. 2010.

22. Showing Business Impact © ESI International, Inc. 2010. Requires actual results data and advanced analysis techniques High Investment Once you determine Business Impact what analysis is appropriate, select a method: Causal Modeling Actual Results Correlations Typically employs surveys Business Impact Templates Advanced Business Results Evaluation Human Capital Approach Smart Sheets Low Investment Complex Simple

23. Smile Sheets  Smart Sheets © ESI International, Inc. 2010. Level 1 REACTION Level 2 Learning Level 3 Job Impact Level 4 Results Level 4 RESULTS MANAGEMENT SUPPORT Level 5 ROI/ROE Evidence of Success

24. Making Decisions With Impact Data © ESI International, Inc. 2010. Cut or improve programs that aren’t providing impact to the business “We need our measurement tools and process now more than ever; if we don’t have access to this data, how will we be able to make intelligent decisions on what to cut?” - KnowledgeAdvisors Client

25. Human Capital Approach Uses improved job performance as the overall measure of business impact Easy to use, practical, scalable, replicable Provides a financial ROI relative to the improvement in human capital performance © ESI International, Inc. 2010.

26. Human Capital Approach (continued) Uses principles of estimation, isolation, and adjustment Assumption is that human capital’s value (i.e. salary) is like any other asset that has expected value that can appreciate or depreciate over time © ESI International, Inc. 2010.

27. Business Results Evaluation Approach Use specific business results as a measure of business impact Uses an advanced survey to estimate improvement in results Leverage technology to make practical and scalable Provides a financial ROI relative to the change in the business result that prompted the training Uses principles of estimation, isolation, and adjustment specific to the business result © ESI International, Inc. 2010.

28. Business Results Evaluation Process © ESI International, Inc. 2010.

29. Case Example: Advanced Business Results © ESI International, Inc. 2010. Background: ERP company performing customer education Business Issue: Show value of training to customers in terms of increased productivity and quality Measurement Solution: Collected 12,770 end of program evaluations within 6 months asking participants to forecast impact Analytics Results: 18.2% estimated increase in productivity from the training (adjusted down for self-reported bias)

30. Business Impact Template © ESI International, Inc. 2010. Monetize Results Estimate the Impact of Outside Factors Compare to Cost of Training

31. Use Business Impact Templates © ESI International, Inc. 2010. Background: Large CPG based in Canada,sales training Business Issue: Visible sales program, requested to link between it and sales results Analytics Solution: Used process to evaluate sales change and impact training had on sales Analytics Results: 7% increase in sales, 20% of the increase was due to training, this was a $74,000 increase per sales person when the training was $5000 per person

32. Use Actual Result Correlations This is a method where actual results can be input or imported at an aggregate, periodic frequency. It is then correlated in time periods against training evaluation data for impact indicators. The example below is from sales data and sale training. © ESI International, Inc. 2010.

33. Causal Modeling High level statistical analysis using Structural Equation Modeling Determines the key factors that influence business impact Causation not correlation The Causal Model can be used to predict future results and determine investments © ESI International, Inc. 2010. WorthwhileInvestment 0.483 0.625 Courseware Quality IndividualLearning 0.337 0.556 0.083 InstructorEffectiveness

34. Performance Optimization © ESI International, Inc. 2010.

35. Actionable Metrics © ESI International, Inc. 2010. Goals: Actual to Goal Benchmarks: Internal or External Dashboards: Concise Data Trends: Time based analysis

36. IMAGINE… © ESI International, Inc. 2010. Online course end evaluations: Level 1 & 2 Predictive Level 3 Predictive Business Impact STANDARD REPORT STANDARD REPORT Online follow-up 60-90 days—with participant and with their manager about b & c above

37. Agenda Current market trends Top five challenges for measuring impact Six Proven Practices in Measuring Learning Impact A Note on Informal Learning and Measurement Sustainability Project Management Training—Imagine the Impact to Your Projects (Case study) © ESI International, Inc. 2010.

39. Quick Poll evaluation at point of use (ex. Amazon-like star rating)

40. Comprehensive periodic evaluation to review quality and impact

44. Data integrity exists 5

45. Agenda Current market trends Top five challenges for measuring impact Six Proven Practices in Measuring Learning Impact A Note on Informal Learning and Measurement Sustainability Project Management Training—Imagine the Impact to Your Projects (Case study) © ESI International, Inc. 2010.

46. Project Management – Bringing Value to the Business Effective and profitable management of complex projects Global consistency and capability Effective use of resources Higher customer satisfaction Increased employee satisfaction Significant competitive advantage © ESI International, Inc. 2010.

48. Increased quality

49. Increased productivity

50. Increased revenue

51. Decreased costs / cycle times

54. Think CSI © ESI International, Inc. 2010. “In the absence of proof, let evidence be enough.” Donald Kirkpatrick

55. Questions?Thank you!November, 2010 Raed S. Haddad, Senior Vice President ESI International