A recent study by Brandon Hall Group found nearly 80% of L&D departments have little to no direct impact on improving skills or affecting business performance. So how's the other 20% able to do it? With planning, measurement, and the right technology.
Value Proposition canvas- Customer needs and pains
Â
Learning's Big Data Problem: Measuring & Analyzing Impact Part 2
1. Learningâs Big Data Problem:
Measuring & Analyzing Impact
Part 2
David Wentworth,
Principal Analyst,
Brandon Hall Group
@DavidMWentworth
Mike Rustici,
CEO, Watershed
@mikerustici
Michael Rochelle,
Chief Strategy
Officer
Brandon Hall Group
@MichaelRochelle
2. HCM Research and Advisory Firm
Š Brandon Hall Group 2017 2
FOUNDED IN 1993
Community of 300,000+
10,000 Global Clients
Leading Independent HCM
Research & Analyst Firm
Publishing 2+ Pieces of
Research Each Day
Sample of Our Clients
3. How We Help You
Š Brandon Hall Group 2017 3
MEMBERSHIP
STRATEGIC
CONSULTING
EXCELLENCE
AWARDS
PROFESSIONAL
DEVELOPMENT
Learning
& Development
Talent
Management
Leadership
Development
Talent
Acquisition
Workforce
Management
4. Open Surveys
Š Brandon Hall Group 2017 4
Visit www.brandonhall.com click Open Surveys
If you would like to join a panel of survey takers,
please contact us at success@brandonhall.com
⢠Extended Enterprise Learning 2017
⢠State of Talent Acquisition 2017
⢠HCM Technology 2017
5. How To Ask Questions
Š Brandon Hall Group 2017 5
⢠Submit your questions
or comments about the
discussion to our
presenters using the
Questions tab on your
control panel.
⢠Presentation slides and
giveaways can be found
in the Handouts tab on
your control panel.
Recording & Slides will also be sent out after the webinar.
6. Agenda
The State of Learning Measurement
Moving and Measuring Beyond Formal
How Do We Get Better?
The Role of Learning Standards
Š Brandon Hall Group 2017
10. Measuring Outcomes
Š Brandon Hall Group 2017 10
63%
54%
44%
41%
41%
31%
30%
33%
31%
24%
16%
39%
36%
14%
18%
13%
14%
15%
12%
9%
13%
10%
Individual performance
Employee engagement
Team effectiveness
Business process improvement
Voluntary turnover/retention
Ability to respond to market
Revenue growth
Rate of knowledge transfer
Leadership promotion rates
Profitability
Revenue per employee
Levels 1-3
Levels 4 & 5
11. Model Takeaways
⢠Organizations at all levels measure transactional
learning outcomes: completion, grades, etc.
⢠Organizations at levels 4 and 5 are more focused on
behavioral and organizational outcomes
⢠More than half (54.9%) of Level 1-3 companies say
they donât have the proper metrics to measure
beyond Kirkpatrick Level 2, while only 32% of Levels
4 & 5 say the same.
11Š Brandon Hall Group 2017
13. Kirkpatrick
⢠Does not measure learning duration and uptake
⢠Learner sentiment does not correlate with change
in behavior and performance
⢠Most organizations are at lower/lowest level but
that doesnât mean that something is wrong with
learning measurement
⢠Requires additional level (level 5) which is
enterprise return, hardest to quantify and attain
13Š Brandon Hall Group 2017
14. Kaufman
⢠More like Kirkpatrick plus
⢠Focuses on inputs and processes which should be broken
out and evaluated separately
⢠Does shine a light on how the organization is doing with
developing and delivering learning not just the learner
reaction
⢠Does embody the idea of immediate yet small benefits for
learners
⢠States training does not encompass all forms of learning
⢠Similar approaches to evaluating the application of training
⢠Similar in approach to measuring performance impact
⢠Societal output is a stretch but how your customers benefit
is not
14Š Brandon Hall Group 2017
15. Brinkerhoff Success Case Method
⢠Does focus on connecting learning objectives to
business outcomes
⢠Learner surveys can be effective if constructed
properly
⢠Analyzing data is great if it is the right data
⢠Communicating what works and what doesnât work
is always good but it needs to be accompanied with
a solution
15Š Brandon Hall Group 2017
16. Andersonâs Model for Learning Evaluation
⢠Does emphasize the need for a strong connection
between learning and organizational performance
⢠More of an evaluation model for the overall
learning strategy of the organization not an
individual program
⢠Program does measure stakeholder returns
compared to overall organizational returns for the
long and short term
⢠Contemplates different metrics for measuring
organizational and stakeholder requirements
16Š Brandon Hall Group 2017
17. Key Takeaways
⢠All four models fall short in taking into consideration
the following:
⢠Efficiency and effectiveness coefficients for learning
⢠Multimodal learning approaches
⢠Learning audience variability
⢠Learning environments
⢠Learning content impact vs. learning delivery impact
⢠Require individual learning assessments
⢠Link closely to performance management processes
⢠Regular and frequent measurement intervals for behavioral
change and application of learning
⢠Correlation of learning data to performance data
⢠Manager vs. contributor learning
⢠Team development learning
⢠Unstructured learning or experiential learning
⢠Quality of data being collected
Š Brandon Hall Group 2017
19. âScalable learning is the new reason for
large organizations to existâ
John Hagel
Founder
Deloitte Center for the Edge Innovation
A world of exponential change
20. âWithout data youâre just another person
with an opinionâ
W. Edwards Deming
Management and Systems Legend
Use the data
21. So much data! Where do I start?
Š Watershed 2017 21
categories + complexity
learning analytics
=
26. The role of learning standards
Š Watershed 2017 26
all data
Learning is so much bigger than
the LMS. Track informal and
experiential activities.
xAPI and the LRS present a transformative
technology for learning analytics. Seamlessly
bring all your data together in one platform
and perform real-time analytics tailor made
for L&D.
business impact
Include data about behaviors and
performance in your learning
analytics real time
Create, view and distribute reports
seamlessly in real-time.
27. Š Watershed 2017 27
A next generation
learning ecosystem
brings this all together.
This example from
Visa shows that the
LMS is just one small
part of a modern
learning ecosystem.
A new ecosystem
28. Getting started
Š Watershed 2017 28
01
Gather your data
Start by collecting data in one place
and in a common format. Make it low
friction and ensure you have access.
04
Explore your data
Starting to understand not just âwhatâ
is happening but âwhyâ. Form questions
and see if your data can answer them.
Often the best answer leads to another
another question.
02
Get to know your data
Understand what you have. What is
out there, what is reliable and what is
missing? Do some simple evaluations
and create baselines.
03
Operationalize your data
Automate your way out of âExcel hellâ.
Define some interesting metrics and
KPIs, start monitoring them routinely.
What trends do you notice?
05
Experiment with your data
Start a new learning program with
analytics in mind. Set up well controlled
experiments and A/B tests to validate a
hypothesis. Create a culture of
continuous improvement.
06
Show off your data
Weâve long known that learning is vital,
not we can probe it. Go forth and show
the world!
29. Questions?
Submit your questions
to the presenters via the
questions tab on your
control panel
29
Type your questions here
Š Brandon Hall Group 2017
30. Thank you for
joining us today!
If you have any additional questions, please email us at
success@brandonhall.com.
31. Learn More About Us
Products and services that we offer
Š Brandon Hall Group 2016 31
32. Membership Offers Tailored
Support
RESEARCH ACCESS & EVENTS
⢠Reports
⢠Case Studies, Frameworks & Tools
⢠DataNowŽ & TotalTechŽ
⢠Webinars and Research Spotlights
⢠Annual HCM Conference
ADVISORY SUPPORT
⢠Ask the Expert
⢠1 on 1 Consultations
⢠Research Briefings
⢠Benchmarking
CLIENT SUCCESS PLAN
⢠Your Priorities
⢠Executive Sponsor
⢠Client Associate
⢠Monthly Meetings
Š Brandon Hall Group 2016 32
To learn more about becoming a member or to sign up for a free trial, please email us at
success@brandonhall.com
33. Strategic Consulting Offers
Expert Solution Development
BENCHMARKING
⢠Competitive/Comparative
⢠Maturity Model
⢠Custom Research
STRATEGY
⢠Business Case
⢠Planning
⢠Organization & Governance
TECHNOLOGY SELECTION
⢠Vendor Selection
⢠Architecture Design
⢠Systems Evaluation
DEVELOPMENT & INTEGRATION
⢠Program Design
⢠Assessment
⢠Survey
⢠Process Integration
Š Brandon Hall Group 2016 33
To learn more about consulting, please email us at success@brandonhall.com
34. Why Brandon Hall Group
Š Brandon Hall Group 2016 34
1
2
3
4
5
We offer flexible, high-value memberships that offer
everything you need at a cost you can afford.
Our research is better â more targeted, easily digestible,
focused on business results, predictive, prescriptive.
Our solutions gets better results â research-driven, focused
on the business, simple, targeted, flexible approach.
Our case studies are unique for their diversity
and focus on business results.
Our professional development is practical, tailored
to your needs, and equips you to take action.
35. We Do This ThroughâŚ
Research and Analysis that Define New HCM
Standards
Success Stories from Global Organizations that Show
those Standards in Action
Professional Development that Assesses Current
State and Enables Innovation
Flexible Services to Design and Implement
Transformative Solutions
Recognition that Validates the Transformation
Š Brandon Hall Group 2016 35
36. What Our Clients Have to Say
Brandon Hall Groupâs Excellence Awards allows you
to take a deeper dive and evaluate the programs that
youâve created at a level that you might not have the
opportunity to do⌠They have always been a
cornerstone for me.
- Katrina Williams, Director, Sales Capability
CDW
https://youtu.be/OgdA_tpM55U
Š Brandon Hall Group 2016 36
37. What Our Clients Have to Say
I describe the attention and that customer service
that Iâve experienced as being excellent. They are
very attentive and wherever I need to be connected
to whomever, they are there to help me.
- Joanne Veech, Global Talent Leader
PwC
https://youtu.be/YkY-v8gfW5k
Š Brandon Hall Group 2016 37
38. HCMx Radio
Š Brandon Hall Group 2016 38
THE ONLY PODCAST IN THE HCM ARENA THAT WEAVES CURRENT MARKET
RESEARCH, HR TECHNOLOGY AND INDUSTRY LEADERS INTO EACH EPISODE
To catch up on the latest episodes, visit www.brandonhall.com/hcmxradio or subscribe on iTunes
Hinweis der Redaktion
There are so many things required to achieve scalable learning
Weâre not just talking about dissemination of information we know, but also about the acquisition and discovery to new knowledge in a scalable way
How do we get there.
To achieve scale, we need measurement, we need a culture of constant improvement and we need alignment with business objectives
Because this isnât just about scaling learning, itâs about scaling an organizationâs ability to adopt to the exponential rate of change the digital economy is creating
To achieve scale, we need analytics, we need data
There are so many things required to achieve scalable learning
Weâre not just talking about dissemination of information we know, but also about the acquisition and discovery to new knowledge in a scalable way
How do we get there.
To achieve scale, we need measurement, we need a culture of constant improvement and we need alignment with business objectives
Because this isnât just about scaling learning, itâs about scaling an organizationâs ability to adopt to the exponential rate of change the digital economy is creating
To achieve scale, we need analytics, we need data
You canât manage what you canât measure.
You canât measure what you canât define.
Learning and development analytics donât have to be complicated, as long as you stay organized. Start with the basics and work your way out from there. The simplest way to think of learning analytics is to divide it up into categories and complexities.
Learning experience analytics seek to help you understand more about a particular learning experience or set of learning experiences.
Learner analytics seeks to understand more about a specific person or group of people engaged in activities where learning is one of the outputs.
Learning program analytics seeks to understand how an overall learning program is performing.
The most basic form of analytics is measurementâthe simple act of tracking things and recording values.
Evaluation is the process of trying to make meaning out of the data youâve measured. And does the data mean something good or bad?
Advanced evaluation techniquesâsuch as exploratory and inferential analyticsâ search for relationships within your data and discover powerful insights.
The first three levels help you understand whatâs already happened. Predictive and prescriptive analytics help you understand what will happen in the future.
When categories and complexities are combined, it forms the matrix (or pyramid) of learning analytics. Many organizations master the lowest level of complexity within a specific category and then work their way outwards. Some organizations may reach high levels on one category, but remain in the lower levels for the others.
As long as you can identify where you fall within the pyramid, youâll know where youâre going, which helps you to set goals, determine metrics, and evaluate the maturity of your program. So doesnât this contradict the evaluation methods Michael discussed before? Nopeâthey fall nicely in the middle. Evaluation is result of measuring data and the cause for predictive and prescriptive analysis.
During the last webinar we talked more in depth about xAPI and LRS. As a quick recap, xAPI is a technical specification that makes it easier and cheaper to connect learning technologies and an LRS is the place where information from all of those systems can easily be analyzed to drive a culture of continuous improvement.
During the last webinar we talked more in depth about xAPI and LRS. As a quick recap, xAPI is a technical specification that makes it easier and cheaper to connect learning technologies and an LRS is the place where information from all of those systems can easily be analyzed to drive a culture of continuous improvement.
HOW WE DO IT
We Put Your Business First
We Listen and Respond to Your Needs
We Keep Our Approach Simple, Flexible, Principled
We Communicate