Courtesy of Intrepid by Vital Source

1. Digital Learning Sample Metrics
What you should be looking for to know if you’ve created something truly engaging.

2. It’s not just about completions.
It’s about so much more.
The new ecosystem you are creating will give you access to much more data than you’re used to from your
traditional LMS. With a proper measurement strategy, you’ll have a much better understanding of your users
and will be able to analyze needs faster, as well as make better content and design decisions.
Remember, every organization has different needs and the ecosystems you create will be different. You may not
be able to capture all of the information listed in these examples, but having an idea of what measurements you
will need early on will help you make more informed decisions going forward.

3. 3
Past and
Present
As you lead your organization through this digital transformation, another thing to keep in mind
is how you measure success. How we measure success today is different than we used to in
measure it in the past, so it’s important to adjust your mindset for that. Here’s a quick reminder.
Physical
presence in
class
Completed
an eLearning
(clicked to the end)
Created an
action plan to use on
the job after an ILT
Passed
a Quiz
Improvement in
scores from Pre and
Post tests
How we used to measure learning
How we measure learning now
Engagement
with content
and peers
Meeting
completion
criteria (points,
badges, etc.)
Did users
demonstrate
evidence of on the
job application during
the experience?
Were users able to
generate and
share personal
insights from the
content?
Did users
create a
working job
deliverable?

4. Use Web
Analytics
Involvement, Interaction, Intimacy and Influence define “engagement” in Web Analytics, and
translate well for engagement in digital experiences.
Category Definition Metric
Involvement Measurable aspects of an individual learner’s
connection to the digital experience
• Frequency of visits
• Average time of session
Interaction Depth / breadth of learner connection to the digital
experience
• Content page views per visit
• Number of pieces of content accessed
• Return rate per piece of content
• Search volume
Intimacy Direct sentiment of a learner towards the digital
experience
• Net promoter score
• Feedback collected
Influence Indicators that a learner will encourage the learning
of others
• Discussion forum participation rate
• Public posting of assignments
• Number of “likes” given to content

5. Learner
Activity
Another way to look at measuring engagement is to follow the flow of learner activity and
measure every step along the way.
Invitation to
join/register (email)
Acceptance/
Registration action
completed
(email/other system)
Course launch email
sent/ auto enrollment
link sent
Learner accesses
site/class and logs in
Learner clicks on
content but doesn’t
complete it
Learner completes at
least one content item
Learner accesses
multiple types of
content
Learner interacts with
others (discussions,
projects, feedback,
support)
Learner meets
completion criteria
Learner applies on the
job
Awareness
andInterest
Consumption
Social Completion Application

6. Awareness
and Interest
Here are some things you can track, and what you can do with that information.
What to Track Why track it
Total # of logins How many people have logged in vs. the total population? This could be helpful in seeing if
your communications or marketing campaign was effective. If the logins are low, perhaps you
need a different round of communications, or a high profile leader to push your initiative so
that it reaches more people. Think of new ways to attract people.
# of logins per day / avg.
length of visit
How often are users logging in? Multiple times a day? Once a week? Once a month? How
long are they staying once they are there? This type of information can be helpful in multiple
ways. If people aren’t logging in, why? Poor communication or marketing of the program? For
users accessing multiple times per day, what are they using it for?
Time of day for peak
usage
Being able to see the time of day that users access content the most can be a powerful tool. If
there is always a spike in usage around 10am, then that may be the best time to release new
content. It will get the most eyes on it and generate more engagement that is likely to hold it
over longer for others to see it later when they log in.

7. Consumption
Metrics
Here are some things you can track, and what you can do with that information.
What to Track Why track it
Most viewed or
completed content
Knowing what content is getting the most views is important data for your larger data story.
What’s the topic of this content? What type of content is it (article, video, etc)? Were users
required to complete this content or was it organic? It’s difficult to use required content as a
look into engagement. If you’re forcing users to complete something, then you’re not
measuring their actual interest in the content, and that could lead you down the wrong path.
Likes, dislikes,
comments, and shares
This will give you an idea of how well a content is being received. Likes and dislikes will give
you a quick look at perception of the videos. Compare these with the # of views and see what
the % of views vs likes is. A high like % is most likely telling you that content is what users need
for that topic.
Then explore the comments to see if you uncover additional needs. “This was great. I’d love to
see more about … too.” or “I wish this would have talked about ...”
Type of content What types of content are your using viewing the most? Videos, articles, podcasts, links, etc?
Identifying trends like this can tell you the types of content your users prefer most and you can
cater to those preferences as you curate or development new content.

8. Social
Metrics
Here are some things you can track, and what you can do with that information.
What to Track Why track it
How deeply are people
interacting with each
other?
We know how many views a piece of content has, but let’s go one layer deeper. Are people
engaging and interacting with each other?
How many posts are there on a topic? How many people are replying to other comments (actual
conversations)? How many projects have been submitted? How many peer reviews have been
completed?

9. Application
Metrics
Here are some things you can track, and what you can do with that information.
What to Track Why track it
How are individual learners
actually changing their
behavior on the job?
Before course launch, identify behaviors that you want to see changes in.
How is the class as a whole
changing their output as a
result of this course?
Identify a deliverable learners should be able to create on the job? Design the program
around step by step creation of the deliverable and allow for peer to peer learning and
feedback.

10. The stories are in the data.
This is only a small sample of things to measure but you can see how individual data points can give you valuable
insight. What’s even more powerful is when you combine the data. Don’t just look at one piece, compare several
pieces to get the full story.
See appendix for some high impact stories, and visit the Intrepid by VitalSource Learning Library to find more
detailed case studies.

11. Situation
• Consumer goods organization transforming from a holding company to an operating
company
• Goal: cascade the core components of in-person and virtual leadership program delivered to
top executives to the entire company
Solution
• Blended approach embedding Harvard Business Publishing content in the Intrepid platform
with client context and applied learning
• Combination of live webinars and self-directed learning paths with recorded videos, Harvard
content, discussion threads, and individual and group missions
Results
• Very high satisfaction, program impact, and applicability to job ratings (4.5/5, higher than
traditional classroom training historically offered) from learners and strong support for the
blended learning approach from the CEO
• Program participant: “Once everyone is through the program and has engaged in the content
we will operate as a ‘well-oiled machine’.”
• Learners report the platform activities made the content come alive and made the company-
specific context clear
• 800 participants through the program globally across roles in 8 languages
• Program has won 2 CLO Learning in Practice Gold Awards and 2 Brandon Hall Excellence in
Learning Awards
Story: Company Wide Leadership Development

12. Situation
• Highly distributed global locations
• Rigorous, 18-week role-based training
• 100% in-person training model
• Large team of skilled facilitators required to travel constantly to locations
Solution
• Blended learning model enables new hires to learn at their own pace & connect
regularly with facilitator for VILT events. Includes role-specific training.
Results
• ~10,000 new hires onboarded
• 15%+ reduction in onboarding cycle time for call center new hires. Thousands of
hours of productive time added to the business.
• Increased instructor productivity – reduce teaching time & over-time for admin;
increase course prep & development time.
• Increased training availability – now available world-wide regardless of new hire
numbers or geographic location.
• Accelerated adoption across multiple roles.
Story: Health Insurance Call Center & Operations Onboarding

13. Situation
• Enterprise PLM software rollout and change management strategy to drive
efficiency and throughput across the organization.
• Training required for 70+ roles
• Initial approach was an 18 month software rollout/migration and change
management plan that included 50+ ILT events tailored for the unique roles – at
massive time, expense and opportunity cost.
Solution
• Blended learning online + classroom approach of 34 courses delivered to 8,000
learners in 4 countries and in 2 languages. “Tell Me, Show Me, Let Me, Help Me”
format guides self-paced training with same training site used for ongoing
performance support. Applied PLM practice through real-world assignments.
Results
• PLM software is live and learners are using the site for just-in-time performance
support. The PLM training program received positive participant and leadership
reaction. VP of Customer Service: “This is the best training and only approach we
should use ongoing.”
Story: Aerospace Manufacturing Software Rollout

14. Situation
• Firm sought to broaden partner awareness and knowledge related to top
strategic themes impacting their global clients
• Partners traditionally met every other year for development
• Most sophisticated, time-starved, senior-level executives in firm
Solution
• Developed 10 MOOC-like courses on strategic themes as pre-work leading up to
2 in-person global events
• 1600 learners; ~ 5 hours/course
Results
• 600 client conversations
• 1100 partners took 2+ courses
• Vast majority of senior partners rated online learning “very effective preparation”
for live learning events
• Based on results, program being rolled out to broader audience
Story: Management Consulting Senior Partner Development

15. Situation
• 2 year program to accelerate leadership development of entry-level talent capable of
meeting challenges of global supply chain operations. Longitudinal program includes 3 eight-
month rotations.
• Global program needed an integrated learning experience that supports the three rotations.
Solution
• Developed two 3-month MOOCs that enables collaboration and real-world application for
new cohort. Phase 1 focused on Onboarding and Phase 2 focused on supply chain. New
MOOC being developed for 3nd rotation on leadership and a ‘refresh’ of first MOOC for
winter new hires.
Results
• 85% of participants earned 80% of possible points
• 35% of participants completed all content and earned extra credit
• 43,300 unique content views
• Global Program Manager: “We have been so happy with the design and performance of the
Intrepid platform. The discussion forums have been the most powerful aspect, and I love the
simplicity and openness of the format—our high-potential leaders are able to have great
conversations amongst themselves easily.. It’s so simple to use but so powerful for our
learners, offering them a dynamic experience that really increases the impact of our
program. It’s invaluable to our company’s future.”
Story: Continuous Learning Journey for Leadership