Jeremy C. Lundberg, CEO of EthosCE Learning Management System and DLC Solutions moderates this expert panel with Derek Dietze and Dr. Christopher Viereck of ImproveCME.com. Derek and Christopher shared their insights and expertise on what they consider the “5 Keys to Successful CME Outcomes Assessment.” The feedback from the program has been excellent and we hope that you will share this program with your colleagues.

1. Five Keys for Success in
Outcomes Assessment
Presented by:
Jeremy C. Lundberg, MSSW
Derek T. Dietze, MA, FACEHP, CCMEP
Christopher Viereck, PhD, CCMEP

2. EthosCE LMS
Enterprise-level learning
management system.
2012 Brandon Hall Award-Winner
Designed for CME and ACCME
PARS compliance.
Integrated
assessment, evaluation, and
certificate.
Personal “My CME” transcript.
RSS Management
Automates and streamlines “back
office” administration.
Powerful 24/7 reporting engine.

3. Common Model in CME
Meeting Logistics
Assessments
Online Courses
Evaluations

4. A Strategic Model for CME
Learning Management System
HR Management System
Association Management System
3rd Party Applications
External Education Partners

5. Question 1. What are your challenges with
outcomes assessment?
(Check all that apply)
1. I don’t know where to start
2. I don’t understand the differences in outcomes levels
3. I can’t get the level of outcomes data I want
4. I have trouble linking gaps to learning objectives, to outcomes
5. I don’t know how to meet commercial supporter expectations

6. 1. Plan Ahead…Way Ahead
•
Create a detailed descriptive outcomes assessment plan at the
earliest stages of grant request or activity development
•
Develop assessment questions in concert with content, not as
an afterthought
•
Will you have enough resources available to create assessment
questions, implement the plan, and analyze/report the results?
•
Plan with the end in mind
•
Examples

7. 2. Measure to Your
Educational Design
• What do you expect to change in your learners?
•
In what domains do these expected changes fall?
• Knowledge/awareness, confidence, competence,
performance, patient outcomes
• Accurately represent the Moore level(s)
• For higher outcomes levels, distinguish between subjective
and objective assessment

8. 2. Measure to Your
Educational Design (cont.)
• If your learning objectives are focused on knowledge
and competence change, why assess performance
change or patient outcomes?
• When you measure to your design you have:
• Enhanced credibility
• More effective use of resources

9. Question 2. Please rate your confidence in
optimally aligning outcomes questions with
learning objectives and gaps.
1. Not confident at all
2. Not very confident
3. Somewhat confident
4. Very confident
5. Extremely confident

10. 3. Align, Align, Align
• Bring all of the following into alignment:
• Gaps
• Learning objectives
• Desired results
• Instructional design
• Content
• Format
• Outcomes questions

11. 3. Align, Align, Align (cont.)
•
Demonstrate alignment in your planning documents
•
Examples:
• Each learning objective is linked to a specific gap
• Each outcomes assessment question is linked to a specific
learning objective
•
When your alignment is good: much greater opportunities for
positive results

12. Question 3. When you promise “Level 5
Outcomes Results”, what does this really mean
to you?
1. Chart pull data on participant performance
2. Performance changes self-reported by participants
3. Planned performance changes
4. Patient outcomes reported by a participant
5. I have absolutely no idea!

13. 4. Keep the Plan
Simple and Specific
•
Best option: Describe what you are going to measure, how you are
going measure it, and when you plan to deliver on your promises.
•
Be sure to include:
•
Outcomes objectives and overall methods
•
Activity-specific outcomes details
•
Your approach to statistical analysis
•
What is in the results reports, and when they will be delivered
•
How you will share results?

14. 4. Keep the Plan
Simple and Specific (cont.)
•
•
•
Your outcomes plan is the implementation guide
Complex, overly ambiguous or ambitiousplans work against you:
• Ambiguous plans create misunderstandings
• Falling short of original planshurts your long-term credibility
Reality check: Do you have the internal and external resources?

15. 5. Use Multiple Types of
Assessment Questions
•
Don’t put all your eggs in one basket by using only one question type.
•
Consider using more than one question type to address your learning
objectives.
•
Example: Use a combination of pre/post multiple-choice, case-based
or agreement scale questions.
•
Mandatory: Align the assessment questions with the gaps and
learning objectives.

16. Five Keys for Success in
Outcomes Assessment
• Plan Ahead…Way Ahead
• Measure to Your Educational Design
• Align, Align, Align
• Keep the Plan Simple and Specific
• Use Multiple Types of Assessment Questions

17. Leverage the Value of Your
Outcomes Results
• Summarize your results succinctly
• Validate your needs assessment
• Determine the degree to which your learning objectives
were met
• Relative effectiveness of different question types
• Identify effective versus less effective content
• Inform future content development and improvements

18. Leverage the Value of Your
Outcomes Results (cont.)
• Obtain more financial support: insert results in an
upcoming grant request
• Assist commercial supporters in selling the value of CME
within their organization
• Justify department budgets
• Allocate resources more efficiently
•
Share results in clinical and CME community

19. Tools for Success
Image of EthosCE report screen
Collaborative partnerships

20. Question & Answer

21. Thank You
Jeremy C. Lundberg, MSSW
(267) 234-7401
jlundberg@dlc-solutions.com
www.EthosCE.com
derek.dietze@improvecme.com
christopher.viereck@improvecme.com