1. Welcome!
Please …
1. Take a seat with your grant team members (or
anywhere if you are solo).
2. Grab some coffee.
3. Complete the GLF Evaluation Capacity Assessment.
4. Get connected to the wireless network: NCSU Guest.
Instructions are in your folder.
5. Get access to the GLF Essential Skills Evaluation
“Moodle” website (where we will organize project
information). Instructions are in your folder.

2. Accessing the Moodle Website
Before we get started, we want to make sure everyone has
successfully accessed the private Golden LEAF Essential Skills
Evaluation Moodle website.
• Instructions are in your folder.
• Raise your hand if you are having
difficulty and an evaluation team
member will assist you.

3. Golden LEAF Essential Skills Initiative
Summer Evaluation Institute
June 27, 2013
The Friday Institute – Raleigh, NC

4. Golden LEAF Essential Skills Initiative
Evaluation Team
The Friday Institute at NCSU
• Jeni Corn
• Latricia Townsend
• Rob Maser
• Malinda Faber
• Laurie Brummit
NCSU College of Education
• Diane Chapman
• Julia Storberg-Walker
NC Community College System
• Kristen Corbell

5. Agenda
• Welcome and Introduction
• The Evaluation Process & Logic Models
• Team Logic Modeling
• LUNCH
• Asking Good Questions
• Balanced Scorecard
• BREAK
• Culture of Curiosity & Data Definitions
• Closing and Q&A

6. The Golden LEAF Essential Skills Initiative
“Supporting collaborative programs aimed at increasing
the talent pool of a highly skilled technical workforce
aligned with identified employment opportunities in
tobacco-dependent, economically distressed and/or
rural communities of North Carolina.”
14 Grants
22 Community Colleges
14 School Districts

7. The Initiative Evaluation
• Grant artifacts
• CC registrar data
• Student surveys
• Instructor surveys
• Employer surveys
• Focus groups
• Interviews
• NC Education Research Data
Center administrative data
• NC Community College System
administrative data
• Employment Securities
Commission workforce data
Evaluation Questions Data Sources
TBD

8. Who Is in the Room?
One person from each grant team
stand up and please share:
• What Community Colleges your
team is representing
• One sentence description of your
grant’s main focus

9. The Evaluation Process

10. Plan, Do, Study, Act (PDSA)
Loyola University Stritch
School of Medicine, 2011,
http://www.stritch.luc.edu/lu
men/MedEd/softchalkhdht/C
MEFacDevWebPage/CMEFacD
evWebPage10.html
CYCLE OF
CONTINUOUS
IMPROVEMENT

11. Five Evaluation Steps
1. Logic Model: Identify the critical elements of the
workforce development program.
2. Evaluation Questions: Ask important questions about
your workforce development program.
3. Data Sources: Identify what data are available to help
answer your questions and determine the additional data
needed.
4. Data Analysis: Collect, analyze, and interpret data to
answer your questions.
5. Decisions: What changes to your workforce development
program should you make based on your result.
Repeat steps 1-5
11

12. Communicate & Collaborate
• Include key people in the
evaluation planning process:
• Especially the primary users of
the results.
• This is a great opportunity to
build a shared understanding
and buy-in for your project.
• Communicate regularly that
this is on-going process of
improvement – “This is not a
gotcha evaluation.”

13. Keep it Simple and Focused
• An evaluation doesn’t
have to be big! Match
the number of
evaluation questions in
your plan to your
resources.
• Focus on efficient,
effective data
collection strategies.
A basic chart is a great way to
organize your own thinking and
to share easily the plan with
others.
Evaluation
Questions
Data
Sources

14. STEP 1: Logic Model
Identify the critical elements of the workforce
development program

15. Formative Evaluation Steps
1.Logic Model: Identify the critical
elements of the workforce
development program.
2. Evaluation Questions: Ask important questions about your
workforce development program.
3. Data Sources: Identify what data are available to help answer your
questions and determine the additional data needed.
4. Data Analysis: Collect, analyze, and interpret data to answer your
questions.
5. Decisions: What changes to your workforce development program
should you make based on your result.
Repeat steps 1-5
15

16. ACTIVITY
Write answers to questions 1-3, answering them in that order, on post-
its. You will likely have multiple answers - write one idea per post-it.
1. Ultimately, what good is your program going to do (for students,
employers, etc.)?
2. If you are successful implementing grant activities, what will your
program accomplish this year?
3. What are the major activities of your program? (How are you
going to spend your time, money, and other resources?)
Arrange your post-its on a piece(s) of paper from left to right, from
major activities (3), to immediate results (2), to the ultimate good the
program is going to do (1).
3  2  1 .

17. What is a logic model?
A logic model is a
graphic representation
of the relationships
among the key
elements of a project:
• inputs,
• strategies,
• objectives,
• long-term goals.
A logic model …
• Helps to articulate the key
elements of the project.
• Enables evaluation efficiency
and effectiveness.
• Promotes stakeholder buy-in
by helping clarify how the
project works.
• Provides a great opportunity
to involve stakeholders in
planning.
They are living documents – they can change over time!

18. What is a logic model?

19. What is a logic model?
www.ScienceCartoonsPlus.com

20. What is a logic model?
Inputs Activities Outputs
Short-term
Outcomes
Medium-
term
Outcomes
Long-term
Outcomes
What is
invested in the
process? Time,
money, human
resources,
partners,
equipment,
etc.
What are the
major
activities the
program will
entail? What
will be done
with the time,
money, etc.?
How many
participants
are involved?
How many
teachers
and/or
students are
reached?
What is
produced?
Within life of
grant:
What changes
in knowledge,
skills,
attitudes,
motivations,
or awareness
resulted?
Within 1-3
Years:
What changes
in behaviors,
practices,
policies, or
procedures
resulted?
After 3 or
more Years:
What are the
ultimate goals
and did
changes
results?
Student
achievement,
teacher
retention,
school culture,
etc.

21. What is a logic model?
Terms aren’t consistently
used … meaning is what
matters.

22. What is a logic model?
Whenever possible, it is helpful to try to use
the “SMART Goals” strategy for creating your target
outcomes:
Specific
Measurable
Attainable
Results-oriented
Time-bound
E.g.: Increase number of students taking advanced science
courses by 25% by fall 2015.

23. Strategies Short Term Goals Long Term Goals
Regional career work-based
experiences supported by
business community
Partnerships to provide
enrichment activities for
students and differentiated PD
for teachers
Curriculum alignment for vertical
and cross curriculum through
PLCs
Increased access to content
specific technology for science
and math
Increase enrollment in Algebra 1
for middle school students and
upper level math and science
Increase percentage proficiency
in Math and Science
Increase use of inquiry-based
learning in lessons
Raise STEM interest and
awareness
Increase student engagement
and achievement in the STEM
courses
23
Development of Leadership for
supporting STEM instruction
PD in Inquiry-Based learning
Increase student interest and
teacher awareness in STEM
careers
Promote STEM educator
effectiveness through
professional development
Early diagnosis and intervention
for college readiness assessment

24. Team Logic Modeling 11:00am – 12:15pm
1. In your grant teams, on a computer, draft a logic
model for your grant. (Use
PPT, Word, Inspiration, …)
2. After drafting your logic model, reflect on your
PMPs - are there changes you want to make?
3. (You will get another 20 minutes to reflect on your PMPs in the afternoon.)

25. Lunch!
12:00 – 12:30pm
Pack up your tables a little, grab
lunch, and find a new table to sit at
with a conversation topic you’re
interested in (look for table tents).
THIS AFTERNOON: Asking Good Questions; Balanced
Scorecard; Community College System Data Initiative.

26. Asking Good Evaluation
Questions

27. Formative Evaluation Steps
1. Logic Model: Identify the critical elements of the
workforce development program.
2. Evaluation Questions: Ask important
questions about your workforce
development program.
3. Data Sources: Identify what data are available to help
answer your questions and determine the additional data
needed.
4. Data Analysis: Collect, analyze, and interpret data to
answer your questions.
5. Decisions: What changes to your workforce development
program should you make based on your result.
Repeat steps 1-5
27

28. ACTIVITY
What do you think of when you think of
the term “evaluation question”?
Look at your logic model. Jot down 1 or
2 examples of what you think could be
evaluation questions for your program.

29. Developing Evaluation Questions
• Evaluation questions provide the direction and
foundation for the entire evaluation.
• The process for identifying the questions to be answered
by the evaluation is critical.
EVALUATION QUESTIONS
Data Collection
Data Analysis
Results

30. Developing Evaluation Questions
Why do we need to ask good questions?
• To determine what is really important, and to whom
• Considering “to whom” will help you when you need to
decide what kind of data you want to collect to answer the
particular question.
• To focus data collection efforts
• What do we need to find out?
• How will we collect that information?

31. Developing Evaluation Questions
Every evaluation question can’t be answered - finding out the
answers costs time, money and people. Pick the most important
questions that provide the most valuable information to users.

32. Keep it Simple and Focused
• An evaluation doesn’t
have to be big! Match
the number of
evaluation questions in
your plan to your
resources.
• This will help you focus
on efficient, effective
data collection
strategies.
A basic chart is a great way to
organize your own thinking and
to share easily the plan with
others.
Evaluation
Questions
Data
Sources

33. Developing Evaluation Questions
The main types of evaluation questions are:
1. Questions about STRATEGIES: these questions ask
about how well the strategies were implemented.
2. Questions about OBJECTIVES: these questions ask
about impacts.
Logic models are great guides
for developing evaluation questions.

34. Developing Evaluation Questions
Quick tips for writing good questions:
1. Try to avoid simple “yes or no” questions
2. Consider QUANTITY questions, e.g:
– “How many”
– “How much”
– “How often”
3. Consider QUALITY questions, e.g.:
– “How well”
– “How effectively”
– “In what ways”
4. Be able to be tuned-in to unexpected results.

35. Developing Evaluation Questions
IMPLEMENTATION/STRATEGY
questions:
- How many hours of sleep am I getting
each week? (quantity)
- How soundly am I sleeping? (quality)
IMPACT/OBJECTIVE questions:
- How much weight have I lost?
(quantity)
- How has my stress level changed?
(quality)

36. ACTIVITY
Brainstorm a list of evaluation questions, and write them down.
1. Brainstorm implementation questions about the program’
strategies
For example:
THE STRATEGY IS: Teachers attend 4 quarterly sessions on science kits.
A QUANTITY QUESTION: How many faculty/staff attended the sessions?
A QUALITY QUESTION: How did the participants rate the quality of the sessions?
2. Brainstorm impact questions to evaluate how well the
outcomes are being met.
Crosswalk your list of evaluation questions with your PMP. Do
the activities and outcome measures in your PMP capture all of
your evaluation questions and the data you would use to answer
them? Fill in any gaps.

37. Balanced Scorecard

38. What is the Balanced Scorecard?
• A 4-quadrant framework that provides a comprehensive
picture of performance
• Originally designed for business and industry, now regularly
adapted to different sectors
– Original four quadrants are financial, customer, internal
processes, learning and growth
• Adapted to the GLF Essential Skills project, the quadrants are
partnerships, workforce education, internal
processes, learning and growth
• GLF evaluation objective: Aggregate reporting of four
quadrants

39. Development of the Balanced Scorecard
Combined
Evaluation
Strategy
Workforce
Education
Internal ProcessPartnerships
Learning &
Growth

40. Combined
Evaluation
Strategy
Workforce Education
Outcomes associated with changes
in educational access and
achievement including dual degree
and 3rd party certifications
Internal Process
Outcomes associated with changes
in enrollment and capacity
Partnerships
Outcomes associated with partners
(community, industry, education)
including partnership strength,
activities and impact
Learning & Growth
Outcomes associated with changes
in grantee
knowledge, skills, attitudes, and
capabilities
Essential Skills Evaluation Balanced Scorecard

41. Gallery Walk – What are the data sources?
Combined
Evaluation
Strategy
Workforce Education
Increase access to programs in targeted areas
Increase program completion in targeted areas
attributed to grant proposal and collaborative
partnerships
Increased dual degree opportunities attributed
to grant proposal and collaborative
partnerships
Increase in 3rd party credentials attributed to
grant proposal and collaborative partnerships
Internal Process
Increase enrollments in targeted courses and
programs
Increase capacity attributed grant proposal
Increase capacity attributed collaborative
relationships
Partnerships
Increase students served
Increase student employment in targeted
industries
Strengthen and increase targeted industry
partnerships
Increase activity with partners
Learning & Growth
Increase learning attributed to collaborative
partnerships
(community, students, industry, colleges)

42. Break!
For 10 minutes.

43. Hope Opportunity Jobs
Dr. Kristen Corbell, Coordinator of Research Projects
North Carolina Community College System Office
Basic Skills Focus Area Team Chair, Data Initiative Coordinating Team member, Data
Access and Advisory Team member, External Sources Focus Area Team Member
Data Initiative
Overview
4/15/2014

44. Hope Opportunity Jobs
Data Initiative Overview
The Data Initiative includes the
review, assessment and revision of our
current data collection and reporting
processes
that results in a robust data system which
provides accurate and accessible information
fostering a culture of data-driven decision
making which addresses research questions
and informs policies. 44

45. Hope Opportunity Jobs
Data Initiative Motivation
• Concerns over data quality at the college and state level
• Increasing demands for data from federal, state, and grant-providing
agencies/organizations
• Expectations of collaboration between state agencies to share data
• The disconnect between data access and decision makers at the
college and state level
• Increasing demands on colleges from accreditation bodies to practice
and exhibit informed decision-making
• Current reporting tools do not adequately supporting the increasing
research and reporting demands
• The need to ensure policies are based on sound data
45

46. Hope Opportunity Jobs
Data Initiative Objectives
• Expand the individual and collective inquisitiveness of stakeholders
within the NC Community College System to ask research questions
that inform decisions and policies.
• Develop data definitions based on research needs and reporting
requirements.
• Ensure data quality and consistency through appropriate and
uniform entering and collecting of information across the System so
data extracted are valid and reliable.
• Incorporate a data review process that ensures accountability
through the validation of submitted data.
46

47. Hope Opportunity Jobs
Data Initiative Objectives
• Develop focused topic-based data marts that will serve a wide
variety of analytical research needs including, but not limited
to, strategic initiatives, grants, and business and student centric
needs.
• Expand information accessibility by making all data users aware of
the availability of predefined reports, web-based
dashboards, statistical software, and training.
• Enhance advanced analytical capabilities to empower researchers
focusing on specific topics and initiatives.
• Utilize research findings and reports to educate policy makers on
the extent to which various factors impact outcomes, including
student enrollment, student learning, student completion, faculty
and staff development, budget allocation, etc.
47

48. Hope Opportunity Jobs
Continuous
Improvement
Cycle

49. Hope Opportunity Jobs
Culture of Curiosity / Strategic Initiatives
49
Culture of Curiosity
Represents individual and collective inquisitiveness to ask research
questions that inform decisions and policies
Data Needs
•Strategic Initiative Variables
•Reporting Mandates
•Research Needs
Data Definitions
Developed based on research needs and
reporting requirements

50. Hope Opportunity Jobs
How the Essential Skills Grant fits in with the
Culture of Curiosity
• Within this grant you are asking questions about
how the innovation at your college(s) is impacting
student success and employment outcomes.
These questions are building your culture of
curiosity.
• From your questions, you are identifying the data
that will be collected to answer the questions.
• For analysis throughout and at the end of the
grant, all data must be collected with clear
definitions from the beginning and carried
throughout.
10/17/2012

51. Hope Opportunity Jobs
Data Quality, Consistency, and Accountability
51
Data Quality and Consistency
Ensures information is collected and
entered appropriately and uniformly
across the system so data extracted for
reporting purposes is valid and reliable
Data Definitions
Direct what, where, when,
and how information is
entered into the data
collection systems
Data Entry
Proper understanding of data
definitions will sustain the quality of the
data collection and reporting processes
Data Review
Process which ensures accountability
through the validation of data
submitted for reporting purposes

52. Hope Opportunity Jobs
How the Essential Skills Grant fits in with Data
Quality and Consistency
• Using the data definitions, your college should
ensure that the data are collected and entered
uniformly.
• We would suggest reviewing the data entered
prior to submission to ensure the data make
sense. For instance, make sure dates entered
are logical, program codes are correct, and
student grades are recorded on time for
submission with the semester files.
10/17/2012

53. Hope Opportunity Jobs
Data Collection and Warehousing
53
Data Extraction, Transformation, and Loading
Procedural routines of extracting data files
from local systems to a centralized warehouse
Data Marts
The creation of specific data sets dedicated to
strategic initiatives and research focus areas
Data Access
The ability to query variables within data
marts for basic informational purposes and
detailed research needs
Data Collection and Warehousing
Centralized and technical processes that impact the way in which data is
transferred and transformed between data systems and stored for reporting
purposes

54. Hope Opportunity Jobs
Culture of Evidence / Policy Implications
54
Culture of Evidence / Policy Implications
•Determination of optimal decisions and policies based on the analysis and
reporting of collected data
•Likely to result in development of additional research and evaluation needs
General Reporting
Automated informational reports updated
routinely
Advanced Analytics
Research and analysis focused on specific topics and
initiatives to inform current and future practices.
Research Findings & Reporting
Educate policy makers on the extent of impact of various
factors on outcomes, including student learning and progress

55. Wrap-Up

56. Calendar of Evaluation Activities – Year One
ACTIVITY DATE
Summer Institute 2013 June 2013
Grant Coordinator Phone Interview Fall 2013
Site Visit Fall-Spring 2013
Surveys Winter 2014

57. Please Take 5 Minutes to Complete Survey
Go to GLF ES Moodle site.
Go to the Summer Institute 2013 Section
Click on the Essential Skills Professional Development
Summer 2013 Survey link.
Thank you.

58. Thank You
Questions? Comments? Concerns?
Have a good trip home!
Michael Maslin Item
#: 8476186