Assessment pre-conference session at the 2013 ACUHO-I conference of residence life professionals.

1. Data-Driven Student Success
Programming in Residence Life
Presented by: Ryan O’Connell
Associate Director, Student Success Programs
June 2013
@CampusLabsCo #labgabwww.campuslabs.com/blog

2. WE LIVE AND BREATHE
dedicated
employees
90+
CHANGED THE CAMPUS FOCUS FROM COLLECTING DATA TO CONNECTING DATA
700+
Member campuses and
valued partnerships
Location:
Buffalo, NY & Atlanta, GAmanyrootedinhighered
DATA

3. CAMPUS LABS SOLUTION
One system that collects and reports
data, and brings data together for new
insights.
Organize
planning,
program review
and
accreditation
Assess student
learning and
success
Elevate
teaching
and learning
Retain
students
through
early alert
Create
meaningful
involvement
experiences

4. This session will not be a product demo.
Rather, a
of residential life promising practices,
peppered with core competencies of
assessment.
CELEBRATION

5. Agenda
Intentional Program
Design
Binghamton University
Example
Learning Outcome
Development
Assessing Outcomes
Student Completion
Stats
NAU’s Non-Cognitive
Approach
Continuous Assessment
NASPA/ACUHO-I
Benchmark Data

6. Supported by data,
residential program design can impact your
students’ lives and your institution's retention
rate.
INTENTIONAL

7. Learning

8. Program Program Program Program Program Program
Learning

9. Student learning is not the
result of discrete experiences
but rather the product of
many different kinds of
experiences in and outside
the classroom over an
extended period of time.

10. Student Life
Classroom
Advising
Residential Life

11. College Experience
College Experience
College Experience
College Experience
College Experience
College Experience
College
Experience
College
Experience
College Experience
College Experience
College Experience
College
Experience

12. CAS & Learning Reconsidered 2
6 Domains:
o Knowledge acquisition, construction, integration
and application
o Cognitive complexity
o Intrapersonal development
o Interpersonal competence
o Humanitarianism and civic engagement
o Practical competence
Dimensions within each domain provide
further clarification and direction.
http://www.cas.edu/getpdf.cfm?PDF=D87A29DC-D1D6-D014-83AA8667902C480B

13. Other Sources
General education learning
outcomes
Institutional mission
statement
Accreditation standards
Other professional
organizations

14. Strategies for Developing
Outcomes
Strategy 1: Mission Statements
Strategy 2: Professional Organizations
Strategy 3: Student Work
Strategy 4: An Epistemological and Ethnographic
Process
Strategy 5: Deep and Surface Approach to Learning
Read Peggy Maki’s "Assessing for Learning: Building a
Sustainable Commitment across the Institution”
Chapter 3: Making Claims About Student Learning Within
Contexts for Learning

15. Process for Developing Own
Framework
1. Research
2. Reflection
3. Collaboration
4. Consensus
Read Linda Suskie’s “Assessing Student Learning: A
Common Sense Guide”
Chapter 8: Developing Learning Goals

16. Our Suggested Process
Gather
documents
Assemble
core group
to draft
outcomes
Send draft
to
leadership
for
feedback
and review
Use
feedback
to refine
and revise
Publicize
and utilize
outcome
framework
Continue
to review
and revise

17. Jennifer Hapgood, Business
Manager and Assessment Specialist

18. Public Northeastern University
4-year Institution
Over 14,000 Students
10% Attrition after Freshman Year
Implemented Card Swiping System to Monitor Programming
Participation
Comprehensive Data Systems in Student Affairs to track
Participation in Programs
Binghamton University

19. Pillars of Residential Life
Global
Assisting residents to
develop relationships
with others as well as
preparing them to
become informed
global citizens
Academic
Assisting residents to
successfully navigate
the collegiate
experience and
presenting them the
greatest chance of
being a successful
student

20. Pillars of Residential Life
Personal
Assisting individual
residents to further their
individual growth
Community
Development

24. EXERCISE
Write down no more than 7 learning domains that
you think are addressed through Residential Life
at your campus.
As a small group, share your domains and agree
on 4-6 domains along with a campus name.

25. Program Design

26. • Who does the outcome pertain to?Audience/Who
• What do you expect the audience to know/be able
to do?Behavior/What
• Under what conditions or circumstances will the
learning occur?Condition/How
• How much will be accomplished, how well will the
behavior need to be performed, and to what level?
Degree/How
much
ABCD Learning Outcomes

27. Learning outcome examples
1. After attending a financial aid session, students will
be able to accurately fill out the FAFSA form.
2. As a result of three meetings with an academic
coach, students will be able to compare study
strategies that is appropriate for their learning style.
3. After participating in a leadership retreat, organization
presidents will be able to differentiate between tasks
to complete themselves versus those to delegate to
their executive board before their first spring
semester planning meeting.

28. Co-Curricular Mapping

29. Staff Training
• Module inserted into normal RA/RD training
processes
• Began emphasis on domains
• Review of each Hall’s programs and
demographic make-up prior the term

32. Event Promotion & Tracking

33. Scanfobs
Used at each Residential
Life Event
Very portable and easy
to use
Attendance added to B-
Engaged easily

34. EXERCISE
List 5 programs that your RA staff could host that
map to the outcome domains that you resolved in
the previous exercise. As a group, discuss how
you might engage in this exercise on campus.
How do you train your RA’s to be good
programmers?

35. Assessing Learning
Outcomes

36. Any process employed to
gather data which
requires subjects to
display their knowledge,
behavior, or thought
processes.
Direct Methods Indirect Methods
Any process employed to
gather data which asks
subjects to reflect upon
their knowledge,
behaviors, or thought
processes.
List three sections that should be listed on your
resume:
I can name all of the sections that should be
included when I create my resume.
Strongly agree
Moderately agree
Neither agree nor disagree
Moderately disagree
Strongly disagree

37. Direct or indirect?
As a result of three meetings with an academic
coach, students will be able to compare study
strategies that are appropriate for their learning
style.
Asking students to tell you how they are studying
and comparing that to the study strategies guide
Direct!

38. Direct or indirect?
As a result of three meetings with an academic
coach, students will be able to compare study
strategies that are appropriate for their learning
style.
At the conclusion of the meeting, asking students
to raise their hand if they know what study
strategy they will use (and tracking that)
Indirect!

39. Learning Surveys

40. • Examples from
Delaware

41. Ways to Gather Survey Self
Report Data
As a result of ____, I can _____:
Agreement: Strongly agree, Moderately agree, Neither agree nor disagree, Moderately
disagree, Strongly disagree (another version removes the “moderately” qualifier and/or
uses “neutral”)
To what extent did ____ help you_____:
Extent (5 pt): A great deal, Considerably, Moderately, Slightly, Not at all
How would you rate your ability to _______:
Proficiency: Beginner, Developing, Proficient, Advanced, Expert
What did you learn about _____?
Can you ____?
Yes, No

43. Beginning
1
Developing
2
Accomplished
3
Exemplary
4
Score
Assessment of the
Situation
Staff did not identify the
key areas of concern in the
case study
Staff identified some of the
key areas of concern in the
case study
Staff identified the most
significant areas of
concern in the case study
Staff identified all areas of
concern in the case study
Approach with
Student(s)
Staff recommended an
approach with the
student(s) that is
inappropriate and/or
unprofessional
Staff recommended an
approach that was
acceptable, but could be
improved
Staff recommended an
approach that was
professional and suitable
for the situation
Staff recommended an
approach that was exactly
what is expected
Referrals to Campus
Resources
Staff did not make any
referrals to campus
resources
Staff acknowledged the
resident’s need for a
referral, but didn’t provide
any specific examples
Staff acknowledged
resident need for referral
and gave appropriate
examples
Staff acknowledged
resident need for referral,
provided appropriate ones,
and offered to personally
assist resident to obtain
services
Reporting
Staff did not recommend
reporting the situation to
the appropriate person and
did not write an incident
report
Staff only reported the
situation to the RD or
wrote an incident report –
not both.
Staff reported the situation
to the supervisor and
wrote an incident report
Staff informed the RD,
wrote an incident report,
and mentioned keeping
the RD continuously
informed of any follow up
Community
Education
Staff did not recommend
any community education
as a result of the scenario
Staff mentioned
community education but
did not provide examples.
Staff recommended
specific community
education components
Staff recommended a
variety of community
education components
Follow Up
Staff did not follow up with
the student(s) involved
after the initial contact
Follow up was done, but
not appropriately
Staff followed up with the
student(s) regarding the
initial concern.
Staff followed up with the
student regarding the initial
and also maintained a
positive relationship with
the student over time.
Comments: Overall
Average:

44. Break
Student Completion Stats
NAU’s Non-Cognitive Approach
Continuous Assessment
ACUHO-I Benchmark Data

45. First-year retention rate
http://www.higheredinfo.org/dbrowser
77%
Student Completion

46. 6 year college graduation rate
55%
Student Completion
http://www.higheredinfo.org/dbrowser

47. Student Completion
Percentage of
students who
graduate from high
school in four years,
go directly to
college, are retained
to second year, and
graduate in six
years.
http://www.higheredinfo.org/dbrowser

48. Public Southwestern University
4-year Institution
Over 17,000 Students on the Flagstaff Campus and 1,100
Faculty
27% Attrition after Freshman Year
Goal to increase retention rate by 10%
Student academic services unit within Student Affairs
Northern Arizona University

49. A Non-Cognitive
Approach

50. Non-cognitives
Non-cognitive attributes are those academically
and occupationally relevant skills and traits that
are not specifically intellectual or analytical in
nature. They include a range of personality and
motivational habits and attitudes that facilitate
functioning well in school. Non-cognitive traits,
skills, and characteristics include perseverance,
motivation, self-control, and other aspects of
conscientiousness (Borghans et al., 2008).

51. Literature
Student characteristics including psychosocial factors
influence retention (Brown et al., 2008; Tinto 1975,
1993).
Influence of psychosocial factors on retention extend to
two-year institutions (Napoli and Wortman, 1998)
Robbins, S., Lauver, K., Le, H., Davis, D., Langley, R., &
Carlstrom, A. (2004) examined 109 studies and found
that psychosocial and study skill factor
(noncoginitives) were predictive beyond traditional
factors.

52. The Student Strengths Inventory
Framework
Risk Scale Definition Sample Item
Academic
Success
Academic Engagement The value an individual places on
academics and attentiveness to
school work.
I turn my homework in
on time.
Academic
Success
Academic Self-Efficacy An individual’s confidence in his or
her ability to achieve academically
and succeed in college.
I will excel in my
chosen major.
Retention Educational
Commitment
An individual’s dedication to college
and the value placed upon a college
degree.
I see value in
completing a college
education.
Retention/
Academic
Success
Resiliency An individual’s approach to
challenging situations and stressful
events.
I manage stress well.
Retention Social Comfort An individual’s comfort in social
situations and ability to communicate
with others.
I am comfortable in
groups.
Retention Campus Engagement Involvement in campus activities and
attachment to the college/university.
Being active in extra-
curricular activities in
college is important to
me.

53. Implement a readiness program to
increase
persistence and improve student
success.
NAU’s Goal

55. Institution Needs
Could identify some of their “at-risk” student
groups
• First generation college students
• Ethnic minorities
• Low income students
• Lower SAT/ACT scoring students
BUT needed to know
• More about which students within those groups were
most in need.
• Which services might best help which students.

56. NAU needed to LEARN
MORE about
each INDIVIDUAL
STUDENT.

57. Strategic Intervention Model:
Freshman Outreach Program
Implemented early in the first semester
Include groups known to be at risk
Collaborative and not duplicative
• academic advising and first year programs
• Summer bridge program and first year mentoring
• any key groups interfacing with new freshmen
Systematic
Whole student
Grounded in research

58. Program Model: Student Affairs
Prioritize
Outreach Connect Evaluate
Ethnic minority students
First generation college
students
Low income students
Low academic
preparedness
Predictive Model Scores
Meeting one-one
Review results
Refer resources
Track connections
Student perspectives
Track referrals and use

59. Primary Data Sort
Creating lists of students to be seen by each office
“Waterfall”
– Summer Bridge Students (FG, LI)
– Native American Students
– Other Ethnic Minorities
– Other First Generation/Low Income
– Other Low Scoring (low ACT or HS Core GPA)

60. Secondary Data Sort
Non-cognitive Results
– Low Index Scores
– Low Academic Scale Scores
In House Predictive Modeling Scores
– Likelihood of Leaving

61. Prioritizing Student Groups
One-on-One Meetings in Student Affairs
Systematic Outreach
– Look up schedules and set times
– Postcard home with appointment time
– Postcard to campus with appointment time
– Personal e-mail with appointment time
– Appointment time on their web calendar
– Reminder phone call 1-2 days before appointment
– If miss scheduled meeting, protocol for rescheduling
at least twice more
– Social Media: Facebook

62. The student’s peer group is the
single most potent source of
influence on growth and
development during the
undergraduate years.
Astin, A. What Matters in College? Four Critical Years Revisited. (p. 398). San Francisco:
Jossey-Bass, 1993.

63. Purpose of Meeting
Above all else, connect with student
Acknowledge student’s strengths and identify
campus activities to reinforce them
Match student’s needs to campus resources
Gather some data on SSI experience
– How did you hear about your appointment?
– How seriously did you take the SSI when you
completed it at orientation?

64. Incoming Data Fall 2010 Outcomes
Total
Students
Fall 2010 % Ret'd F11
Ave HS Core
GPA Ave ACT Ave SAT
Census Cr
Hrs
Earned Cr
Hrs Term GPA
Good Acad.
Standing
Freshman
Outreach
SA Target Met 786 74% 3.32 23 1049 15.0 13.3 2.90 88%
SA Target Not
Met 1030 67% 3.22 23 1043 14.9 12.4 2.65 80%
GSSC Target
Met 195 77% 3.01 20 949 14.9 13.3 2.78 88%
GSSC Target
Not Met 117 69% 3.12 20 956 15.1 13.2 2.84 85%
Expected to Do
Well 644 78% 3.43 26 1169 15.2 13.9 3.10 90%
No Survey 460 64% 3.17 22 1047 14.6 11.7 2.65 80%
Assessment of Approach

65. Provide all students with program/service information, and
target needy students for higher-level contact
EARLY
Direct outreach and referrals to needed services must happen
STRATEGICALLY

66. < Student Success
Advisor
< Student Success
Advisor
< RA

67. Erin Grisham
Executive Director, Educational Support Services

68. EXERCISE
Review the non-cognitive report of your partner
and hold a mock intervention conversation.

69. Graphical
Representation of
Results
Percentile Ranks
Result Descriptions
Recommendation
Statements
The Student Report

70. EXERCISE
Review the non-cognitive report of your partner
and hold a mock intervention conversation.
How would you approach RA training?
How could you incorporate non-cognitives into
your programming efforts?

71. This information seems very personal, I’m not
trained in psychology/counseling.
Should I share the risk indices with students?
What if my conversation reveals an urgent
issue?
Common Concerns

72. Ongoing Data
Collection

73. Integrate key questions into your ongoing surveys
to serve as triggers for success outreach
Option to target groups of students based on
known issues (e.g., social comfort, academic
engagement)

74. Micro Surveys
Fall
How does it feel to be away
from home?
How are things going with your
roommate?
Have you made friends?
Spring
How do you feel about your
academic performance last fall?
How much do you like your
major/studies?
Academic resources are available
on campus to help you succeed in
your classes. Which of the
following reflects your use of
these support services?

75. Creates a time-
appropriate data
point that informs
intervention
activity.

76. EXERCISE
List assessment items that you could embed into
operational, satisfaction, and follow-up surveys
that would be good success indicators.

77. As a result of ____, I can _____:
Agreement: Strongly agree, Moderately agree, Neither agree nor
disagree, Moderately disagree, Strongly disagree (another version
removes the “moderately” qualifier and/or uses “neutral”)
To what extent did ____ help you_____:
Extent (5 pt): A great deal, Considerably, Moderately, Slightly, Not at all
How would you rate your ability to _______:
Proficiency: Beginner, Developing, Proficient, Advanced, Expert
What did you learn about _____?
Can you ____?
Yes, No

78. Recap

79. National Data

81. The most important housing considerations when deciding
to attend this institution
www.naspaconsortium.org

82. How frequently did you interact
with…
www.naspaconsortium.org

83. What are the most common ways you learn about
programs and events offered by residence life?
www.naspaconsortium.org

84. Programmatic Engagement
www.naspaconsortium.org

85. Has your involvement in residence life programming and
events provided you with skills and abilities you will use
after college?
www.naspaconsortium.org

86. Which issues have you learned more about/become more
aware since living on campus/in the residence halls?
www.naspaconsortium.org

87. HAVE A GREAT
CONFERENCE!
campuslabs.com