A seminar for Edge Hill University, January 2013

1. DEVELOPMENTAL EVALUATIONS
FOR INSTITUTIONAL IMPACT
Dr. Rhona Sharpe
rsharpe@brookes.ac.uk
For Edge Hill University
January 2013
Oxford Centre for Staff and Learning Development
brookes.ac.uk/services/ocsld

2. Research at the Oxford Centre for
Staff and Learning Development
Jude Carroll‟s research on
- The impact of tariffs on deterring plagiarism
- Helping students understand what is
expected
Leading to… holistic, institutional approaches
to deterring and detecting plagiarism. The
Brookes Academic Conduct Officer model has
been replicated around the UK and
internationally.

3. Chris Rust,
Margaret Price and
Berry O‟Donovan on
assessment and
feedback …

4. OCSLD Research and Knowledge
Transfer Strategy (2009-2013)
Three strands of activity:
1. National and internationally recognised
pedagogic research
2. Institutional research and knowledge transfer
practices
3. Scholarship of teaching as a core element of
our CPD framework
https://wiki.brookes.ac.uk/display/pedres/Research+strategy

5. Strategy for Enhancing the
Student Experience (2010-2015)
“Making use of our internationally recognised in-house
expertise in educational development, we commit to
routinely carrying out impact assessment, review and
revision of all significant academic development
initiatives and of measures taken forward in the SESE
and the consequent strategy maps.
Impact assessments will be sustained and iterative.
They will assess the process as well as the end
product of developments. [SESE 4.4.3]

6. Participatory research
“Nothing about me, without me”
Involving learners as
consultants and partners
Early and continued
participation
Meaningful and useful
outcomes

8. Approaches to institutional
research which are:
• Iterative and ongoing
• Inform the community tasks and
operation
• Be a tool for positive change
• Be a shared, collective inquiry
• Involve multiple data collection methods
and times

9. Institutional commissions
1. Evaluation of the embedding of our
Assessment Compact
2. Evaluation of our new models of
faculty based student support
3. Evaluation of the embedding of our
Graduate Attributes
4. Attainment of students from black and
other minority ethnic groups

10. Evaluation of faculty-based
student support (2010-12)
 Where do students prefer to go
for support and on what issues?
 What is the impact of the new
model of support on the student
experience?
 How do the roles of the Student
Support Co-ordinators and
Academic Advisers interact?
https://wiki.brookes.ac.uk/display/pedres/Student+Support

11. Method
Surveys: Interviews:
478 student survey Interviews with 20
respondents (May students
2010 & Nov 2011)
Interviews with SSCs
187 „comments cards‟
on SSCs Interviews with 18
Academic Advisers

12. SATISFACTION: HOW USEFUL IS THE
ADVICE AND INFORMATION FROM
SSCS?
Not
n Very Quite very Not at all
2010 24 9 4
survey 37/228 (64.9%) (24.3%) (10.8%) 0 (0%)
2011 47 29 3
survey 80/214 (58.8%) (36.3%) (3.7%) 1 (1.2%)
2011
Comments 161 25 1
cards 187 (86.1%) (13.4%) (0.5%) 0

13. HAVE YOU EVER NEEDED ADVICE
AND NOT FOUND IT?
Hardly Some-
Never ever times Often Total
2010 45.3% 23.6% 26.4% 4.7%. 212
2011 49.8% 23.9% 20.2 % 6.1% 213

14. Where do you prefer to go for Academic SSC Other
advice and information . . . Advisor
1. ..when you have questions 92 42 34
about your programme and (54.8%) (25.0%) (20.2%)
PiP e.g. adding/deleting
for 9 of your
modules fromthe 11 issues asked about, the
programme of study or Stage
Academic Adviser was the person selected
II planning?
as preferred source of support.
2.. on changing the subjects 75 21 32
you are studying? (58.6%) (16.4%) (25.0%)
3. .. on obtaining and 54 27 41
presenting a Medical (44.3%) (22.1%) (33.6%)
Certificate?
4. .. on careers? 50 27 38
(44.2%) (22.1%) (33.6%)

15. Stories from individual students
Lizbeth is a second year International student who began
her first semester one January when other students were
entering their second semester ... (read more from Lizbeth)
Marion. In her third year of study, feedback from lecturers
that referred to her work‟s „sloppiness‟, „carelessness‟ and
„poor spelling‟ provoked Marion‟s decision to formally
register her dyslexia ... (read more from Marion)
Sonya sought advice from a lecturer on arrival. Having
always received good advice and support over the
intervening three years, Sonya has continued to make her
senior lecturer her first port of call .. (read more from Sonya)

16. Early findings
1. Overall high rates of satisfaction with
support when it is accessed
2. 25% student survey respondents
report not having found support when
they need it
3. A preference for academic advisers,
despite promotion of SSCs role

17. Research Qs version 2
Why do students not use our services in
the ways that we had intended?
Why do students make the choices they
do about where to go for support?
What can be learnt about students‟ help
seeking behaviour to improve our
services?

18. Why do students not seek help?
Help-seeking is perceived as dependent
(low autonomy)
Help seeking is perceived as
incompetent (low self-efficacy)
Mistaken belief that help-seeking will
not be effective
Ryan et al. 2005

19. Have you ever needed advice and
information and not found it?
73 responses in open
comments box, open
coding, resulting in 10
explanatory categories
Top 2 categories
 the quality of the advice
given is poor
 it is not specific enough
to my issues

20. Why do students not seek help?
Help-seeking is perceived as dependent
(low autonomy)
Help seeking is perceived as
incompetent (low self-efficacy)
Mistaken belief that help-seeking will
not be effective
Ryan et al. 2005

21. Helping students to see support as
effective (turning round table 9)
Demonstrate knowledge of staff
Make support relevant to students‟ needs
Promote different ways of accessing support
Make timely responses
Direct students to appropriate support first time
But, someone needs to make the first move

22. WHO MAKES THE FIRST MOVE?
A total of 151 (73.3%) of students reported
in the online survey that they had been to
see their Academic Adviser.
Another way of looking at that is that 26.7%
of students by Week 8 in Semester One
have not seen their Academic Adviser
this academic year.

23. Recommendations for support-givers
1. promote service on basis of being effective
i.e. knowledgeable, available, relevant,
able to resolve issues and/or offer high
quality referrals.
2. establish early contact and regular
interactions with students
3. be proactive in making contact

24. So, what’s changed?
Associate Dean asked to convene a working
group to implement the recommendations
From Sept 2013 there will be mandatory training
for Academic Advisors
We have university statement on academic
guidance and the role of the Academic Advisor
AAs will be required to meet with, and log that
they have met with, their students at at least 3
points during this year.

25. What’s changed?
Associate Dean asked to convene a working
group to implement the recommendations
From Sept 2013 there will be mandatory training
for Academic Advisors
AAs will be required to meet with, and log that
they have met with, their students at at least 3
points during this year.
This is a long way from saying the new system is
working well because of high satisfaction rates!

26. How did we do it?
Developmental evaluation
• Engaged Student Support Coordinators were
important actors, piloting data collection tools
and helping interpretation of data.
• Interim reports of findings used to raise
questions for further action or inquiry
• Highest level formally (University T&L
Committee) AND many other levels informally
• Produced useable, engaging outputs
• Reports shared openly on web from the start

27. What we’ve learnt
1. Value the sharing of multiple
perspectives
Observations and interpretations that do not
agree do not necessarily indicate a failing of
evaluation but perhaps the complexity of the
program and its contexts. It is problematic to
assume there is a simpler world behind the
world that people see.
Stake, R.E. (2004) p 286

28. What we’ve learnt
1. Value the sharing of multiple perspectives
2. Revisit the research questions along the
way
3. Encourage the use of metaphors, stories
and images
4. Facilitate participation in making sense of
the finding, constructing recommendations
and producing outputs
5. Build in multiple opportunities for data
collection
6. Disseminate from day 1

29. What next?
We are currently:
Deep in NVivo analysing Programme
Specifications to show visibility (or not) or our
graduate attributes in documentation
Preparing a project plan to understand how
innovative academics become academic
managers, and how we can support them.
Training student interviews for a project to
investigate the attainment gap between BME and
white students.

30. References
Oxford Brookes University (2010) Strategy for Enhancing the Student
Experience
http://www.brookes.ac.uk/about/strategy/development/docs/sese2010-
15.pdf
Ryan, A. M., Shim, S., Patrick, H. (2005) Differential profiles of students
identified by their teacher as having avoidant, appropriate, or
dependent help-seeking tendencies in the classroom. Journal of
Educational Psychology, 97(2), 275-85
Sharpe, R. & Benfield, G. (2012) Internet based methods, in J. Arthur, R.
Coe, M. Waring and L. Hedges (eds) Research methods and
methodologies in education, pp 193 – 201. Sage
Sharpe, R. & Clarke, P. (2009) A community based programme of support,
in JISC Emerge: a user-centred social learning media hub: Supporting
the Users and Innovation R&D community network, pp 20-25 at
http://reports.jiscemerge.org.uk/
Stake, R.E. (2004). Standards-based and responsive evaluation.
Thousand Oaks, CA: Sage.