1. Session 1013
Session1013
QAF, UUDLEs and GDLES
QAF, UUDLEs and GDLES
What Ontario's new Quality Karen Nicholson, McMaster
Carol Stephenson, Laurier
Assurance framework means Dana Thomas, Ryerson/Scholar's Portal
for Academic Librarians 1

2. Session Agenda
• Introduction to QAF, UUDLES, GDLES
• Are you ready to play?
• Quality Assurance in Higher Education
• Accountability, quality and
outcome-based education
• Quality assurance and OBE in
Ontario
• Degree Level Expectations
• Impact on OCUL Libraries
• Overview of QAT: from inputs to
outcomes
• Your Input
2 http://www.flickr.com/photos/thru_the_night/

3. Setting the Stage...
http://www.cou.on.ca/Related-Sites/The-Ontario-Universities-Council-on-Quality-
Assura/Policies/Quality-Assurance-Framework---Guide.aspx
3

4. Quality Assurance Framework
• Institutional Quality Assurance Process
• New program approvals & existing programs
• Audit
• Undergraduate Degree Level Expectations (UDLEs)
• Graduate Level Degree Expectations (GDLEs)
4

5. I. Quality in Higher Education
5

6. • Knowledge economy • Greater accountability
• Accessibility • Measuring of outputs &
outcomes through QA
• Increased funding processes
6

7. Quality Assurance
• "Policies, attitudes, actions and procedures necessary to
ensure that quality is being maintained and
enhanced" (Woodhouse, 1999)
• Accountability = External
• Enhancement = Internal
7

8. History of Quality Assurance in Ontario Universities
8

9. History of Quality Assurance in Ontario Universities
1968 Ontario begins approvals of new grad programs
8

10. History of Quality Assurance in Ontario Universities
1968 Ontario begins approvals of new grad programs
1982 OGCS begins review of existing grad programs
8

11. History of Quality Assurance in Ontario Universities
1968 Ontario begins approvals of new grad programs
1982 OGCS begins review of existing grad programs
1996 UPRAC begins studies on new and existing undergrad programs
8

12. History of Quality Assurance in Ontario Universities
1968 Ontario begins approvals of new grad programs
1982 OGCS begins review of existing grad programs
1996 UPRAC begins studies on new and existing undergrad programs
2005 OCGS prepares GDLEs, COU endorses UDLEs
8

13. History of Quality Assurance in Ontario Universities
1968 Ontario begins approvals of new grad programs
1982 OGCS begins review of existing grad programs
1996 UPRAC begins studies on new and existing undergrad programs
2005 OCGS prepares GDLEs, COU endorses UDLEs
2007 Review of OCGS processes
8

14. History of Quality Assurance in Ontario Universities
1968 Ontario begins approvals of new grad programs
1982 OGCS begins review of existing grad programs
1996 UPRAC begins studies on new and existing undergrad programs
2005 OCGS prepares GDLEs, COU endorses UDLEs
2007 Review of OCGS processes
2010 Development of Quality Council; Quality Assurance Framework
Implemented
8

15. Defining Quality
• Exceptional
• Perfection
• Fitness for purpose
• Value for $
• Transformation
(Harvey & Knight, 1996)
http://www.flickr.com/photos/
dieselbug2007/414348333
9

16. The identification of program-level
outcomes is a key feature of
outcome-based education that
"places student learning at the
center of assuring and advancing
quality of higher education."
(Abate, Stamakis, & Haggett, 2003)
10

17. "One obvious way in which
universities have sought to
articulate their role and purpose is
through a description of the
qualities of their graduates."
(Barrie, 2006)
11

18. II. Quality and OBE in ON's QAF
http://www.flickr.com/photos/bean/
322616749
12

19. "The Degree Level Expectations, combined with the expert
judgment of external disciplinary scholars, provide the
benchmarks for assessing a program's standards and
quality."
13

20. Quality Assurance Framework
• Institutional Quality Assurance Process (IQAP)
• New program approvals & existing program reviews
• Audit
• Undergraduate Degree Level Expectations (UDLEs)
• Graduate Degree Level Expectations (GDLEs)
14

21. Outcome-based Education in Ontario
Degree Level Expectations
Program Outcomes
Course Outcomes
15

22. Areas of competence in UDLEs
1. Depth and Breadth of Knowledge
2. Knowledge of Methodologies
3. Application of Knowledge
4. Communication Skills
5. Awareness of Limits of Knowledge
6. Autonomy and Professional Capacity
16

23. Areas of competence in GDLEs
1. Depth and Breadth of Knowledge
2. Research and Scholarship
3. Level of Application of Knowledge
4. Professional Capacity and Autonomy
5. Level of Communication Skills
6. Awareness of Limits of Knowledge
17

24. Mapping Program Outcomes
18

25. Impact on OCUL Libraries
19

26. Standardized Library Reporting
20

27. Standardized Library Reporting
OCGS Executive Director asked OCUL to
Dec 1994 consider evaluation measures for libraries
and standardized library reports...
First standardized OCUL guidelines
Oct 1996 OCGS By-Laws & Procedures Appendix H,
Library Assessment
Nov 2004 Review of Appendix H
May 2006 Current OCGS guidelines, Appendix G
QAF Framework, discussion item at May
May 2010 OCUL Directors meeting - pass to OCUL
IR - formation of QAT Working Group
20

28. Standardized Library Reporting
OCGS Executive Director asked OCUL to
Dec 1994 consider evaluation measures for libraries
and standardized library reports...
First standardized OCUL guidelines
Oct 1996 OCGS By-Laws & Procedures Appendix H,
Library Assessment
Nov 2004 Review of Appendix H
May 2006 Current OCGS guidelines, Appendix G
QAF Framework, discussion item at May
May 2010 OCUL Directors meeting - pass to OCUL
IR - formation of QAT Working Group
20

29. Standardized Library Reporting
OCGS Executive Director asked OCUL to
Dec 1994 consider evaluation measures for libraries
and standardized library reports...
First standardized OCUL guidelines
Oct 1996 OCGS By-Laws & Procedures Appendix H,
Library Assessment
Nov 2004 Review of Appendix H
May 2006 Current OCGS guidelines, Appendix G
QAF Framework, discussion item at May
May 2010 OCUL Directors meeting - pass to OCUL
IR - formation of QAT Working Group
20

30. Standardized Library Reporting
OCGS Executive Director asked OCUL to
Dec 1994 consider evaluation measures for libraries
and standardized library reports...
First standardized OCUL guidelines
Oct 1996 OCGS By-Laws & Procedures Appendix H,
Library Assessment
Nov 2004 Review of Appendix H
May 2006 Current OCGS guidelines, Appendix G
QAF Framework, discussion item at May
May 2010 OCUL Directors meeting - pass to OCUL
IR - formation of QAT Working Group
20

31. Standardized Library Reporting
OCGS Executive Director asked OCUL to
Dec 1994 consider evaluation measures for libraries
and standardized library reports...
First standardized OCUL guidelines
Oct 1996 OCGS By-Laws & Procedures Appendix H,
Library Assessment
Nov 2004 Review of Appendix H
May 2006 Current OCGS guidelines, Appendix G
QAF Framework, discussion item at May
May 2010 OCUL Directors meeting - pass to OCUL
IR - formation of QAT Working Group
20

32. From the QAF, the IQAP Should...
Provide evidence that there are
adequate resources to
sustain the quality of scholarship produced
by undergraduate students as well as graduate
students' scholarship and research activities,
including library support, information
technology support, and laboratory access
21

33. Why Change the Current OCGS
Library Guidelines?
Demonstrate the range, scope and depth of
materials...
Describe library consultation, instruction and
assistance services...
Describe library services...
22

34. QAT Template Transitioning Similar Structure
2006: OCGS Appendix G
2011: QAT Template
Library Template
Collections Support Collections
Information Literacy
Instruction & Assistance
Instruction
Library Services Services & Spaces
Resource Sharing
Appendix: Library Learning
Environment
Recommendations
23

35. QAT Template Changing Focus of the Content
Inputs* Outputs Outcomes*
100
75
50
25
0
2007 2008 2009 2010
ACRL Standards for Libraries in Higher Education
http://www.ala.org/ala/mgrps/divs/acrl/standards/standardslibraries.cfm
*Images courtesy of Laurier Archives
24

36. What Does this Mean for OCUL Libraries
• In practical terms, we will need to shift library reports from
predominantly descriptive documents with high level
collection counts to reports that use data to provide the
evidence we are now being asked to provide.
• QAT Template provides the bare bones - ideas for
structuring and planning content to include.
• Accompanying Instructions document also provided
detailing possible sources of data with links to more
illustrative how to's for the use of tools recommended.
25

37. QAT Template and Instructions Approach
• Philosophy used:
• Brief, outcomes-based and program specific
• Buffet of measures and tools (pick and choose for local
needs and data availability)
• Focus on what is available to all or most OCUL schools
• Provide instructions to minimize duplication of effort
within OCUL
26

38. Your Turn: Activity Instructions (5
Minutes)
• Both sides of the room has a purple handout describing suggested
inputs and outputs to measure to demonstrate how information literacy
programs and library collections will sustain the quality of student
scholarship produced.
• Read the suggested measures on your handout and answer the 3
questions that follow.
• Work with your neighbours or individually (5 mins)
• We will ask a couple of participants from each side to share their
responses
• We will also collect your papers so that we can use your input to improve
our work!
27 /

39. Collections Measures Identified (so far...)
Tool to Measure Outputs and Tool to Measure
Inputs
Input Outcomes Outcome
UStat, Jasper
SCOT, SFX, ILS, Journals for Use of program-
Reports (SFX
Ulrich's program specific resources
Queries)
Databases for Satisfaction with LibQual, local
SCOT, (ERM)
program resources surveys, forms
Books for Faculty Web of Science,
ILS, WorldCat
program publications SCOPUS
SCOT, WorldCat, Comparison of all ILL Requests
Jasper Reports
Ulrich's above to peers filled
Comparison of all
SCOT, WorldCat, Purpose of use
above to core MINES
Ulrich's, JCR for resources 28
lists

40. Example Question (collections)
• Question: What journals should I recommend to add to the
collection for this program?
• Data to answer the question:
• Journals by subject requested but not available
• Sources available to OCUL:
• SFX statistics query 12 (local or via jasper reports)
• ILL requesting activity (jasper)
29

41. IL Measures Identified (so far...)
Tool to Measure Output/
Inputs Outputs and Outcomes
Outcome
Number of students in Evidence of impact on Assessments, SAILS,
IL classes student skills Citation analysis
Number of Reference LibQual, local surveys,
Satisfaction (?)
questions feedback forms
Satisfaction (faculty,
LibQual, local surveys,
students)
Number of IL classes feedback forms,
Evidence of integration
Cirriculum maps, DLEs
into cirriculum
Hits on tutorials,
Impact on student skills Assessments
integration into LMS
30

42. Your Turn: Activity
Instructions (10 Minutes)
• Collections side of the room (the
groom) has a blue handout with
suggested measures for measuring
"quality and assessment", IL (bride's
side) has a pink handout for "teaching
and assessment"
• Read the suggested measures on your
handout and answer the 3 questions
that follow.
• Work with your neighbours or
individually (5 mins)
• We will ask a couple of participants
from each side to share their responses
• We will also collect your papers so that
we can use your input to improve our
work!
31 http://www.flickr.com/photos/lifeinaflashbyshaeree/