Over the last few years the Australasian Council of Online, Distance and eLearning (ACODE) have facilitated two major Benchmarking Summits in Sydney and Canberra, using the ACODE Benchmarks for TEL. These have been unprecedented events within the Australasian higher education, with 35 higher education institutions from five countries involved. To participate each institution first had to undertake a self-assessment of their capacity in TEL against the Performance Indicators in the tool, and then be willing to share this with the other institutions involved. Each institution assessed, at a minimum, two of the benchmarks, with many institutions doing far more. However, for an institution to maximize its experience with this activity they need to be starting their engagement well prior to the formal benchmarking activity. This workshop will help those new to the ACODE benchmarks understand what is required when using this tool. It will provide the wherewithal to ensure your participation is undertaken in a rigorous way. More importantly, it will provide a practical way to facilitate an internal activity, potentially with a view to then be involved in an inter-institutional activity. We will work through a number of different scenarios to help you understand the many facets needing to be considered in undertaking such an activity, and you leave with a strong plan of action for your institution to enhance its capacity in TEL.

2. Preparing for an inter-institutional
Benchmarking activity using the ACODE
Benchmarks for TEL
Associate Professor Michael Sankey
Vice-President ACODE
Tuesday March 7, 2017
www.acode.edu.au

3. Introduction
• In 2014 & 2016 ACODE ran two major Benchmarking Summits
in Sydney and Canberra, using the ACODE Benchmarks for TEL.
• Unique events in Australasian HE, with 35 institutions from 5 countries
engaged so far.
• Each first undertook a self-assessment of their capacity in TEL against the
Performance Indicators, then shared this with the other institutions involved.
• Each assessed a minimum of two benchmarks, with many doing far more.
• To maximize the value of this activity you need to start engaging with the tool
well prior to the formal benchmarking activity.
• This workshop will help those new to the ACODE benchmarks:
– understand what is required when using the tool,
– provide a guide to ensure participation is undertaken in a rigorous way, and
– find a practical way to run an internal activity, before being involved inter-institutionally.
• We will work through a number of different scenarios to help you understand
the many facets needing to be considered in undertaking this activity, and
look at a plan of action for your institution to be involved.

4. ACODE
ACODE is a representative body; a council, not a society or guild. It exists to:
– disseminate and share knowledge and expertise;
– support prof development & provide networking opportunities;
– investigate, develop & evaluate new approaches;
– advise and influence key bodies in higher education; and
– promote best practice.

5. Well used e-learning quality & benchmarking tools
Tool Description Change Theory Validation References
ACODE
Benchmarks
for TEL
Set of benchmarking statements designed to assist institutions
improving the quality of technology enhanced learning. Statements of
good practice provided along with a ranking scale. Focus is on a team-
based self-assessment. CC Licensed
Collaborative
Benchmarking
Face validity supported by expert
review. Revised following experience in
implementation
Sankey et al.
(2014, 2016)
EADTU E-
xcellence Next
A framework operated by the European Association of Distance
Teaching Universities. Set of quality indicators /benchmarks provided
to be used to engage in self-assessments which may be referenced by
external QA schemes. CC Licensed
Collaborative
Benchmarking
Face validity supported by expert
review. Revised following experience in
implementation
Ehlers (2012)
EADTU (2012)
EFMD
Certification
of E-learning
(CEL)
3 year accreditation scheme for e-learning management operated by
the European Foundation for Management Development (EFMD). It
including mix of self-assessment and a detailed accreditation audit. It
has an 18 month review
None Face validity supported by expert review Ehlers (2012)
EFQUEL
UNIQUe
Certification
A European Foundation for Quality in E-Learning quality certification ,
for courses, programmes, institutional systems, to certifies the whole
institution. The Guidelines are supplied with supporting questions.
Formalised process of of self-assessment and peer review similar to an
accreditation audit. Restricted to eligible institutions.
None Face validity supported by literature
review and extensive reviews
undertaken by experts and quality
assurance bodies
EFQUEL (2011)
Ehlers (2012)
e-Learning
Guidelines
(eLG)
A guide to designing, implementing and enhancing eLearning. A
framework of questions to encourage reflection by a range of
stakeholders. No detailed good practice guidance provided. CC
Licensed
None Face validity supported by expert review
and literature review. Revised following
experience in implementation.
Suddaby and
Milne (2008)
e-Learning
Maturity
Model (eMM)
Quality improvement framework incorporating a benchmarking
process and extensive knowledgebase. Extensive set of processes
broken down into detailed organisational practice statements. CC
Licensed
Maturity
Model.
Process & practice revised after 3
rounds of expert consultation conducted
internationally, extensive set of cases &
a peer-reviewed framework.
Marshall (2012a;
2012b)
Neal & Marshall
(2008)
Taking the
Lead
Not a quality framework, rather a tool for identifying the strategic
goals for e-learning to be improved.
None Face validity supported by literature
review and case studies.
Quality
Matters (QM)
Quality checklist designed to improve individual online courses
through a form of audit process. Checklist items supported by
descriptions of good practice applied by reviewers, after training.
Focus is on PD of staff for online teaching and quality assurance of
courses. Not for profit framework requiring a license to use.
None Face validity supported by literature
review and case studies.
Varonis (2014)

7. The Benchmarks
• Originally developed
back in 2004
• Used many times
• In 2014 we shifted the
focus away from
‘eLearning’ to
‘Technology Enhanced
Learning’ (TEL)
• Developed the self-
assessment templates
• Provide guidelines for
using the instrument

8. 1. Institution-wide policy and governance for technology enhanced
learning;
2. Planning for institution-wide quality improvement of technology
enhanced learning;
3. Information technology systems, services and support for
technology enhanced learning;
4. The application of technology enhanced learning services;
5. Staff professional development for the effective use of
technology enhanced learning;
6. Staff support for the use of technology enhanced learning;
7. Student training for the effective use of technology enhanced
learning;
8. Student support for the use of technology enhanced learning.
The 8 Benchmarks for TEL

9. Extension
• The methodology provides an institutions with:
1. a platform to self-access their standing against
some/all of the 8 benchmarks, and to stimulate
meaningful conversations, at a local level, around
how you are using technology to support your L&T.
2. an opportunity to share & learn from each other,
based on their individual institutions responses
(via an inter-institutional event every two years).
3. an enduring record of this via the newly developed
online tool (or site)

10. This resulted in…

11. Institution BM 1 BM 2 BM 3 BM 4 BM 5 BM 6 BM 7 BM 8
Asia Pacific International College X X
Auckland University of Technology X X
Australian Catholic University ¢ X¢ X¢ X¢
Australian National University ¢ ¢
Charles Sturt University ¢ ¢
Christchurch Polytechnic X X
Curtin University X X
Edith Cowan University ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢
Federation University X X¢ X¢ ¢ X¢
Flinders University X X
La Trobe University ¢ ¢ ¢
Lincoln University ¢ X¢ ¢ X¢
Macquarie University ¢ X ¢ X¢
Monash College ¢ ¢ ¢ ¢ ¢ ¢
Open University - UK X X X¢ X¢
Queensland University of Technology X X
RMIT University ¢ ¢ ¢
Swinburne University ¢ ¢
University of Auckland ¢ X ¢ ¢
University of Canberra X¢ ¢ X ¢ ¢
University of Melbourne ¢ ¢
University of New England X ¢ ¢ X¢ ¢ X X
University of Notre Dame ¢ ¢
University of Otago X¢ X¢ X¢ X¢ X¢ ¢ ¢ ¢
University of Southern Queensland ¢ ¢ ¢ ¢ X¢ X¢ X X
University of South Africa X¢ X¢ X¢
University of the South Pacific X X ¢ ¢
University of the Sunshine Coast ¢ ¢ ¢
University of Tasmania ¢ ¢ ¢
University of Technology Sydney X X ¢ ¢
University of Western Australia X X
University of Western Sydney X ¢ X¢ X ¢
University of Wollongong X X X X
Victoria University (Melbourne) X ¢ ¢ ¢ X
Victoria University Wellington X¢ X¢ X¢ X¢ X¢ X¢ X¢ X¢
Total (2014) 11 8 8 10 12 9 5 6

12. acode.edu.au

14. But to get to this point…
Managing the logistics
• We first had to do a self
assessment
• Pull people together from
different sections
• Agree on where we stood
• Provide a rationale and
evidence as to why

15. Short activity: Benchmark 1
Institution - wide policy and governance for
technology enhanced learning
• Performance indicator
1. Institution strategic and operational plans support and promote the use of
technology enhanced learning.
2. Specific plans relating to the use of technology enhanced learning are aligned with
the institution’s strategic directions and operational plans.
3. Planning for the ongoing use of technology enhanced learning is aligned with the
institution’s budget process.
4. Institution policies, procedures and guidelines provide a framework for how
technology enhanced learning should be used at both a course and program level.
5. Policies, procedures and guidelines on the use of technology enhanced learning are
well communicated and integrated into processes and systems.
6. The institution has established mechanisms for the governance of technology
enhanced learning that include representation from key stakeholders.
7. Authority and responsibility for the operational management of the technologies
used to enhance learning and teaching are clearly articulated.
8. The institution uses a clearly articulated policy framework and governance structure
then deciding on the adoption of new technologies.

16. At USQ
• Worked with the word
templates
• Set up in SharePoint
• Had a person .2 of a
workload to wrangle
• Each shared their
individual ratings
• Then collated the
feedback on the word
docs
• Then entered the
collated data into the
online tool

17. The online tool

18. Split indicators

19. Victoria University Wellington
• They used Google
Docs as the main
tool for
collaborating on
the assessments.
• Created by a simple
copy/paste of the
word template.
• The goal was to use
something easy for
a range of staff to
edit the files and
add content from a
range of sources
directly.

20. • VU staff made individual assessments based on
local knowledge and evidence (policy and other documentation).
• They were completed by groups of 2-3 staff with a sub-group
leader responsible for collating and reporting the assessments.
• The staff involved were drawn from across the university:
 Centre for Academic Development
 Information Technology Services
 Vice-Provost Research
 Library
 Faculty of Science
 Faculty of Education
 Student Learning
• The initial assessments were then workshopped with all
participants and held in an active learning space that encouraged
collaborative editing.

21. • This information was uploaded into the
online system prior to the Canberra workshop.
• Edits and revisions were made back into the Google Docs during
the workshop and immediately after to reflect the feedback and
discussion at Canberra.
• The resulting information was then edited by CAD and ITS staff to
produce an internal report with 13 recommendations arranged in
four major themes:
– Digital governance;
– Digital teaching;
– Digital learning; and
– Digital understanding.
• The draft report was shared with the complete team to ensure
accuracy of the content and agreement that the
recommendations reflected a consensus on the important
elements needing improvement.

22. • Each benchmark performance indicator result was
presented in context for Victoria against the complete data set:
• These were accompanied by a narrative outlining the rationale and
evidence for the assessment and a section labelled “possibilities for
improvement.” The latter section outlined what could be done to
improve the individual indicator. These were used to identify the
recommendations, which typically addressed a range of indicators
across the benchmarks.

23. Key to success
• Early involvement with a diverse team of staff.
• Two workshop sessions were needed to introduce
the tool and complete the initial evidence
gathering, each running for three hours.
• Engaged team leaders made a significant
difference in the extent to which evidence was
gathered and synthesised, but all teams provided
a comprehensive body of evidence.
• Project leaders needed to spend a significant
amount of time turning the information into a
consistent report but this was useful as a means of
summarising progress to date against our Digital
Strategy which is due for renewal in 2017

24. RMIT University
• WE EVALUATE EACH BM by…
1. Form groups of 6 – 10 RMIT staff representing different
areas to evaluate each BM
2. Each BM group participant completes & submits a self
assessment template (via Google form)
3. Group evaluation meetings are held for each BM, where
participants discuss their performance indicator ratings &
evidence. The group must then agree on an overall rating
for each performance indicator
4. Group evaluations & recommendations for improvement
are collated into a report for the VCE

25. Step 1. Complete the self assessment form
• The Google form link in ‘BM Group Member Information’ or
‘Benchmark Self Assessment Forms’ on the site
• Select a rating of 1-5 for each performance indicator (PI)
• Note your rationale & evidence to support each rating
• Note suggestions for improvement at the end of the form
• If you cannot answer one of the sections on the form, leave it
blank and move to the next PI (the reason we have groups)
• You should not spend more than 2 hours completing the form,
or 10 minutes per PI
The self assessment must be submitted prior to group
evaluation meetings
PROCESS FOR GROUP MEMBERS

28. 1. Please note your
email, so we don’t
keep pestering you
2. Read the Scoping
and Good Practice
statements
3. These inform your
consideration of how
you will rate an
indicator

29. Example: SELF ASSESSMENT

30. Example: SELF ASSESSMENT
1
2

31. Step 2. Group evaluation meeting
• Each group member briefly describes their PI ratings &
supporting evidence/rationale
• The group will discuss & choose overall PI ratings for RMIT
• Final ratings are based on group consensus, where
possible
• BM group leaders will facilitate discussions
Group evaluations will be included in the final report
Self assessment forms will be not be reported
PROCESS FOR GROUP MEMBERS

32. Then we have the conversation

33. In 2016 on average 15 participants
per institution, with some 401 people
participating overall.4
Head of TEQSA

34. The beauty of the beast
• The beauty of benchmarking is not around which tool
or set of standards you are using, it's about the
dialogue that emerges and the sharing of practice that
is the real winner for all concerned.
• It opens the door for further collaboration.
• It serves as a mechanism to facilitate discussion at
senior leadership level.

35. Conclusion
• Many of the issues we face can be remediated
by simply taking the time to self-assess against a set
of quality performance indicators.
• We then extend this by sharing our current practice
with those in similar circumstances.
• This builds stronger ties and provides our
institutions with the wherewithal to meet the
unique challenges of building a strong digital future.
• The ACODE Benchmarks provide a
catalyst to help make this happen
• We are not alone
www.acode.edu.au

36. This June
http://www.open.ac.uk/acode-uk/

37. http://www.open.ac.uk/acode-uk/