1. TOWARDS QUALITY IN E-LEARNING QUALITY
ASSURANCE - EPPROBATE INTERNATIONAL
COURSEWARE LABEL
Marjan Miloshevic
Danijela Miloshevic
Technical Faculty Čačak - University of Kragujevac, Čačak, Serbia
{marjan, danijela}@tfc.kg.ac.rs
Suzana Loshkovska
Faculty of Computer Science and Engineering Skopje, Macedonia
suzana.loshkovska@finki.ukim.mk

2. Content
2
 Quality of E-learning
 Certification approaches and quality schemes
 DL@WEB – BADEN
 Epprobate
o Aim, objectives
o Procedure
o Self-assessment document
o Evaluation
 Conclusion

3. E-Learning – all present
paradigm3
E-Learning
Formal
educatio
n
NON
FORMAL
EDUCATI
ON
LL
L

4. The issue of quality
The standard quality assurance (QA) procedures
are not sufficient.
QA in e-learning is a non issue for many, especially
for the QA agencies
Numerous international projects were conducted
to form a comprehensive, yet usable framework for
quality assurance of learning.
Specialized organizations have developed their own
benchmarking procedures and tools and
established labels as brands
4

5. Quality of E-Learning – The
Terms5
 There are several definitions of quality of e-
learning, and each depends on its
scope, objectives, focus and the methodology
of the quality approach.
 In this context, quality of e-learning is related
to all: processes, products and services for
learning, education and training supported by
the use of information and communication
technologies.

6. Issues of internationality
6
 We need
 an unified label
 We have...
 Language issues
 Cultural differences
 Law issues

7. International Quality Labels
7
Label name Level Review process Type Reviewers
UNIQUe Universities Publicly documented, on site Quality mark Internal
E-xcellence University
program
Publicly documented; on-line
and on-site (optionally) Benchmark Internal
ECBCheck University
program
Documents and
accompanying web-site are in
construction phase
Quality Mark Internal
SEVAQ+
Universities,
Vocational
studies
Supporting documents and
web-site are are available; on-
line
Self-evaluation
tool
Stakeholders
(teachers,
students, ...)
Procert
Courseware
for IT
specialists
Documents are not publicly
available Benchmark N/A
epprobate General
courseware
Quality grid is public; review
conducted on-line Quality Mark Internal and
external

8. BADEN – Gathering Western
Balkan for Improved Education8
 BAlkan Distance Education Network
 BASIC GOALS:
 affirmation and use of professional knowledge via e-
learning and distance education
 Еducation аccessible to all, no matter of place
 Improvement of academic space in WB region

9. BADEN’s goals
9
 Development of informatics society in the region of Balkan
through improving and promoting quality e-learning and
distance education,
 Modernization of education by promoting regional and
international cooperation by connecting institutions and
individuals as well as by exchange of information in the fields
of e-learning and distance education,
 Tuning into The European Higher Education Area (EHEA),
 Raising the level of pedagogical and technical knowledge and
skills in the fields of e-learning and distance learning,
 Strengthening of educational and civil sector for successful
application of e-learning and distance education,

10. Goals of BADEN (2)
10
 Fostering and cherishing research spirit for further
development of e-learning and distance education,
 Strengthening of responsibility within pedagogic practice in
the field of e- learning and distance education,
 More effective employment and improvement of economic
development,
 Raising social awareness about the significance of e-learning
and distance education in all aspects of formal and non-formal
education
 Fostering and strengthening of responsibility for individual
development and improvement of status of smaller and wider
communities through activities of protection human and
minority rights, labor, racial and sex equality and right on life in

11. Activities
11
 Education through professional and research seminars and
summer schools to enable providing mastering skills and
knowledge in the fields of e-learning and distance education,
 Organization of professional lectures, trainings, symposiums and
conferences with topics about e-learning and distance education
 Promotion of distance learning through cooperation and
participation on manifestations of domestic, regional and
international importance, competitive, propaganda and
commercial character,
 Participation in realization of projects at national, regional and
international level as well as in other fields which have the need
for implementing this kind of education, independently or
collectively, applying at funds in the country and abroad,
 Promoting and placing of innovative and creative professional
and research solutions,

12. Activities
12
 Carrying out commercially readymade
solutions, individual courses or moderation services in
the fields of e-learning and distance education,
 Publishing activity from the primary and related fields
which include application of e-learning techniques to
realize their activities,
 Creating and continual improvement of electronic
knowledge basis from the fields of e-learning and
distance education,
 Promotion of the lifelong learning concept by applying e-
learning and distance education,
 Creating and realizing accredited programs for
professional training of the employed in education.

13. epprobate
13
 Self-paced e-Learning products and services are
mainly based on courseware.
 At present, the worldwide market reached $32.1 billion -
with a five-year compound annual growth rate of 9.2%
according to Ambient Insight!
 There is a huge provision of eLearning courseware in
the world.
 Every day different kind of new material like Web Based
Training (WBT), Computer Based Training (CBT), Game
Based Learning (GBL), Videos, Open Educational
Resources (OER), Mobile Learning Apps etc. are added.
 For the learner it becomes very difficult to distinguish
what is of good quality and worth buying.

14. epprobate
14
 epprobate (from the Latin 'approbare', to
approve) is the first international quality label
for eLearning courseware.
 The quality label is an initiative of three
organisations:
 Learning Agency Network (LANETO),
 Agence Wallonne des Télécommunication (AWT)
 e-Learning Quality Service Center (eLQSC).

15. epprobate process
15
1. Contacting the National Partner (BADEN for Western Balkan).
2. A short review – feasibility study.
3. The National Partner gives a recommendation and sends an offer
4. Reviewers who are able to evaluate the courseware from the points of
view of eLearning pedagogy, content accuracy, technical delivery, and
acceptability by the target group are selected.
 The producer is asked to help identify representatives of the target group
 Payment for review is paid at this point
5. Producer completes a Self Assessment Document answering a
range of questions about the courseware and gives access to the
courseware to the review team.

16. epprobate process
16
1. After the reviewers have evaluated the courseware and reached an initial set of
conclusions, the head reviewer contacts the producer and sends a report to brief
the producer for an on-line meeting with the review team.
2. During the meeting the producer receives feedback from the reviewers about the
courseware in terms of the quality criteria of the epprobate Quality Grid, and has
an opportunity to provide additional information about the courseware to address
any concerns.
3. After this meeting the review team revises their report and sends it together with
their recommendation as to whether the label should be awarded or not to the
international awarding committee.

17. Epprobate process
17
1. The international awarding committee is responsible
for the final decision about the awarding of the
epprobate label and it will consider the
recommendations of the review team within the wider
context of its knowledge of global e-learning
standards.
2. The producer receives formal written feedback from
the international awarding committee, and if the
courseware has been successful in its application it
will be officially labelled by epprobate.
3. Each year an epprobate award for excellence will be
awarded to the courseware judged to be most
outstanding amongst those awarded the epprobate
label during that year.

18. Epprobate quality grid
18
Each courseware is evaluated for
A. Course design
B. Learning design
C. Media design
D. Content

19. A. COURSE DESIGN
19
 Provision of course information, learning
objectives and instructional guidance
 The courseware provides learners with all the
information needed, including clear and appropriate
learning objectives and sufficient instructional
guidance.
 Constructive alignment
 The learning objectives, instructional strategies and
assessment processes (where provided within the
courseware) are congruently aligned with each
other, so that the instructional strategies used can be
seen to be effectively assisting learners in achieving
the intended learning objectives.

20. Self-evaluation
20
Assessment and
importance
Comments
For each topic the producer fills information in the following table
Example
Assessment
and
importance
Comments
C3
This course is intended to meet the needs of students who wish to have a
basic introduction to the nature of design thinking. It is expected that the
audience will therefore be self selecting. There is no attempt made to adapt
the materials to individual needs though the activities allow for expression of
personal perspectives.

21. Review
21
Assessment and importance Good features Issues Other comments
For each topic, each reviewer fills information in the following table
There is no intention for any single criterion to be essential, but a
courseware supplier in their self assessment document indicates to
what extent they meet a specific criteria.
To every characteristic is assigned a mark:
A - Exceeds the requirements of the criterion
B - Meets the requirements of the criterion
C - Meets some but not all of the requirements of the criterion
D - Fails to meet the requirements of the criterion

22. Review - example
22
Assessment
and
importance
Good features Issues Other comments
A1 The content and
activities are well
balanced and form a mix
of examples and
individual tasks, followed
by a narrative guidance.
Assessment is modelled
to be mostly self-
conducted in form of a
practical task or
discussion.

23. B. LEARNING DESIGN
23
1. Learner needs
 The instructional strategies meet the learner's needs, and are matched to the learner’s
profile (skill level, age group, specific constraints, etc.)
2. Personalisation
 The courseware allows personalization of the learning process and supports the
development of learner autonomy.
3. Instructional strategies
 The courseware provides an appropriate range of instructional strategies that motivate
and engage learners and support active learning.
 The instructional strategies support learning through an appropriate range of the forms
of learning:
 Acquisition – where learning is done by listening to, reading, or watching materials that are
presented in such a way as to gain the attention and interest of the learner
 Inquiry - where the learner is provided with a focus for enquiry, resources, and guidance for the
learning process
 Practice - where the learner is provided with interactive exercises, games, opportunities to
interact with representations of the domain, or other activities that provide feedback
 Communication - where the learner is encouraged and supported in communicating with other
learners (formally and/or informally) and/or with tutors (e.g. coaching)
 Construction – where the learner is provided with opportunities to learn, or to demonstrate the
outcome of their learning, through producing something - such as an essay, or a design.

24. C. MEDIA DESIGN
24
1. Media integration
 The utilization of media (i.e. text, pictures, audio, and
video) and tools effectively enhances comprehension of
the course content and empowers implementation of the
instructional strategies.
1. Interface
 The courseware interface (in terms of navigation and web
design) is user friendly, corresponds to current
practice, and allows learners to efficiently monitor their
progress through the course.
1. Interoperability and technological standards
 The courseware complies with common technical
standards (e.g. SCORM, IMS) thus supporting
interoperability.

25. D. CONTENT
25
1. Accuracy and values of content
 The content is an accurate representation of the domain
from a particular perspective or range of perspectives.
Where appropriate issues of values are addressed openly.
The content is written so as avoid culturally biased terms
and examples.
2. Intellectual property rights
 The courseware supplier has the rights to use the
materials in the courseware. Where material is provided as
an OER the rights of use and reuse are clearly stated with
an appropriate Creative Commons license.
3. Legal compliance
 The content abides by all relevant national and
international legislation with reference to content, including
laws related to bias, slander, and promotion of specific
values.

26. Conclusion
26
 There is a certain progress made in quality
assurance in e-learning.
 The challenges remains to be faced with, such as
internationalization, distinction in QA scale, and
usage of objective and comprehensive – yet not
too complex – methodology.

27. Conclusion
27
 Epprobate is an initiative that tries to handle
mentioned challenges.
 It is an international label, which engages
numerous national partners, it treats e-learning at
the courseware level, making the QA available to
different types of producers (universities,
education agencies, companies) and it uses a
methodology that compounds all vital criteria
through a well defined procedure.