Wilbert Kraan of JISC CETIS discusses profiling IMS QTI for UK HE and FE.

1. Defining a pragmatic QTI profile

2. Introduction
Goals of a CETIS QTI profiling workgroup
•
Assessment system infrastructure
•
The role QTI plays in the infrastructure
•
Ways and methods to determine a profile
•
Means of testing a profile
•
The choice
•
2

3. Goals of a CETIS QTI profiling WG
In order of priority:

1. Determine production profile(s) for UK education (HE? FE +
HE? All ?)
2. Enabling IMS QTI 2.1 public release by producing a testable
profile
3. Help determine a new QTI profile for IMS Common Cartridge
3

4. Assessment system infrastructure
4

5. Assessment system infrastructure
Institution owns

everything
Easy coordination of all

interoperability points (in
theory)
Few resources to make

that coordination
happen (in practice)
Very difficult to meet all

subject communities'
needs
5

6. Assessment system infrastructure
Third party owns

everything but content
Easy coordination of all

interoperability points
Enough resources to

make that coordination
happen
Enough resources to

meet all subject
communities' needs
Politically impossible /

unlikely
6

7. Assessment system infrastructure
Compromise:

Subject centre bank

Institutional learning

system
3d party delivery

Doable coordination of

most interoperability
points
Good spread of

resource load
Still a few potential

bottlenecks
7

8. Assessment system infrastructure
Therefore, for greatest interoperability:

Inverse relation between the complexity of the data

exchanged, and the variation in applications that process that
data
Hand responsibility for component to party with greatest

interest
For profiling this means

Subjects set requirements for rich profile (assuming

compromise or centralised infrastructure)
Else: lowest common technical denominator profile

8

9. The role QTI plays in the infrastructure
QTI as exchange format across the system

+ Consistent semantics

- Difficult profile coordination problem between systems and

over time
QTI as intermediary format between systems

+ Supports legacy systems now

- Semantic roadblocks (unacceptable degradation between

authoring and use)
For profiling, this means:

Intermediary format suits lowest common technical

denominator profile
Exchange format suits rich subject profiles

9

10. Ways and methods to determine a
profile
Community requirements led

+ surest means of achieving fitness for purpose

+ surest means of getting uptake

- can lead to profiles that are technically very difficult to realise

- size of group v. consensus building delicate

Implementation led

+ process of determining a profile is quick and easy

+ near instant implementation

± accurately reflects current practice

- likely to be unfit for purpose (danger of balkanisation)

For profiling, this means

Community led is better, but time-consuming and expensive

Implementation led is quick, cheap but may not meet needs

10

11. Means of testing a profile
Formal testing

+ reliable

- may well not be valid

- v. expensive, continuously (particularly for content)

Self-service testing

± good enough reliability and validity

+ cheapish

Reference implementation

+ reliable

+ valid

+ cheapish

- may cause political ructions

11

12. The choice
Compromise (mostly) institutionally
 
infrastructure owned infrastructure
QTI as an exchange QTI as an intermediary
 
format format
Community led profiling Implementation led
 
profiling
Reference

implementation Self-service testing

Rich, subject specific Small, lowest common
 
profiles denominator profiles
IMS QTI test profile IMS CC profile
 
12

13. Thank you!
Wilbert Kraan

w.g.kraan@ovod.net

13