How to do quick user assign in kanban in Odoo 17 ERP
2011 10 07 (uam) emadrid mfreire ucm sistema analisis usabilidad herramientas autoria contenidos e learning
1. A Usability Analysis System for
e-Learning Authoring Tools
Manuel Freire Morán
eMadrid seminar on Adaptive Systems - 7 October 2011 manuel.freire@fdi.ucm.es
2. Presentation
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
1/20
3. Introduction: authoring & tool evaluation
Contents
System adoption requires content
Introduction
Story-line authoring authoring
Ratings and outcomes reuse
Time-lapse visualization
Future work Better authoring requires creativity support
Concluding remarks
imagine, create, play, share, reflect
creative thinking spiral - Resnick, 2008
low threshold, high ceiling, wide walls
make it as simple as possible - and maybe simpler
Shneiderman et al, 2006
evaluate your tools
2/20
4. Story-line authoring: adventure game authoring
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
3/20
5. Story-line authoring: Weev
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
4/20
6. Story-line authoring: initial question
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
5/20
7. Ratings and outcomes: a-b testing with ratings
Contents
A-B testing: very popular in web usability
Introduction
Story-line authoring Split participants randomly into two groups
Ratings and outcomes Each group uses an interface variant (A and B)
Time-lapse visualization
Outcomes of each group are compared
Future work
Concluding remarks
Outcome of a creative task?
Objective measures: correctness, completeness
Subjective measures: questionnaires (author
satisfaction), ratings (comparative quality)
6/20
8. Ratings and outcomes: evaluating ratings
Contents
N users are requested to rate M users each
Introduction
Story-line authoring All users rated same number of times
Ratings and outcomes No user rated twice by same rater
Time-lapse visualization
No user rates self
Future work
Concluding remarks
Unknown individual rating distribution. How
to determine if results are significant?
H0: count(A rated better than B) is a fair coin toss
one-sided binomial distribution
7/20
9. Ratings and outcomes: statistical treatment
Contents
1x b>a
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization 5/10 4/10 7/10 8/10 6/10
a b a a b
Future work
Concluding remarks
4 x a>b
8/20
10. Ratings and outcomes: experiment
Contents
Experiment with
Introduction
Story-line authoring 20 users (first-time Weev users)
Ratings and outcomes 5-minute tutorial on tool use (applicable to A&B)
Time-lapse visualization
"Little Red Riding-Hood" script and resources
Future work
Concluding remarks
1 hour's time; after editing, rating screen
Outcome
70 total A vs B comparisons; in 40, B > A
p-value: 0.141: does not reject null hypothesis
9/20
11. Ratings and outcomes: simulations
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
10/20
12. Ratings and outcomes: more simulations
Increase ratings-per-user
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
Increase number of users
11/20
13. Ratings and outcomes: increasing ratings
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
12/20
14. Ratings and outcomes: increasing users
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
13/20
15. Time-lapse visualization: idea
Contents
UI instrumentation sends data to server
Introduction
Story-line authoring server timestamps records for each user
Ratings and outcomes UI screen captures
Time-lapse visualization
Action logs
Future work
Concluding remarks
last screen capture sample used for rating
ratings-file is just one more record type
How do you begin to analyze this data?
Build videos from screen-captures
Visualization tool to scan time-lapse data
14/20
16. Time-lapse visualization: interface
user / record
Contents selection time-lapse view of
Introduction records for first
selected user
Story-line authoring
time-lapse view for
Ratings and outcomes next selected user
Time-lapse visualization
Future work
Concluding remarks
detail /
difference view
15/20
17. Time-lapse visualization: insights
User is stuck
Contents
Introduction
Story-line authoring
Ratings and outcomes Creative flow
Time-lapse visualization
Future work
Concluding remarks
Rectangular placement
16/20
18. Time-lapse visualization: tasks and directions
Contents
Tasks for UI evaluation
Introduction
Story-line authoring What did this user do?
Ratings and outcomes
- glance at timeline
Time-lapse visualization What happened between here and there?
Future work - use difference function
Concluding remarks
Still missing
Variable-length time lapses ("time zoom")
Graphical display of textual log data
Filter & Sort
17/20
19. Future work: next experiment
Contents
Server
Introduction
Story-line authoring Finer-grained data collection
Ratings and outcomes Better rating distributions, more ratings
Time-lapse visualization
Support for experimenter note-taking
Future work
Concluding remarks
Analysis tool improvements
Test new automatic layout assistant
Do people use it?
Do people rate "automatically-improved"
storylines as better than-before?
18/20
20. Concluding remarks
Contents
A/B usability testing methodology for
Introduction
Story-line authoring
creative/aesthetic tasks
Ratings and outcomes
Reusable server
Time-lapse visualization
Future work Collects images, logs, rating results
Concluding remarks Configurable rating, allows real-time monitoring
Reusable client
Specific handling for each record-type (ie.: logs)
19/20
21. Questions or comments?
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
20/20