According to studies into learning at work, interpersonal help seeking is the most important strategy of how people acquire knowledge at their workplaces. Finding knowledgeable persons, however, can often be difficult for several reasons. Expert finding systems can support the process of identifying knowledgeable colleagues thus facilitating communication and collaboration within an organization. In order to provide the expert finding functionality, an underlying user model is needed that represents the characteristics of each individual user. With these slides, we present the APOSDLE People Recommender Service which is based on an underlying domain model, and on the APOSDLE User Model.

1. Recommending Knowledgeable
People
in a Work-Integrated Learning System
(RecSysTEL Workshop at EC-TEL 2010)
Günter Beham, Barbara Kump, Tobias Ley, Stefanie Lindstaedt

2. Organisa(ons
try
to
transform
workplaces
into
more
effec(ve
learning
environments
October 20, 10 / 2 Executive Board Meeting, Graz [e.g., Billet, 2000]

3. I am filling out this new Hmm, not really but maybe
report form. Any idea what Paul could help here. He
all these abbreviations filled a similar report last
mean? week.
Knowledge
Workers
seek for
inter-personal
help
[Kooken et al., 2007]

4. Challenge: Finding knowledgeable people
for a topic within a company

5. APOSDLE Vision
Enable learning directly at the
workplace
Support people in sharing
their knowledge
Reuse available resources as
learning materials

6. The APOSDLE Approach:
Connecting user activities with
organisational models
to recommend knowledgeable people

7. September 29, 2010 / 9
The APOSDLE People Recommendation Workflow

8. How APOSDLE looks like
• Screenshots
vom
APOSDLE
Prototypen:
Suggests
und
Coopera;on
Wizard
People
Company
Resources
October 20, 10 / 10

9. September 29, 2010 / 11
3-Tier Architecture of APOSDLE
Services

10. September 29, 2010 / 12
Organisational Models

11. September 29, 2010 / 13
Maintaining the APOSDLE User Model
100 ...
80 Sharing a Resource
60 Being Contacted
40 Performing a Task
20
Viewing a Resource
0

12. September 29, 2010 / 14
Identifying Knowledge Levels
Beginner
Advanced
100 Expert
80
60
Expert
40 Advanced
20
Beginner
0

13. September 29, 2010 / 15
Where APOSDLE Services come into play
Detecting the learning need of a worker
Finding a knowledgeable person who can help

14. September 29, 2010 / 17
Testing and evaluating the APOSDLE
User Model and Services
 Simulation Study
 Comparison of different algorithms for maintaining the user model
 Which algorithm can detect a user‘s knowledge level best?
 Workplace Evaluations
 Deployment of APOSDLE in 2 real work environments
 Comparison of knowledge level as diagnosed by APOSDLE with self
-assessment

15. September 29, 2010 / 18
Simulation Study (Example Design)
 Fixed Parameters
 Number of persons, number of user events, inference algorithm
 Variable Parameters
 User behavior (Beginner, Advanced, Expert)
Level Advanced Level Beginner Level Expert
Behavior 60% Behavior 60% Behavior 60%
norm. norm. norm.
Inference Frequency Inference Frequency Inference Frequency

16. Simulation Result (Example)
6 Persons
1 Topic
50 events/Behavior type
Inference: Weighted
Frequencies

17. Simulation Result (Example)
6 Persons
1 Topic
50 events/Behavior type
Inference: Weighted
Frequencies with
windowing

18. Deploying APOSDLE in real
workplaces

19. September 29, 2010 / 22
Real-world evaluation in 2 Organisations
How well does APOSDLE detect the workers‘
Work topics?
Knowledge levels?
Library of a Distance University Innovation Management (ISN)
 10 Users, only 5 Users willing to  6 Users
participate in the self
 Used APOSDLE for 3 Months
-assessment
 Self-assessment and peer
 Used APOSDLE for 4,5 Months
-assessment (using cards)
 Self-assessment (online
questionnaire)

20. September 29, 2010 / 23
Library of a Distance University
How well does APOSDLE detect the workers‘ work topics?
APOSDLE user model
Work Topic Non-Work Topic Total
Work Topic 133 81 214
self-assessment
Non-Work Topic 4 12 16
Total 137 93 230
In many cases, APOSDLE did not „know“ that topics were a user‘s work topics

21. September 29, 2010 / 24
Library of a Distance University
How well does APOSDLE detect the workers‘ knowledge
levels?
APOSDLE user model
No Work
Expert Advanced Beginner Total
Topic
Expert 1 27 11 44 83
Advanced 3 39 20 31 93
Self- Beginner 2 24 6 6 38
assessment
No Work
0 4 0 12 16
Topic
Total 6 94 37 93 230
APOSDLE classified users mostly „advanced“ where they regarded themselves as
„beginners“ or „experts“

22. September 29, 2010 / 25
Innovation Management
How well does APOSDLE detect the workers‘ work topics?
APOSDLE user model
Work Topic (%) Non-Work Topic (%) Total
Work Topic 356 (41.7) 334 (39.0) 690 (80.7)
self-assessment
Non-Work Topic 51 (6.0) 114 (13.3) 165 (19.3)
Total 407 (47.7) 448 (52.3) 855 (100)
In many cases, APOSDLE did not „know“ that topics were a user‘s work topics

23. September 29, 2010 / 26
Number of user interactions with APOSDLE
The more interaction with APOSDLE, the more correct
detections of work topics and non-work topics

24. September 29, 2010 / 27
Innovation Management
How well does APOSDLE detect the workers‘ knowledge
levels?
APOSDLE user model
No Work
Expert Advanced Beginner Total
Topic
Expert 27 130 29 162 348
Advanced 11 73 19 82 185
Self- Beginner 7 49 11 90 157
assessment
No Work
5 36 10 114 165
Topic
Total 50 288 69 448 855
APOSDLE classified users mostly „advanced“ where they regarded
themselves as „beginners“ or „experts“.

25. September 29, 2010 / 28
Discussion of Outcomes
 In many cases, APOSDLE was not able to identify a
user‘s work topics
 Users NEVER dealt with this topic within APOSDLE
 Evaluation period too short? Rather: not enough system
usage during evaluation period
 In many cases, APOSDLE erroneously diagnosed
„advanced“ level
 Improve algorithms
 Self-assessment may also be erroneous/biased Better „external measure“
for workplace evaluations??

26. September 29, 2010 / 29
Outlook
 Improving algorithms for diagnosing user knowledge
 Cross-validation with existing data
 Further evaluations of the user model in other organisations
 Combination of different recommendation strategies
 Evaluating People Recommendations
 Evaluation Setup?
 Lab studies
 Field studies

27. ut
Find more abo
APOSDLE on org
http://www.aposdle.
Contact:
ham
Guenter Be at
kn ow-center.
gbeham@