Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

LAK13 linkedup tutorial_evaluation_framework

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 34 Anzeige

LAK13 linkedup tutorial_evaluation_framework

Tutorial given at LAK13 conference, Leuven, April, 9th, 2013. The presentation is informed by WP2 of the LinkedUp-project.eu that develops an Evaluation Framework for Open Web Data (Linked Data) Applications for Education purposes.

Tutorial given at LAK13 conference, Leuven, April, 9th, 2013. The presentation is informed by WP2 of the LinkedUp-project.eu that develops an Evaluation Framework for Open Web Data (Linked Data) Applications for Education purposes.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Anzeige

Ähnlich wie LAK13 linkedup tutorial_evaluation_framework (20)

Weitere von Hendrik Drachsler (20)

Anzeige

Aktuellste (20)

LAK13 linkedup tutorial_evaluation_framework

  1. 1.   Using Linked Data in Learning Analytics LAK 2013 tutorial EvaluaHon  of  Linked  Data  tools  for  Learning  AnalyHcs   Hendrik  Drachsler  (@hdrachsler,  drachsler.de)   (CELSTEC,  Open  Universiteit  Nederland,  NL)   Eelco  Herder   (L3S  Research  Center,  DE)   Mathieu  d’Aquin  (@mdaquin,  mdaquin.net)   (Knowledge  Media  InsHtute,  The  Open  University,  UK)   Stefan  Dietze     (L3S  Research  Center,  DE)  
  2. 2.   Example of scientific competitions What are the evaluation criteria of Robot Wars? Criteria: •  Damage •  Aggression probabilistic combination of – Item-based method •  Control – User-based method – Matrix Factorization •  Applause – (May be) content-based method 2
  3. 3. RecSysTEL Evaluation criteria 1. Accuracy 1. Accuracy 2. Coverage 2. Coverage 3. Precision 4. Recall 3. Precision 4. Recall 1. Effectiveness of learning 1. Reaction of learner 2. Efficiency of learning 2. Learning improved 3. Drop out rate 3. Behaviour 4. Satisfaction 4. Results Combine approach by Kirkpatrick model by Drachsler et al. 2008 Manouselis et al. 2010 3
  4. 4.   TEL RecSys::Review study Conclusions: Half of the systems (11/20) still at design or prototyping stage only 9 systems evaluated through trials with human users. Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011). Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F. Ricci, L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp. 387-415). Berlin: Springer. 4
  5. 5.   The TEL recommender research is a bit like this... We need to design for each domain an appropriate recommender system that fits the goals and tasks" 5
  6. 6.   TEL recommender experiments lack results “The performance transparency and of different research standardization. efforts in recommender They need tohardly systems are be repeatable to test: comparable.” •  Validity (Manouselis et al., 2010) •  Verification Kaptain Kobold
 http://www.flickr.com/photos/ •  Compare results kaptainkobold/3203311346/ 6
  7. 7.   Data-driven Research and Learning Analytics" EATEL- Hendrik Drachsler (a), Katrien Verbert (b)" " (a) CELSTEC, Open University of the Netherlands" (b) Dept. Computer Science, K.U.Leuven, Belgium" " 7 7
  8. 8.  
  9. 9.   TEL RecSys::Evaluation/datasets " Drachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N., Beham, G., Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. Presentation at the 1st Workshop Recommnder Systems in Technology Enhanced Learning (RecSysTEL) in conjunction with 5th European Conference on Technology Enhanced Learning (EC-TEL 2010): Sustaining TEL: From Innovation to Learning and Practice. September, 28, 2010, Barcelona, Spain." 9 "
  10. 10. 5. Dataset Framework dataTEL evaluation model Datasets Formal Informal Data A Data B Data C Algorithms: Algorithms: Algorithms: Algoritmen A Algoritmen D Algoritmen B Algoritmen B Algoritmen E Algoritmen D Algoritmen C Models: Models: Models: Learner Model A Learner Model C Learner Model A Learner Model B Learner Model E Learner Model C Measured attributes: Measured attributes: Measured attributes: Attribute A Attribute A Attribute A Attribute B Attribute B Attribute B Attribute C Attribute C Attribute C 17 42 10
  11. 11. 5. Dataset Framework dataTEL evaluation model Datasets Formal Informal In Data A LinkedUp we have Data B opportunity to apply a the Data C structured approach to develop a community accepted evaluation framework. Algorithms: Algorithms: Algorithms: Algoritmen A Algoritmen D Algoritmen B Algoritmen B Algoritmen E Algoritmen D 1.  Top-Down by a literature study Algoritmen C 2.  Bottom-up by Models: with experts in the field Models: GCM Models: Learner Model A Learner Model C Learner Model A Learner Model B Learner Model E Learner Model C Measured attributes: Measured attributes: Measured attributes: Attribute A Attribute A Attribute A Attribute B Attribute B Attribute B Attribute C Attribute C Attribute C 17 42 11
  12. 12. 12
  13. 13.   Development  of  the  Evalua=on  Framework     P1: Initialisation P2: Establishment P3: Exit and and Evaluation Sustainability M0-M6: Preparation M7-M18: Competition cycle M18-M24: Finalising Comp etition Revie Final Expert 3x Draft w of EF proposal EF release validation of EF New Refin versio ement n of EF Literature review Group Concept Documentation Cognitive Mapping Mapping Dissemination Practical experiences and refinement Hendrik Drachsler 25 February 2013 13
  14. 14.   Group  Concept  Mapping     •  Group Concept Mapping resembles the Post-it notes problem solving technique and Delphi method •  GCM involves participants in a few simple activities (generating, sorting and rating of ideas) that most people are used to. GCM is different in two substantial ways: 1. Robust analysis (MDS and HCA) GCM takes up the original participants contribution and then quantitatively aggregate it to show their collective view (as thematic clusters) 2. Visualisation GCM presents the results from the analysis as conceptual maps and other graphical representations (pattern matching and go-zones). Hendrik Drachsler 25 February 2013 14
  15. 15.   Group  Concept  Mapping     brainstorm •  innovations in way network is delivered •  (investigate) corporate/structural alignment •  assist in the development of non-traditional partnerships (Rehab with the Medicine Community) sort •  expand investigation and knowledge of PSN'S/PSO's •  continue STHCS sponsored forums on public health issues (medicine managed care forum) •  inventory assets of all participating agencies (providers, Venn Diagrams) •  access additional funds for telemedicine expansion •  better utilization of current technological bridge •  continued support by STHCS to member facilities •  expand and encourage utilization of interface programs to strengthen the viability and to improve the health care delivery system (ie teleconference) •  discussion with CCHN Decide how to manage multiple tasks. 20 Manage resources effectively. 4 Work quickly and effectively under pressure 49 Organize the work when directions are not specific. 39 e t ive ly ly. he eff e ct tive gS ime eff ec tin g et rc es Ra a ou ic. an n s. M res atio a sk ecif 1 na ge rm t. info tan le t t sp Ma no of por ultip ...organize the 5 e a re 4 2 ud is im age m ns ultit hat an ctio 3 5 am ew om ire an id wt nd 2 4 Sc dec ho he 1 d e r kw 3 5 3 an cid wo nd De he ely na 2 4 et 1 4 niz ctiv atio 3 ga ffe orm issues... 5 Or 2 ee inf nt. 4 5 tim of ta 1 ge de r 3 5 na itu impo 2 4 Ma ult 1 1 a m at is 3 5 an e wh 2 Sc cid 4 de 1 3 3 5 2 4 1 3 2 1 rate Hendrik Drachsler 25 February 2013 15
  16. 16.   Group  Concept  Mapping     Hendrik Drachsler 25 February 2013 16
  17. 17.   Group  Concept  Mapping     Hendrik Drachsler 25 February 2013 17
  18. 18.   Group  Concept  Mapping     Hendrik Drachsler 25 February 2013 18
  19. 19.   Group  Concept  Mapping     •  innovations in way network is delivered •  (investigate) corporate/structural alignment •  assist in the development of non-traditional partnerships (Rehab with the Medicine Community) •  expand investigation and knowledge of PSN'S/PSO's •  continue STHCS sponsored forums on public health issues (medicine managed care forum) •  inventory assets of all participating agencies (providers, Venn Diagrams) •  access additional funds for telemedicine expansion •  better utilization of current technological bridge •  continued support by STHCS to member facilities •  expand and encourage utilization of interface programs to strengthen the …”map” the issues... viability and to improve the health care delivery system (ie teleconference) •  discussion with CCHN organize sort Decide how to manage multiple tasks. 20 Manage resources effectively. 4 Work quickly and effectively under pressure 49 Organize the work when directions are not specific. 39 et he effectiv effectively ely . g S time tin Manage resource s d an Ra ation . ks. ecific Technology 1 na ge inform le tas t sp 5 Ma e of rtant. ltip no 4 2 ltitudimpo ge mu ns are 3 a muat is na ectio 5 an ma dir 2 Sc e wh w to en 4 cid wh 1 3 de e ho rk 3 5 cid wo De d Information Services 2 4 e the ely n an 4 1 3 5 ga niz ectiv ma tio Or e eff or 2 4 5 tim inf 1 3 5 na ge e of rtant. 2 Ma ltitudimpo 4 1 1 a muat is 3 5 an wh 2 Sc cide 4 3 de 1 3 5 2 4 1 3 2 1 rate Community & Consumer Views Regionalization Management STHCS as model Financing Hendrik Drachsler 25 February 2013 19
  20. 20.   Group  Concept  Mapping     •  innovations in way network is delivered •  (investigate) corporate/structural alignment •  assist in the development of non-traditional partnerships (Rehab with the Medicine Community) •  expand investigation and knowledge of PSN'S/PSO's •  continue STHCS sponsored forums on public health issues (medicine managed care forum) •  inventory assets of all participating agencies (providers, Venn Diagrams) •  access additional funds for telemedicine expansion •  better utilization of current technological bridge •  continued support by STHCS to member facilities •  expand and encourage utilization of interface programs to strengthen the viability and to improve the health care delivery system (ie teleconference) •  discussion with CCHN Information Services Technology organize sort Community & Consumer Views Decide how to manage multiple tasks. 20 Manage resources effectively. 4 Work quickly and effectively under pressure 49 Organize the work when directions are not specific. 39 et he effectiv effectively ely . g S time tin Manage resource s d an Ra ation . ks. ecific 1 na ge inform le tas t sp 5 Ma e of rtant. ltip no 4 2 ltitudimpo ge mu ns are 3 a muat is na ectio 5 an ma dir 2 Sc e wh w to en 4 cid wh 1 3 de e ho rk 3 5 cid wo De d 2 4 e the ely n an 4 1 3 5 ga niz ectiv ma tio Or e eff or 2 4 5 tim inf 1 3 5 na ge e of rtant. 2 Ma ltitudimpo 4 1 1 a muat is 3 5 an wh 2 Sc cide 4 3 de 1 3 5 2 4 1 3 2 Regionalization 1 rate map Information Services Technology Community & Consumer Views Regionalization Financing Management Mission & Ideology Management STHCS as model Financing ...prioritize the issues... Hendrik Drachsler 25 February 2013 20
  21. 21.   Group  Concept  Mapping     D2.1 Evaluation Criteria and Methods •  Invited 122 external experts •  56 experts contributed 212 indicators for the evaluation framework •  After cleaning -> 108 indicators remained •  26 experts sorted on similarity in meaning •  26 experts rated on priority and applicability Hendrik Drachsler 25 February 2013 21
  22. 22.   Plus Minus Interesting rating Look at and listen to the presentation of the Evaluation Framework Meanwhile…create notes on P: Plus M: Minus I: Interesting Write down everything that comes to your mind, generate as many ideas as possible, do not filter your ideas.
  23. 23.   Group  Concept  Mapping     A point map Hendrik Drachsler 25 February 2013 23
  24. 24.   Group  Concept  Mapping     A cluster map 15 Hendrik Drachsler 25 February 2013 24
  25. 25.   Group  Concept  Mapping     A cluster map 6 Hendrik Drachsler 25 February 2013 25
  26. 26.   Group  Concept  Mapping     Clusters’ labels Hendrik Drachsler 25 February 2013 26
  27. 27.   Group  Concept  Mapping     Rating Map Priority Hendrik Drachsler 25 February 2013 27
  28. 28.   Group  Concept  Mapping     Rating Map Applicability Hendrik Drachsler 25 February 2013 28
  29. 29.   Group  Concept  Mapping     Hendrik Drachsler 25 February 2013 29
  30. 30.   Group  Concept  Mapping     Hendrik Drachsler 25 February 2013 30
  31. 31.   WP2: Literature review
  32. 32.   WP2: Literature review 1. Literature review of suitable evaluation approaches and criteria 2. Review of comprising initiatives such as LinkedEducation, MULCE, E3FPLE and the SIG dataTEL  
  33. 33.   WP2: Literature review
  34. 34.   Many thanks for your attention! This silde is available at: http://www.slideshare.com/Drachsler Email: hendrik.drachsler@ou.nl Skype: celstec-hendrik.drachsler Blogging at: http://www.drachsler.de Twittering at: http://twitter.com/HDrachsler Hendrik Drachsler 25 February 2013 34

×