Modeling Expertise from StackOverflow Data
•Als PPTX, PDF herunterladen•
0 gefällt mir•651 views
The document discusses designing social mechanisms to support informal learning in online communities. It proposes improving a list of social mechanisms from literature and designing and evaluating mechanisms for two case studies. The overall goal is to design incentives using appropriate mechanisms to motivate learning based on organizational objectives and user models. The research will analyze reputation systems to model expertise, focusing on sources, claims, targets and weights to better understand authority in communities.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Modeling Expertise from StackOverflow Data
Ähnlich wie Modeling Expertise from StackOverflow Data (20)
Mehr von Thieme Hennis
Mehr von Thieme Hennis (12)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Modeling Expertise from StackOverflow Data
- 1. A reputation system to model expertise in online communities Doctoral Consortium UMAP2011 + some social mechanisms
- 2. From reputation to social mechanisms Peer-based learning in online communities Challenges Motivation Quality Initial research direction: reputation
- 3. Reputationprinciples Linkedwith trust Reciprocity & Goodbehaviour Someoneelse’s story about me Linkedwithidentity; long-lived A currency, a resource Narrative, dynamic Based on claims, transactions, opinion, rating, endorsements, … Based on indirect information Context of community Contextual Trade-off between trust & privacy Regular assessment of reputationquality Windley, Phillip J., Kevin Tew, and Devlin Daley. A Framework for Building Reputation Systems. In WWW2007.
- 4. Main challenge What: Aggregating and interpreting meaningfulinteractions between & among objects* in online communities Why: to give insight in the value on the object level based on user feedback and usage. * Objects are people and information objects
- 5. So what did I do? Literature Learning theories, knowledge management Trust and reputation Look at different successful reputation systems/techniques Google PageRank, eBay, StackOverflow, Guru, etc. How? value flow context integration sustainability Hennis, T., Lukosch, S., & Veen, W. (2011). Reputation in peer-based learning environments. In O. C. Santos & J. G. Boticario (Eds.), Educational Recommender Systems and Technologies. IGI.
- 6. Value flow Source objects People, organizations, … Target objects Blog posts, articles, … Claims (value statements) Rates, links, recommendations, etc. Implicit, explicit (Farmer & Glass, 2009)
- 12. Concept reputation model – 1/6claim weight claim weight ~ expressiveness of the claim type (i.e. rating versus click)
- 13. Concept reputation model – 2/6target object weight claim weight ~ expressiveness of the claim type (i.e. rating versus click) target object weight ~ importance of the contribution type (i.e. article versus comment)
- 14. Concept reputation model – 3/6affiliated keyword weight apples (0.8) pears (0.2) claim weight ~ expressiveness of the claim type (i.e. rating versus click) target object weight ~ importance of the contribution type (i.e. article versus comment) affiliated keyword weight ~ expressiveness of tag about the target object
- 15. Concept reputation model – 4/6source object weight = authority match keywords with reputation! apples (64) pears (33) kiwis (0) apples (0.6) pears (0.2) kiwis (0.2) claim weight ~ expressiveness of the claim type (i.e. rating versus click) target object weight ~ importance of the contribution type (i.e. article versus comment) affiliated keyword weight ~ expressiveness of tag about the target object source object weight = (source object’s reputation for keyword) / (global rep. value for that keyword) DIFFERENT WEIGHTS FOR DIFFERENT KEYWORDS
- 16. Concept reputation model – 5/6claim value apples (64) pears (33) kiwis (0) apples (0.6) pears (0.2) kiwis (0.2) claim weight ~ expressiveness of the claim type (i.e. rating versus click) target object weight ~ importance of the contribution type (i.e. article versus comment) affiliated keyword weight ~ expressiveness of tag about the target object source object weight = (source object’s reputation for keyword) / (global rep. value for that keyword) claim value = rating (implicit/explicit), i.e. 4/5 stars
- 17. Concept reputation model – 6/6claim value apples (64) pears (33) kiwis (0) apples (0.6) pears (0.2) kiwis (0.2) claim weight ~ expressiveness of the claim type (i.e. rating versus click) target object weight ~ importance of the contribution type (i.e. article versus comment) affiliated keyword weight ~ expressiveness of tag about the target object source object weight = (source object’s reputation for keyword) / (global rep. value for that keyword) claim value = rating (implicit/explicit) 3 claims (one for each affiliate keyword)
- 18. Why is this a useful approach? Target object weight Affiliate keyword weight Claim for keyword k Claim weight Authority Claim value (rating)
- 21. including other weights and metrics (i.e. trust value, network centrality, etc.)
- 23. taking into account all relevant interactions
- 24. Starting point for the design of such a system
- 26. Rather large distributed online network or communityClaim for keyword k Claim weight Authority Claim value (rating)
- 27. Reputation clouds (object model)
- 28. Peer Support Community Trying to get funding (50k) for first prototype Context Blackboard apps & Tags Comment & Rating functionality Application of reputation model Source: Teacher or Student Target: comment, answer Claims: like, page visit, follow Reuse of reputation Award, status, social comparison, gaming mechanisms (competition)
- 29. Contextualized support: content and user support
- 30. But… Not smart to bet on 1 horse, when 3 already dropped out of the race…
- 31. So.. New scope (since 2 weeks) Social mechanisms to design incentive structures to support informal learning in online communities Hennis, T. A., & Kolfschoten, G. L. (2010). Understanding Social Mechanisms in Online Communities. In G. D. Vreede (Ed.), Group Decision and Negotiation 2010. Delft, the Netherlands. Hennis, T. A., & Lukosch, H. (2011). Social Mechanisms to Motivate Learning with Remote Experiments - Design choices to foster online peer-based learning. CSEDU 2011. Veen, W., Staalduinen, J.-P. V., & Hennis, T. A. (2010). Informal self-regulated learning in corporate organizations. In G. Dettori & D. Persico (Eds.), Fostering Self-regulated learning through ICTs. Genova, Italy: Institute for Educational Technologies Italyʼs National Research Council.
- 32. Bouwman et al. (2007) We argue that social software systems should trigger mechanisms that allow us to associate with or form social groups, whether online or in the real world. Such mechanisms would acknowledge human motivations, like eagerness for exploration, curiosity, inquisitiveness, civilization, valuation of belonging, achieving self-realization, enjoying one-self. Bouman, W., Hoogenboom, T., Jansen, R., Schoondorp, M., Bruin, B. de, & Huizing, A. (2007). The realm of sociality: notes on the design of social software. Amsterdam.
- 33. Research objectives Designing incentives: which mechanism to apply when and how? Design of processes Supportive technologies Focused on informal learning in organizations during initial phase – startup phase Cases Philips Lighting Mediamatic (various communities)
- 34. Step 1 – improve list of mechanisms (literature) Matching objectives Organizational objectives User models Fit / Embedding in practice Rhythm Leadership and roles Heterogeneity & Diversity Learning & Networking Reputation & Identity Reciprocity & Feedback Common Ground & Privacy Self-efficacy & Social comparison Autonomy? Empowerment? Curiosity & Provocation IMPROVE
- 35. Step 2 – Design & Evaluate Philips Lighting 3 communities by December Mediamatic Design team Design & Test new things Evaluate existing communities using the Anymeta platform Qualitative Quantitative 50+ small-medium sized online communities Blogging communities Storytelling Event communities & Professional networks
- 36. Rating & Reputation Reciprocity & Feedback Matching online and offline networks through RFID Notifications & Activity
- 37. Profiling & Identity Interaction types ~ Motivation? Personalization & Recommendations
- 38. Keyword based Recommend content, people, projects, …
- 40. Definitions A social mechanism is a plausible hypothesis, or set of plausible hypotheses, that could be the explanation of some social phenomenon. An incentive is any factor (financial or non-financial) that enables or motivates a particular course of action, or counts as a reason for preferring one choice to the alternatives.
- 41. Overall picture Typicalvaluejudgements/claims Context parameters aggregateandinherit/inference repute value judgements: use/rate/recommend context parameters: tag contribute
- 42. 1: Contribution“I write a blog post” Knowledge (topic) What kind of contribution? What kind of topic? contributions Competencies (process) What kind of action? Which competencies involved?
- 45. Example: researcher Value statements: . IF (Impact Factor) . citing . rating/reviews . linking Context: . author & user profiles . citing article or blog . journal . content analysis . keywords --------------- -------------- ----------------- ---------------- ------------- --------------- -------------- ----------------- ---------------- ------------- DOI IF ORCID reputation citations published_in write papers --------------- -------------- ----------------- ---------------- ------------- link ratings BlogRank
- 46. StackOverflow dump Analyze the concept of authority Steps: Define the sources, targets, claims, and weights Sources: SO-users / Targets: Answers, Questions, Users / Claims: votes up/down Context: keywords Conduct analysis and compare the results with the traditional reputation Improve algorithm
- 47. <foods> <pizza title=“Deluxe Pizza”> <name>The Deluxe</name> <toppings> <topping>peppers</topping> <topping>pepperoni</topping> <topping>mushrooms</topping> <topping>cheese</topping> <topping>tomato sauce</topping> </toppings> <price>7.99</price> </pizza> </foods> Research challenges Reputationontologyforp2p learning Formalize common “Sources”, “Claims”, and “Targets” in these communities Combine formalontologiestodescribe topics and skills, anddynamicfolksonomiestodescribecontextual parameters Reuse of reputation information Open Standards & Search Tool development implement, test andimprovereputation system
Hinweis der Redaktion
- I will discuss with you 2 things: what I did, and what I will do. Because they do not really align, I will try to go through the first part as fast as possible.
- The research is based on a vision of education and learning that is lifelong, interactive, networked, and intrinsically related with flexible employment patterns. In this scenario, your online reputation is like a social currency and can lead to new job opportunities. The infrastructure for this scenario is not yet fully established.Learning is becoming a more social, interactive, lifelong activity taking place more and more on the Web.Educational institutions (as well as professional organizations) will move their formal education partially to the Web, meaning that they will host or support wikis and learning networks where learning takes place.Open CoursewareInstitutions and employers need (analytical) tools to assure quality and assess individuals in these online learning environments.Learners need tools to find the right resources and people who can assist them. Tutors need a reason to help others, other than direct extrinsic reward (money).Online communities can be effective learning environments, but there are issues concerned with Trust in people and content in online communitiesMotivation of people to contribute / add value (engagement)In centralized knowledge environments, we often see extrinsic reward systems and institutionalized quality assurance mechanisms to ensure enough quality and contributions. When there are no resources available to do this (such as in peer-based online learning environments), or when this proves to be inefficient use of resources, a more decentralized approach is needed. Successful online communities or marketplaces adopt a variety of social mechanisms and tools to ensure trust, quality and motivation. Reputation systems can cut at both edges; one the one hand, information about value and quality is generated for improved recommendations or search. On the other hand, this information is used to motivate individuals to add value/behave in a desired way.
- Information about the perceived value of a contribution by a community can be used to recognize and reward this person (motivation) and to predict the quality/value of the person and inform future interactors (trust). For example, employers or colleagues in a large organization could be interested in the online reputation of individuals, when they need someone to do a certain task, consult an expert, or want to assemble a group for a project. Also, in more open and social learning environments, reputation can be used to match learners and tutors, or reward contributors of valuable content. These online reputation profiles are not (yet) widely trusted as a measurement of someone’s or something’s quality or value. Can reputation become an instrument for improving self-organization in peer-based learning environments?
- So I looked into literature on trust and reputation systems, on peer-based learning,
- So, on this picture we have a dentist acting as a car mechanic. We know the example: the transitivity of trust (and reputation) depends on the context. We can say, this is a great guy, but why, and when? We need to give the context of his greatness, otherwise it is useless information. I would say, the more specific the context, the better, because it is easier to infer from specific to generic than the other way around. And is it a static context that is restricted by the system, or dynamically added through tags?This is an important challenge in knowledge-intensive online communities: what is the context of value? What is value?
- A nice example, and very relevant for communities of professionals, of how a reputation system contributes to the sustainability, is StackOverflow. As we see in this picture, the reputation of a user is establised based on questions and answers, and how other users vote for those contributions. The user then earns the tags of that question. Those tag-based profiles form the basis of the employment search database, where employers can look for qualified IT people.Sheet 4:SolutionA reputation system that can deal with the dynamics of information is StackOverflow. (plaatje Jon Skeet)So we developed a method that automates the modeling of reputation that is contextualized using keywords, similar to StackOverflow, but more generic so it can be adapted for any knowledge-sharing environment. In addition, it is better able to model relevant context factors, such as authority. (it works a bit like Google: links coming from pages with a lot of visitors carry more weight in the calculation of PageRank than links coming from not popular websites.)Structure of the argument:Collaborative online environments to share experiences, learn and collaborateusually face 2 main challenges: motivation of individuals to contribute quality of the contributionsReputation is a two-edged sword tackling both challenges at the same time: social recognition of highly valued contributions and peer-assessment of the shared information through implicit and explicit ratingEpistemology: what is knowledge? What is valuable knowledge?Context! Difficulty with knowledge-intensive organizations: information is continuously being added and created, and a static taxonomy will not suffice. Plaatje van taxonomy & tag cloud (beidenzijnnodig)In environments where new knowledge is created, you must be able to add keywords to information, to make it fit to your own context. These keywords cannot be known in advance. Hence, reputation (and trust) also depend on context: when your car broke down, you are interested in a mechanic, not a wine-expert.SolutionA reputation system that can deal with the dynamics of information is StackOverflow. (plaatje Jon Skeet)So we developed a method that automates the modeling of reputation that is contextualized using keywords, similar to StackOverflow, but more generic so it can be adapted for any knowledge-sharing environment. In addition, it is better able to model relevant context factors, such as authority. (it works a bit like Google: links coming from pages with a lot of visitors carry more weight in the calculation of PageRank than links coming from not popular websites.)
- Here we see a profile of the number 1 user on the platform, the infamous Jon Skeet.
- And what we see here, is I think very interesting, and relevant for the learning and education field. Because your StackOverflow reputation is in a way nothing else than a diploma showing your qualifications, even with grades! It might now only apply to some industry areas, including IT, but I think this mechanism is very important.
- Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- Leaving this example aside, I will now present the model I developed based on this work, and the corresponding algorithms.Reputation statementSource object makes Claim about Target objectSource and Target objects have reputationsClaimImplicit (click, time of visit) or explicit (direct rating)Multiplication of different weights and claim value about an affiliate keywordatomic a separate claim for each affiliate keyword (weights differ for different keywords, such as the authority of the source)Authority on a keywordThe value of a keyword of a source object relative to the average value of that keyword within the community: weightAffiliate keywordsKeywords linked to a target object, but not necessarily part of target object reputation.
- Wat u hierziet is eengeneriek model wat we hebbenontwikkelddatondersteuningbiedtaan het ontwerp van eenreputatiesysteem. Hierbijhouden we rekening met de genoemdecomplexiteitrond context en waardering, en gaan we uit van het online gedrag van individuen in eenleercommunity. Je brengtals het ware in kaart hoe mensenzichgedragen, en je kanook door bepaaldetypenbijdragenmeertewaarderen, ook het gedragsturen.
- Wat u hierziet is eengeneriek model wat we hebbenontwikkelddatondersteuningbiedtaan het ontwerp van eenreputatiesysteem. Hierbijhouden we rekening met de genoemdecomplexiteitrond context en waardering, en gaan we uit van het online gedrag van individuen in eenleercommunity. Je brengtals het ware in kaart hoe mensenzichgedragen, en je kanook door bepaaldetypenbijdragenmeertewaarderen, ook het gedragsturen.
- Users can see who is an expert only after the question has been asked (otherwise they will directly ask their colleague without posting the question: you want the interaction to take place online).
- On StackOverflow, users do aggregate a specific reputation profile using the affiliate keyword method, BUT it does not consider authority. We can have a look what difference it will make to reputations of SO-users. Maybe the algorithm can be improved, because of recursive behavior or because the high-authority persons have too much influence on the reputation of low-reputation objects.