“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities

1. “We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities Denae Ford North Carolina State University Kristina Lustig Stack Exchange, Inc. Jeremy Banks Stack Exchange, Inc. Chris Parnin North Carolina State University @DenaeFord

2. 2 source: https://media.giphy.com/media/xT9IgHKDptUFWWMkve/giphy.gif source: https://media.tumblr.com/3d150f8da60feb45ed3254b5721e5c44/tumblr_inline_mjw0l2etMI1qz4rgp.gif

3. 3 source: https://media.giphy.com/media/xT9IgHKDptUFWWMkve/giphy.gif source: https://media.tumblr.com/3d150f8da60feb45ed3254b5721e5c44/tumblr_inline_mjw0l2etMI1qz4rgp.gif What to ask? How to ask? Where to get that info?

4. 4 [Burnett et al., 2016]

5. 5 What to ask? Question may be a duplicate since list of existing questions appear

6. 6 How to ask? Unfamiliar with community expectations and feel like question is unfit

7. 7 Where to get that info? Don’t see other users with a similar new experience

8. 8 [Stack Overflow 2018 Developer Survey] Even though, about 50 million people visit Stack Overflow monthly…

9. 9 [Stack Overflow 2017 Developer Survey] …not everyone contributes. [Stack Overflow 2018 Developer Survey] 7.6% v.s. 88.6% ~87% have accounts—but 20% have never posted

10. Collaborative Editing Two core components: Collaborative question draft Help Room to discuss and edit the draft 10 2 1

11. Design Principles Provide formative and timely feedback Allow mistakes in private space Do not answer questions, help others ask better questions 11 [Epstein et al., 2002] [Baron 1993] [Wenger, 1998] [Kraut et al., 2012] [Kraut et al., 2012][Fedor et al., 1993] [Ford et al., 2016] [Kraut et al., 2012]

12. Just-In-Time Mentorship Study 12

13. Study Design Goals To observe and measure changes in question quality To understand feedback provided by mentors To improve experience for new question askers To learn how to scale these benefits to the whole community 13

14. How it works 14

15. Novice writes question

16. Novice writes question Novice selects to post their question

17. Novice writes question Novice selects to post their question Novice   selects to join   Help Room

18. Novice writes question Novice selects to post their question Novice   selects to join   Help Room

19. Novice receives feedback on question Novice writes question Novice edits question draft Novice selects to post their question Novice   selects to join   Help Room

20. Novice receives feedback on question Novice writes question Novice selects to post their question Novice   selects to join   Help Room

21. Novice receives feedback on question Novice writes question Novice selects to post their question Novice   selects to join   Help Room Petter Friber 2 days later… Mason 3 hours later… Jerem y The last message was posted 5 days ago. }; edit copy draft back to Stack Overﬂow Hey @KristinaLustig, thanks for coming. :) This is the Stack Overflow chat room we're using to help people improve their questions. This message is automated but everything else is real! Before we start: do you have any particular concerns, or are you just doublechecking before you post? Welcome @KristinaLustig I'm here to help you with the question, are you in chat? edited post draft. room mode changed to Private: only approved users may enter this room (this should only be used for moderation purposes) Wed 15:19 chat they will b leave send help | f

22. Novice receives feedback on question Novice writes question Novice edits question draft Novice selects to post their question Novice   selects to join   Help Room Novice posts question

23. 23 [Burnett et al., 2016]

24. 343 conversations Data Collected 24 33 day study 71,068 Eligible Novices 271 Novices 343 conversations Transcripts + Question Scores 63 Mentors Interview Mentors Survey Novices 4 Help Rooms 1 Private Mentor Room 520 Entered Room

25. Findings 25

26. Mentored questions have higher quality. MENTORED QUESTIONS HAD A 50% SCORE INCREASE 26 GOOD NEUTRAL BAD mentored 25% 49% 25% not mentored (control) 18% 51% 30% 6 -30 p <0.05 p <0.05

27. Mentors suggest high-fidelity improvements. Paraphrasing title and content (Question Phrasing) Formatting code to make question scannable Community triage Including all the pieces of a question (Question Framing) Community culture of asking 27

28. Mentors suggest high-fidelity improvements. Paraphrasing title and content (Question Phrasing) Formatting code to make question scannable Community triage Including all the pieces of a question (Question Framing) Community culture of asking 28

29. Community Triage 29

30. Community Triage 30 “ok the fact is that on SO you can’t ask for libs you would need to do that on another site [link] (with some [rules] that we can check if you like), instead if you like some code it would be really great if you tried something, do you have some code, do you have some post that you already checked?” (A30 -Mentor)

31. Community Triage 31

32. Community Triage 32 “Your question is off topic here....We’re not a hacking service.” (C72 -Mentor) "ok the fact is that on SO you can’t ask for libs you would need to do that on another site [link] (with some [rules] that we can check if you like), instead if you like some code it would be really great if you tried something, do you have some code, do you have some post that you already checked?” (A30 -Mentor)

33. Community Culture of Asking 33

34. Community Culture of Asking 34 “You can probably remove ‘Hello’ and ‘Problem’ from the top of the question. While it’s good to be social, it’s kind of just fluff on a Q&A site.” (D1 -Mentor)

35. Community Culture of Asking 35 “You can probably remove ‘Hello’ and ‘Problem’ from the top of the question. While it’s good to be social, it’s kind of just fluff on a Q&A site.” (D1 -Mentor) “You also might want to edit out the “Thank you!” at the end. I know it seems polite, but people object to it on Stack Overflow.” (D5 -Mentor) “Ok thanks for you help. I hope this time people won’t attack me.” (D62 -Novice)

36. Participants are satisfied with their interactions. 36 Likert Statement Median I feel that I am a part of the Stack Overflow community. 4 I feel more comfortable posting on Stack Overflow. 4 The help that I received from this program was useful to me. 5 I would recommend this program to other SO users. 5 I would like to participate in this program again. 5 NOVICES SATISFIEDWITH THEIR EXPERIENCE “If we can get the [original poster] through the first question with a positive experience and they can see how this site really works, then we should get more good questions which feeds in to having more good answers.” (M5 -Mentor) MENTORS INTERESTEDIN OFFERING GUIDANCE

37. Discussion + Implications 37

38. Having a human mentor surprised novices and made them feel welcomed Interpret dialogue for non-native English speaker Organic conversations to work through questions Advantages of humans helping humans 38 Source: https://www.businessinsider.com.au/why-the-iraq-war-cost-2-trillion-2013-3

39. Identity-Based Mentoring 39 Novice: Yeah, actually I am! I thought I picked up on your name too! Mentor: Hey! are you from Egypt by chance? Identity-Based Mentoring [Stack Overflow 2018 Developer Survey] Egypt Yossef

40. 40 Implications for community-based mentorship Asking community members about their interest Mentor many-to-many, one-to-many, and one-to-one Integrate mentorship functionality precisely when needed Source: http://humansofthearsenal.com/post/143654160037/im-taking-part-because-we-want-things-to-change

41. 41 Implications for community-based mentorship http://community.rstudio.com/ https://www.askquestions.tech/faq http://pythontutor.com/visualize.html

42. Online Q&A communities are widely used, but unfamiliar norms can leave novices discouraged from participating. Just-in-time mentoring provided novices with help to ask a well- received question, precisely when they needed it. Identifying novice-mentor interactions can define new onboarding and feedback experiences in other online communities. http://bit.ly/StackMentorPodcast http://bit.ly/StackMentorPaper dford3@ncsu.edu Summary

43. http://bit.ly/StackMentorPodcast http://bit.ly/StackMentorPaper dford3@ncsu.edu @DenaeFord @KristinaLustig @JeremyBanks @ChrisParnin Thank You! Online Q&A communities are widely used, but unfamiliar norms can leave novices discouraged from participating. Just-in-time mentoring provided novices with help to ask a well- received question, precisely when they needed it. Identifying novice-mentor interactions can define new onboarding and feedback experiences in other online communities.

44. ++ 44

45. Feature Implementation 45 Expanded existing Stack Overflow chat rooms Novices: eligible novice drafting a question Mentors: Experienced users

“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities

Du liest eine Vorschau.

Hier ansehen

“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities

Weitere Verwandte Inhalte

Ähnlich wie “We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities (20)

Weitere von Denae Ford (9)

Aktuellste (20)