Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Enabling Expert Critique with Chatbots and Micro-Guidance - Ci 2018

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Nächste SlideShare
Ux maturity
Ux maturity
Wird geladen in …3
×

Hier ansehen

1 von 20 Anzeige

Enabling Expert Critique with Chatbots and Micro-Guidance - Ci 2018

Herunterladen, um offline zu lesen

To enable at scale access to critique we present MATT, a chatbot that micro-guides experts to critique in short bursts of time with mediated communication to address experts' time and privacy concerns.

To enable at scale access to critique we present MATT, a chatbot that micro-guides experts to critique in short bursts of time with mediated communication to address experts' time and privacy concerns.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Ähnlich wie Enabling Expert Critique with Chatbots and Micro-Guidance - Ci 2018 (20)

Anzeige

Weitere von Carlos Toxtli (20)

Aktuellste (20)

Anzeige

Enabling Expert Critique with Chatbots and Micro-Guidance - Ci 2018

  1. 1. Enabling Expert Critique with Chatbots and Micro Guidance Carlos Toxtli, Joel Chan, Walter Lasecki, Saiph Savage
  2. 2. Problem statement ● Critique is important to improve creative work and help learners grow. ● Experts traditionally provided critique within physical studios. But, many never received critique as they were not co-located with experts. ● Online alternatives, such as forums, rarely lead people to receive critique ● Scaling online critique is difficult as: ○ People don’t have the knowledge or expertise to provide critique. ○ The experts with the knowledge to critique have limited time and privacy concerns. ● Dannels and Martin 2008 ● Luther and Bruckman 2008 ● Xu and Bailey 2012
  3. 3. Feedback typology [Dannels and Martin 2008] Feedback type Description Examples Reactive Emotional or visceral feedback that does not provide information on how to improve the work. “That’s wonderful!”, “Great work!” or “Horrible!” Direction Tries to bring the design more in line with her own expectations of what the solution should be. The feedback provides direction but no reasoning behind it. “I would have...” or “I wish...” Critique This type of feedback focuses on identifying decisions made in the creative work; relating that decision to an objective or best practice. “According to … the best way to … is ...”
  4. 4. MATT (Micro-Advicing Through Tutorials) To enable at scale access to critique we present MATT, a chatbot that micro- guides experts to critique in short bursts of time with mediated communication to address experts' time and privacy concerns.
  5. 5. 0- User starts conversation 2- Experts receive the learners work and give feedback 1- MATT deliver tutorials and asks for work 3- Learners receive the expert’s feedback
  6. 6. MATT components MATT consists of two main components: 1. Learner Helper module: Collects learners’ creative work, distributes the work to experts, and then shares experts’ feedback to learners. 2. Expert Micro-Guidance module: Orchestrates experts to volunteer quality micro-feedback – which resembles online critique – to help learners at scale.
  7. 7. Learner Helper module Main functions: ● Allow learners to easily submit their creative work ● Find relevant experts who can critique their work ● Present back to the learner the feedback from experts.
  8. 8. Expert Micro-Guidance module features ● Critique in Short Bursts of Time. MATT guides experts to provide critique to creative work by leveraging task decomposition from crowdsourcing. ● Critique Anywhere. MATT communicates via Facebook Messenger with experts. This design facilitates portability and on-the-go experiences.
  9. 9. ● Privacy. Our design builds upon privacy research that showcases that with anonymity higher quality feedback is produced. [Ruiling Lu and Linda Bol. 2007] ● Conversational. MATT guides experts to produce critique within a conversational setting through chatbots. The conversational aspect of MATT might also help experts to not feel that MATT’s guidance is too dictatorial. Expert Micro-Guidance module features
  10. 10. ● Research Question: Do chatbots micro-guiding experts enable a better approximation of the gold standard of studio design feedback? ● We conducted a field experiment to compare the feedback experts generated on MATT to alternative interfaces. Evaluation
  11. 11. Online forum with guidanceChatbot lacking micro-guidanceChatbot with micro-guidance, MATT
  12. 12. ● We recruited 548 learners, and 76 experts primarily using social media (Facebook, Linkedin). All experts volunteered their time. ● Learners submitted 153 creative work pieces to the MATT condition, 213 to the online forum, and 128 to the chatbot without guidance conditions. Participants
  13. 13. Learners submitted real world creative work pieces from one of these types: website design; poster design for an NGO; t-shirt design for an organization, and entrepreneur product design. Submitted work
  14. 14. Results Three coders classified each of the 548 messages into the category that represented the message the most (either critique, reactive, or direction).
  15. 15. Results - Quantitative analysis Feedback was significantly more likely to be classified as critique when it came from MATT, compared to feedback from the online forum condition (p < .01) or the Bot No Guidance condition (p < .01). MATT facilitated more critique production in experts.
  16. 16. Results - Qualitative analysis ● Experts stated they enjoyed moderately the chatbot interfaces (mean=4.85 for MATT and for the chatbot without guidance). ● Forum interface was also enjoyed, but slightly less (mean=4.77). ● Experts considered all interfaces to be moderately easy to use (mean=4.8) ● Experts felt that MATT addressed their privacy concerns (median=5).
  17. 17. Results - Opinions ● Some experts felt that MATT helped them to produce meaningful feedback by directing the communication into what mattered. ● Some experts expressed that the automated aspect of MATT made its guidance not feel imposing. ● MATT’s automation also seemed to help experts accept its guidance, as they felt that machines were made to help humans in their daily work.
  18. 18. Conclusion ● We introduced MATT a chatbot that guides experts to critique the creative work of learners at scale. ● MATT embodies the vision that chatbots facilitate orchestrating experts to critique while addressing experts’ privacy concerns and without creating an imposing environment on specialists. ● A field deployment provided evidence that MATT could guide experts to critique the creative work of hundreds of learners.
  19. 19. Thanks @ctoxtli carlos.toxtli@mail.wvu.edu http://www.carlostoxtli.com

Hinweis der Redaktion

  • Critique is important to improve creative work and help learners grow.
    Experts traditionally provided critique within physical studios. But, many never received critique as they were not co-located with experts.
    Online alternatives, such as forums, rarely lead people to receive critique
    Scaling online critique is difficult as:
    People don’t have the knowledge or expertise to provide critique.
    The experts with the knowledge to critique have limited time and privacy concerns.
  • Reactive: Emotional or visceral feedback that does not provide information on how to improve the work. Examples: “That’s wonderful!”, “Great work!” or “Horrible!”
    Direction: Tries to bring the design more in line with her own expectations of what the solution should be. The feedback provides direction but no reasoning behind it. Examples:, “I would have...” or “I wish...”
    Critique: This type of feedback focuses on identifying decisions made in the creative work; relating that decision to an objective or best practice; and then describing how and why the decision made supports or does not support the best practices [Luther et al. 2015].
  • To enable at scale access to critique we present MATT, a chatbot that micro-guides experts to critique in short bursts of time with mediated communication to address experts' time and privacy concerns.
  • MATT consists of two main components:
    Learner Helper module: Collects learners’ creative work, distributes the work to experts, and then shares experts’ feedback to learners.
    Expert Micro-Guidance module: Orchestrates experts to volunteer quality micro-feedback – which resembles online critique – to help learners at scale.
  • Main functions:
    Allow learners to easily submit their creative work
    Find relevant experts who can critique their work
    Present back to the learner the feedback from experts.
  • Critique in Short Bursts of Time. MATT guides experts to provide critique to creative work by leveraging task decomposition from crowdsourcing.
    Critique Anywhere. MATT communicates via Facebook Messenger with experts. This design facilitates portability and on-the-go experiences as experts can provide feedback wherever they use Facebook messenger.
  • Privacy. Our design builds upon privacy research that showcases that with anonymity higher quality feedback is produced. [Ruiling Lu and Linda Bol. 2007]
    Conversational. MATT guides experts to produce critique within a conversational setting through chatbots. The conversational aspect of MATT might also help experts to not feel that MATT’s guidance is too dictatorial.
  • Our evaluation asks:
    Do chatbots micro-guiding experts enable a better approximation of the gold standard of studio design feedback?
    We conducted a field experiment to compare the feedback experts generated on MATT to alternative interfaces.
  • We recruited participants that were divided into 3 conditions that studied different guidance and mediated communication settings:
    Chatbot with micro-guidance, MATT
    Chatbot lacking micro-guidance
    Online forum with guidance
  • We recruited 548 learners, and 76 experts primarily using social media (Facebook, Linkedin). All experts volunteered their time.
    Learners submitted 153 creative work pieces to the MATT condition, 213 to the online forum, and 128 to the chatbot without guidance conditions.
  • Learners submitted real world creative work pieces that were from one of these types: website design; poster design for an NGO; t-shirt design for an organization, and an entrepreneur product design.
  • Three coders classified each of the 548 messages into the category that represented the message the most (either critique, reactive, or direction).
  • Feedback was significantly more likely to be classified as critique when it came from MATT, compared to feedback from the online forum condition (p < .01) or the Bot No Guidance condition (p < .01).
    The overall model was statistically significant.
  • Experts stated they enjoyed moderately the chatbot interfaces (mean=4.85 for MATT and for the chatbot without guidance).
    Forum interface was also enjoyed, but slightly less (mean=4.77).
    Experts considered all interfaces to be moderately easy to use (mean=4.8)
    Experts felt that MATT addressed their privacy concerns (median=5).
  • Some experts felt that MATT helped them to produce meaningful feedback by directing the communication into what mattered: “...Chatbots can direct communication efficiently which you don’t really get with other technology [...]”
    Some experts expressed that the automated aspect of MATT made its guidance not feel imposing because there was nothing personal about it. It was “just” a machine: “Machines don’t have feeling at all, so also nothing to feel on my side.”
    MATT’s automation also seemed to help experts accept its guidance, as they felt that machines were made to help humans in their daily work.
  • We introduced MATT a chatbot that guides experts to critique the creative work of learners at scale. MATT embodies the vision that chatbots facilitate orchestrating experts to critique while addressing experts’ privacy concerns and without creating an imposing environment on specialists.A field deployment provided evidence that MATT could guide experts to critique the creative work of hundreds of learners.
  • Thanks

×