1) The researchers explored how blind people experience automatically generated captions for social media images through contextual inquiries with 6 participants and an online experiment with 100 participants. 2) They found that blind people tended to trust captions even when they did not make sense or match the context of the image and post. 3) When captions were phrased to emphasize the possibility of being wrong through negative framing (e.g. "there is a 20% chance"), participants were less likely to fully trust incongruent captions and more likely to rely on their own expectations about the image.

1. Understanding Blind People’s Experiences
with Captions of Social Media Images
Haley MacLeod
CynthiaL. Bennett
Meredith Ringel Morris
Edward Cutrell
Indiana University
University of Washington
Microsoft Research
Microsoft Research
@haley_macleod
ww.haleymacleod.com

2. CaptionBot

4. “I think it's a young man jumping
in the air on a skateboard.”

6. “I think it's a man in a suit and tie
and he seems 😠😠”

8. “I am not really confident, but I think it's
a group of men standing next to a vase
and he seems 😐😐.”

10. These tools can empower
blind and visually impaired people
to know more about these images.

11. Typically, these systems
are evaluated by sighted people.

12. Standardized Metrics
Sighted
Human
Output
Machine
Output

13. User Studies
• Rate the quality of the caption for a given image
• Choose the best from a series of caption

14. The evaluationcriteria for
caption quality are different than
they would be if designed with
accessibility as the primary scenario.

15. We explore how blind people
experience automatically generated
captions in the context of social media.

16. Contextual Inquiry
Experiences with image captions
in the Twitter feed (n=6)
Online Experiment
The impact of caption phrasing
on trust and skepticism (n=100)

17. Study One:
Contextual
Inquiry

18. Observations

19. Observations
Interviews

20. Observations
Interviews
Image Probes

21. Observations
Interviews
Image Probes
More Interviews

22. • Taken from trending hashtags, popular accounts
• Range of topics (news, memes, photography,
cats, selfies, etc.)
• Computer generated captions as alt text
Tweet/Image Probes:

24. @WholeFoodsMarketReady for
#summer entertaining?Plan w/
special diets in mind. #glutenfree
#paleo #vegan

25. “I thinkit’s a plate of pasta and broccoli”
@WholeFoodsMarketReady for
#summer entertaining?Plan w/
special diets in mind. #glutenfree
#paleo #vegan

27. @HillaryClinton Some on
the other side may say our
best days are behind us.
Let's provethem wrong.

28. “I’m not really confident,but I think
it’s a man doing a trick on a skateboardat night.”
@HillaryClinton Some on
the other side may say our
best days are behind us.
Let's provethem wrong.

29. “I felt pretty confident about
them being accurate” (P2)

30. “I have to trust them because
I don’t have any form of reference […]
Unless somebody sighted were saying,
‘I don’t know why they said that,
that’s not what it is at all…’” (P1)

31. “I always double check.
Cause if I’m going to retweet something, I want
to know that I’m retweeting what I think it is…
I definitely wouldn’t retweet that without
somebody looking at it first.” (P3)

32. “I’m not really confident,but I think
it’s a man doing a trick on a skateboardat night.”
@HillaryClinton Some on
the other side may say our
best days are behind us.
Let's provethem wrong.

33. “I probably would have just retweeted it
thinking it was a photo of a skateboarder” (P3)

34. “If you say, there’s an older man on a skateboard
at night that would make more sense.
That’s the way I take ‘our best days are
behind us’ because I’m an old lady…” (P3)

35. @NewYorkTimes
Why isn’t Italy kinder
to gays?
I'm not really confident,but I thinkit's a person
holdinga tennis racquetand she seems neutral face.

36. “Okay, so it’s a tennis player.
Who, if I recognized tennis players,
maybe she’s gay.” (P2)

37. “Why is a tennis racket involved in this story?
The way pictures are eye catching to people,
it would engage a different side
of my perception. Saying, oh tennis racket.
I don’t play tennis, but maybe that’s
even more interesting than I thought.” (P5)

38. Contextual Inquiry
Experiences with image captions
in the Twitter feed (n=100)
Online Experiment
The impact of caption phrasing
on trust and skepticism (n=100)

39. Study Two:
Online
Experiment

40. Framing Effects
X% chance of dying
vs.
Y% chance of surviving

41. Framing Effects
X% chance of dying
vs.
a very small chance of dying

42. PositiveFraming Negative Framing
Natural
Language
Framing
“I'm really confident that's
a cat sitting on a counter
next to a window.”
“There's a small chance I
could be wrong, but I think
that's a cat sitting on a
counter next to a window.”
Numeric
Framing
“There's an 80% chance
that's a cat sitting on a
counter next to a
window.”
“There's a 20% chance I'm
wrong, but I think that's a
cat sitting on a counter next
to a window”

43. Background

44. Background
Understanding

45. Background
Understanding
Overall Trust

46. Background
Understanding
Overall Trust
Further Information

47. Background
Understanding
Overall Trust
Further Information
Optional Follow Up

48. Congruent Captions
vs.
Incongruent Captions

49. @ALensOnAmerica
Share your best wildlife
photo from a national
park and get feedback
from @chamiltonjames.
#YourShot.
“There's a 25% chance that's a bear is swimmingin the water.”

50. @SeattlePoliceDepartment
Bank robber arrested after
abandoningpants.
“There’s a 5% chance that’sa personon a surfboardin a skatepark.”

51. People found tweets with high congruence
to be more trustworthyand more helpful.

52. People who received negatively framed captions trusted
incongruent captions significantly less often.

53. Negatively phrased captions better
encouraged a lower level of reliance on
incongruent captions, and a higher level of
reliance on congruent captions.

54. Thus, we recommend that
automatically generated captions
be phrased in a way that reinforces
the possibility the caption might be wrong
(negative framing).

55. Negatively framed captions help
blind and visually impaired people
rely more on their intuition about a caption,
rather than unquestioningly trusting a caption
and making decisions based off
misinformation.

56. Contextual Inquiry
Experiences with image captions
in the Twitter feed (n=6)
Online Experiment
The impact of caption phrasing
on trust and skepticism (n=100)

57. • We observed that blind people are very trusting of
captions, even when they don’t make much sense.
• If we phrase captions in such a way that
reinforces the possibility the caption might
be wrong, people rely more on their
intuition and expectations of a caption.
• They create stories to resolve these differences,
rather than suspect that captions might be wrong.
Haley MacLeod
CynthiaL. Bennett
Meredith Ringel Morris
Edward Cutrell
www.haleymacleod.com
@haley_macleod