The growing importance of APIs creates a need to support developers learning APIs with effective documentation. Prior research has generated important findings regarding information needs of developers and expectations they form towards API documentation. Several guidelines have been proposed on the basis of these findings, but evidence is lacking whether such guidelines actually lead to better documentation. This paper contributes the results of an empirical test that compared the performance of two groups of developers working on a set of pre-defined tasks with an API they were unfamiliar with. One group had access to documentation which was optimized following guidelines for API documentation design proposed in the literature whereas the other group used non-optimized documentation. Results show that developers working with optimized documentation made fewer errors on the test tasks and were faster in planning and executing the tasks. We conclude that the guidelines used in our study do have the intended effect and effectively support initial interactions with an API.

1. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 1
29/04/17 | Seite 1
Optimizing API documentation:
Some guidelines and effects
Michael Meng | Stephanie Steinhardt | Andreas Schubert | Merseburg University of Applied Sciences
API the Docs Virtual Series, November 4, 2020

2. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 2
29/04/17 | Seite 2
1 A research perspective on technical documentation
 Which information to present? And how to present it?
(Source:
https://helpx.adobe.com/framemaker/archive.html

3. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 3
29/04/17 | Seite 3
 How can we help technical communicators make informed
decisions?
(adapted from Boekelder & Steehouder, 1999, p. 67f.)
1 A research perspective on technical communication
Develop and
validate theories
and models
Study effect of design
variables on user behavior,
develop evidence-based
principles of information
design
Basic
research
Applied
research
Usability
testing
Examine usage of
specific
information
products to
identify trouble
spots

4. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 4
29/04/17 | Seite 4
2 API documentation: the problem
 Which information to present? And how to present it?
(Source: https://developers.shipcloud.io)

5. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 5
29/04/17 | Seite 5
2 API documentation: the problem
 Which information to present? And how to present it?
(Source: https://www.twilio.com/docs/sms/api/message-resource#create-a-message-resource)

6. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 6
29/04/17 | Seite 6
2 API documentation: the problem
 What makes learning an API hard?
(Source: Robillard & DeLine, 2011, p. 709)

7. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 7
29/04/17 | Seite 7
2 API documentation: the problem
 What makes learning an API hard?
(Source: Meng, Steinhardt & Schubert, 2018, p. 314)

8. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 8
29/04/17 | Seite 8
2 API documentation: the problem
(Source: https://stackoverflow.com/documentation)
 Meeting the information needs of developers is not a trivial question.

9. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 9
29/04/17 | Seite 9
 API documentation often misses information needs of developers;
poor learning resources are main obstacle for learning new APIs
 Research explored information needs and expectations of developers
and derived guidelines for API documentation (e.g. Meng et al., 2019,
Robillard & DeLine, 2011).
 Do such guidelines lead to better documentation?
2 API documentation: the problem

10. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 10
29/04/17 | Seite 10
3 Some findings from the research literature
 Developers adopt different strategies for learning and coding:
opportunistic vs. systematic approach
 Conceptual knowledge on the domain covered by the API facilitates
learning, but developers tend to ignore concepts documents
 Developers want to solve problems, not to learn an API as a whole.
Documentation is accessed in a task-oriented manner.
 Code examples used for learning and as a starting point for new
solutions.
 Mapping business objects to API elements and identifying entry points
into an API constitute main challenges.
(Clarke, 2007; Meng et al., 2018, 2019; Robillard & DeLine, 2011; Jeong et al., 2009)

11. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 11
29/04/17 | Seite 11
4 Guidelines for API documentation design
 Enable efficient access to relevant content
• Organize content by tasks & use cases
• Present important concepts integrated with relevant use cases
• Provide transparent and consistent navigation
• Structure content consistently
 Facilitate initial entry into the API
• Provide working code examples
• Provide relevant background knowledge
• Support mapping of business objects to API elements
• Provide a concise API overview
(Meng et al., 2019)

12. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 12
29/04/17 | Seite 12
4 Guidelines for API documentation design
 Support different learning strategies
• Enable selective access to code
• Signal text-to-code relations
• Present important concepts redundantly
• Enable fast and productive use of the API
(Meng et al., 2019, 2020)

13. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 13
29/04/17 | Seite 13
4 Guidelines for API documentation design
 Starting page: non-optimized version
(Meng et al., 2018, 2019)

14. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 14
29/04/17 | Seite 14
4 Guidelines for API documentation design
 API overview, task-oriented structure, selective access to code
(Meng et al., 2018, 2019)

15. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 15
29/04/17 | Seite 15
4 Guidelines for API documentation design
 Return shipments: non-optimized version

16. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 16
29/04/17 | Seite 16
4 Guidelines for API documentation design
 text-to-code relations, concepts integrated with use cases

17. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 17
29/04/17 | Seite 17
5 Testing the guidelines: research questions
 RQ1: Do the guidelines proposed in Meng et al. (2019) enable more
successful interactions with a new API?
 RQ2: Do the guidelines affect searching & processing information, or
planning and executing tasks, or both?
control (non-optimized) optimized

18. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 18
29/04/17 | Seite 18
5 Testing the guidelines: Method
• Test object: REST API for integrating shipping services into Web
shops (www.shipcloud.io)
• Five test tasks to be solved with documentation
• Two groups: group working with unmodified documentation
(control) vs. group working with optimized documentation
(optimized)
• Participants: 22 developers, 11 per group
• Data sources for analysis: screencasts (video+audio), eye
tracking, questionnaires before and after the test

19. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 19
29/04/17 | Seite 19
5 Testing the guidelines: Accuracy on tasks
 Higher accuracy on tasks for group working with optimized
documentation (p<.01)

20. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 20
29/04/17 | Seite 20
5 Testing the guidelines: Time on tasks
 Small (but nonsignificant) tendency to solve tasks faster for group
working with optimized documentation

21. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 21
29/04/17 | Seite 21
5 Testing the guidelines: Reading vs. acting
 Faster task execution for group with optimized documentation (p<.05);
small (but nonsignificant) tendency to spend more time in optimized
documentation

22. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 22
29/04/17 | Seite 22
6 Summary & Conclusions
 API documentation should meet the expectations and information
needs of software developers as discovered and described in past
research.
 Optimizing API documentation in response to findings from research is
worth the effort. It has a measurable positive effect on developer
performance.
Future (large-scale) empirical investigations should disentangle the
contribution of individual guidelines on the overall effect as well as
long-term effects and the impact of individual experience.

23. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 23
29/04/17 | Seite 23
6 Summary & Conclusions
 For full details see our ACM SIGDOC 20 conference paper – available
at https://doi.org/10.1145/3380851.3416759

24. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 24
29/04/17 | Seite 24
7 References
Boekelder, A., & Steehouder, M. (1999). Switching from instructions to equipment: the effect of graphic design. In H.
Zwaga, T. Boersema, & H. Hoonhout (Eds.), Visual information for everyday use: Design and research perspectives
(pp. 67-73). London, Philadelphia, PA: Taylor & Francis.
Clarke, S. (2007). What is an end user software engineer?. In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-
Zentrum für Informatik. Retrieved from http://drops.dagstuhl.de/opus/volltexte/2007/1080/
Jeong, S. Y., Xie, Y., Beaton, J., Myers, B. A., Stylos, J., Ehret, R., & Busse, D. K. (2009). Improving documentation for
eSOA APIs through user studies. In International Symposium on End User Development (pp. 86–105). Berlin,
Germany:Springer.
Meng, M., Steinhardt, S., & Schubert, A. (2018). Application Programming Interface Documentation: What do Software
Developers want? Journal of Technical Writing and Communication, 48(3), 295–330.
Meng, M., Steinhardt, S., & Schubert, A. (2019). How Developers Use API Documentation: An Observation Study.
Communication Design Quarterly, available online first, doi CDQ 10.1145/3274995.3274999.
Robillard, M. P. (2009). What Makes APIs Hard to Learn? Answers from Developers. Software, IEEE.
(November/December), 27-34.
Robillard, M. P., & DeLine, R. (2011). A field study of API learning obstacles. Empirical Software Engineering, 16(6), 703-
732.
St Amant, K., & Meloncon, L. (2016). Reflections on research: Examining practitioner perspectives on the state of research
in technical communication. Technical Communication, 63(4), 346-364.

25. The effect of visual signaling in screenshots: An eye tracking study
29/04/17 | Seite 25
29/04/17 | Seite 25
Thank you for your
attention
Michael Meng | Stephanie Steinhardt | Andreas Schubert | Merseburg University of Applied Sciences
API the Docs Virtual Series, November 4, 2020