Walking the Talk
•Download as PPTX, PDF•
0 likes•45 views
Three related coverage risks stood out when I begain to test a chatbot API for a medical symptom checker. With an infinite space of possible chats, how could we: • look for unintended consequences of changes as we built the API • discover some of the edge and corner cases bugs that would surely exist • exercise the API to any significant extent after each iteration To help mitigate these risks I built a client which would randomly walk through dialogs, unattended, and report on what it had found. In this talk, I'll describe how I implemented that client by iteratively adding functionality that I hoped would facilitate my exploration of changes and fixes to the emerging API. I'll give examples of features that worked well (such as configuration of probabilities for different types of answers) and those that did not (such as checking for specific classes of medical outcome), explain how I built on top of the client to make a load testing tool, and think about what I'd do differently next time.
Recommended
Recommended
More Related Content
Similar to Walking the Talk
Similar to Walking the Talk (20)
More from James Thomas
More from James Thomas (14)
Recently uploaded
Recently uploaded (20)
Walking the Talk
- 1. Walking the Talk: Random Exploration of a Chatbot API @hiccupps James Thomas Ministry of Testing, Norwich 2023-06-28
- 4. @hiccupps
- 5. Project: Extract a Basic Chatbot API @hiccupps
- 6. The Task Presentation-agnostic API. Integration with text-based clients. Turn-based medical assessment. Short deadline. @hiccupps
- 7. Risks, Questions, Test Ideas @hiccupps
- 8. Internal IDEs Medical case runners E2E tests MQEs Case permuting MKEs Unit and integration tests Exact match to report Expected questions asked Assessment without error Expected suggestions present Thresholds for various medical metrics Literature Existing behaviour Expertise Requirements Internal KB Vignettes Personas The “assessment space” is vast Data Tools Comparisons
- 10. Covering the Assessment Space Unintended consequences. Discover edges and corners. Exercise the API extensively. @hiccupps
- 11. The idea Run unattended medical assessments. Can be iterated and customised quickly. Identify places for deeper inspection. @hiccupps
- 12. Start “OK” Random Choose randomly Card type? More turns? Stop yes no text input choice The Simplest Thing That Could Possibly Work “Welcome to Ada.” “What is your name?” “Which option, A, B, or C?” @hiccupps
- 13. ✅❌ ✅❌ ✅❌ ✅❌ ✅❌ ❓ ❓ ❓ ❓ ❓ ❓ ❓ Navigate Explore Survey @hiccupps
- 17. Configuration Error checks ~1/200 turns Another symptom ~1/5 turns @hiccupps
- 20. Long Assessments
- 21. 500 Server Error ❌ ❌ Back end Back end Back end ❌ @hiccupps
- 24. LOGIC MODEL
- 26. What is a Model? A way of describing a system … … typically to aid understanding … … often to permit predictions about how the system will behave. The "system" need not be all of whatever you're looking at. @hiccupps
- 28. @hiccupps
- 29. @hiccupps
- 31. @hiccupps
- 33. @hiccupps
- 34. @hiccupps
- 35. @hiccupps
- 36. What Worked Well Verbose logs. Asserting generally and on fixes. Randomisation for unknowns. Configuration for directed exploration. Toolkit (replay, parallel, analysis, …). Question-driven development. Code changes and dependencies. Card identifiers. Medical testing. No explicit model. What Worked Well … and What was Challenging @hiccupps
- 37. GREAT SHOT KID, THAT WAS ONE IN A MILLION @hiccupps
- 38. YEAH! I TOOK A MILLION SHOTS @hiccupps
- 41. References Ada screenshots: https://www.uisources.com/explainer/ada-diagnosing-via-chat-bot Rope images: https://unsplash.com/photos/pqx4G7EPomY, https://unsplash.com/photos/y5XnFwUgr-k Wipotec: https://www.wipotec-ocs.com/en/product-inspection/ Microscope: https://londonlaboquip.com/product/microscope-binocular-biological-sc302 Messy lab: https://imgur.com/gallery/bQiK6 Dice: https://www.richardhughesjones.com/luck-randomness/dice-gif/ Altwalker: https://altom.gitlab.io/altwalker/altwalker/ Star Wars: https://www.starwars.com/video/one-in-a-million-shot @qahiccupps
Editor's Notes
- **Navigating** Error handling Consistency of API E.g. Male only assessments - compromise test code to get a walking skeleton; POST vs GET, two similar but slightly different schemas in the API itself. These activities make me ask questions… what if I …? How about when? Could it possibly be that …? Problems with the walker and with the product. While automating I’m testing. Don’t be too quick to restrict to what you think the system wants. Can I get from start to finish? What assumptions are required? What workarounds are required? How might developers struggle here? … ** Checking** E.g. certain kinds of dialog turns have different properties to assert on - keys in DTO must be present, or in some kind of relationship. What can I assert specifically and generally? Where are the edge cases? (e.g. by general global assertions failing) What are the error cases? How valuable is it to check these things here? ** Exploring** When you explore you don’t know if you’ll find anything, and if you find something you won’t know whether it’s relevant, and if it’s relevant you won’t know whether it’s important. Code is a tool and a toolkit. Extend it to the next question you have. (Can I get to “call ambulance” outcomes? What would need to happen to do that? How could I avoid it?) Log paths and outcomes. Analyse outside the code for patterns (seen and missing) Don’t error check too heavily. Catch and investigate failures. (Expose assumptions) Check some positive cases by hand. Check failures by hand. Look for patterns. (Don’t have to catch all failures; TODO outcomes are fine because then you can filter them in later analysis.) Explore the data you produce. Replay for repeatability. Configuration to guide the direction of exploration. (Initially, I’d just hack the code) How can I take the extreme choice each time? How can I make the longest assessment? Can I run an assessment for ever? A toolkit to gather data … … for human analysis. Failures are targets. Patterns are indicators. …
- Parallel nothing clever - just run two or more copies!