Kristian Brimble and Sam Kavanagh, dev and test leads on the JustEat chatbot, take us through the automated testing strategy they used

1. Robots Testing Robots
@LDNBotFramework
❖ Kristian Brimble / Sam Kavanagh
❖ Bot Team: Dev Lead / Lead QA
❖ @kristianbrimble / @justeat_tech

2. Just Eat Help Chatbot

3. What does it do?
3
● Provides customers with support
when they have problems with
their orders
● Can compensate customers with
account credit when something
was missing from their order
● Hands over to a human when it
gets stuck

4. How does it work?
Microsoft
Directline
Native App Bot Framework LUIS
Junior Flows
Principal Flows

5. Testing our bot

6. When, where and how?
Unit Tests
Run on developer machines on demand (NCrunch FTW) and on every check-in & build
Integration Tests
Can be run on developer machines. Mainly run as part of our automated deployment process
System Tests
Can be run on developer machines but only targeting staging environment. Automatically run
when a build is deployed to staging
UI Tests
Run on app developer machines and as part of their CI pipeline

7. Unit Testing
● Originally started unit tests only on services called as part of a conversation
● Dialogs are hard to test due to difficulty to mock the framework and large amounts of setup
required (very brittle)
● Separated “conversation flow” from dialogs
● Conversation flow classes are very self-contained and do not reference any Bot Framework
features

8. Integration Testing
● Calls to Just Eat dependencies are proxied by a service that returns pre-recorded responses
● We forked the Bot Framework emulator so that it could be used by automated tests
○ No more dependency on the Bot Framework being contactable!
○ Reduced time to run our test suite by a factor of 10
● We wrote a small BDD-style framework to make testing a conversational interface easier (more
on that later)

9. System Testing
● Still uses our hacked emulator and BDD-style framework
● No dependencies mocked - “real data” created on our staging environment
● Smaller number of tests that exercise as much of a conversation topic as possible in one test

10. UI Testing
● Automated iOS and Android UI tests
● Written in each platform’s native language (Swift and Java)
● Run when the native bot client’s change
● All calls to Direct Line are mocked so that only UI functionality is tested

11. Example of a system test

12. Any questions?

13. Thanks to:
@LDNBotFramework
❖ Just Eat
❖ Jamie Dalton @daltskin
[Handing off to a human]
❖ Kristin Brimble @kristianbrimble
❖ Sam Kavanagh @justeat_tech
[Robots Testing Robots]
❖ Robin Osborne @rposbo
[Speaker Recognition API]