Crowd sourced application testing is a hot new way of testing! It is estimated that by 2018 20% of testing will be done using crowd sourcing. Instead of keeping testers in-house crowd sourcing gives you access to an unlimited number of testers, at any time and any place. It also allows you to test using a previously unattainable number of combinations of devices and software. Marc Fasel is the Technical Director of the Melbourne-based SaaS company PerformanceCentre and will present to us on how they leveraged Crowd Sourced Testing using the Rainforest QA platform within his organisation. Using real world case study examples Marc will explain how after a failed attempt to automate their regression test suite, they successfully managed to reduce the two week regression testing effort to less than a single day! Marc is amazed by the reliability and results being produced by their Crowd Sourced Testing, not to mention the great bugs being found. http://www.rainforestqa.com

1. Solution Proposal 1
Crowd Sourced Testing
March 22, 2017
Marc Fasel
Technical Director
Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform

2. Crowd SourcedTesting
PerformanceCentre
2
•PerformanceCentre is a SaaS platform
•We calculate commissions for sales people
•Clients:
•Companies with large sales teams: 50-5000
•Commissions cannot be accurately calculated using Excel
•Example Suncorp, Optus, Bankwest …
•Headquarters: South Melbourne, team of 17
•Cater to the Australian- and US- market
•I am the technical director, lead an agile team of 4 developers, 1 tester

3. Crowd SourcedTesting
Problem: Our Regression is not Agile
3
•Agile development: 2 week sprints, deploy after each sprint
•Lots of functionality built over the last 10 years
•Web application with > 300 pages
•Regression test
•Before every deployment
•Written Regression test: 300 test cases, average10 test steps
•Execution time: 2 person weeks
•Not agile!

4. Crowd SourcedTesting
Test Automation
4
•High initial development effort; estimate 2 hours per test; Every time we added a new feature
there was a new problem to overcome with Selenium
•High maintenance effort; many tests broken with each run
•Tests were flaky (timeouts etc)
•We could hire a single person only doing Selenium
•The tests never found anything (or it was drowned out by the false-positives)

5. Crowd SourcedTesting
The Search
5
•Add local test engineer
•Management Overhead
•Expensive
•Regression test time still too long: 1 week
•Outsourcing
•Recruit tester in the Philippines
•Testing is not on-demand: Gotta keep person busy 40hrs/wk
•Management overhead
•Regression test time still too long: 1 week

6. Crowd SourcedTesting
The Solution: Crowd-Sourced Testing
6
•Emerging trend in testing
•Gartner report “Market Guide for Crowdsourced ApplicationTesting Services” Oct 2016:
•By 2018 20% of testing will be done using crowdsourcing
•Testing-as-a-Service
•Instead of using in-house test engineers for QA the testing is outsourced to a company
•Crowd: Large number of testers, parallel execution of tests
•Typically use case: web-based or mobile testing
•Web-based SaaS platform to manage the testing
•Testers can be qualified or unqualified
•Either virtual machines or use their own devices

7. Crowd SourcedTesting
Solution: Crowd Testing
7
•We ended up using Rainforest QA SaaS platform
•Written tests are entered into platform
•Run now button: select tests you want to run
•Tests are executed by human testers
•Testers get a virtual machine spun up with the correct OS and browser
•Tests run in parallel
•Time to run all tests is determined by the time to run the longest test
•On-demand
•Regression test time:1 hour

8. Crowd SourcedTesting
PlatformView
8

9. Crowd SourcedTesting
Test Details
9
•Typical test consists of 5-10 steps
•Each step is an instruction followed by aYes/No question
•Any qualified tester can take on the test
•Tester getsVM with correct OS and browser, and the step description shown.
•Tester executes the step and answersYes/No question.

10. Crowd SourcedTesting
Individual Test
10

11. Crowd SourcedTesting
Test Step
11

12. Crowd SourcedTesting
Results
12

13. Crowd SourcedTesting
Individual Failure Result
13

14. Crowd SourcedTesting
Where do Testers come from?
14
•In our case they come from HIT market places
•Example:Amazon MechanicalTurk
•Human IntelligenceTasks (HIT): Micro-jobs that cannot be automated
•Marketplace: requestors and workers
•Platform is requestor, workers are the testers
•Each test becomes a HIT
•Workers have to earn qualifications and have reputation to be allowed to take test HITs
•Earn qualifications through training
•Earn reputation through successful HITs

15. Crowd SourcedTesting
Amazon Mechanical Turk
15

16. Crowd SourcedTesting
HIT Example
16

17. Crowd SourcedTesting
Cost
17
•Cost is per test step: US $0.66-$0.77 depending on pre-paid number of credits
•Example: US $0.70/ test step
•200Tests * 10 Steps * 2 browsers * US $0.70 = US $2800 per regression test
•2 regression tests per month = US $5600
•Cheaper than an localTest Engineer, more expensive than offshoreTester

18. Crowd SourcedTesting
Advantages
18
•Test results in an hour: super agile!
•Maintenance is a breeze:
•1 hour to add new tests per sprint
•5 tests broken per cycle, and it takes <30mins to fix
•0 management overhead
•Easy testing against different browsers, mobile devices
•Test crowd works 24/7
•Issues easy to reproduce
•They actually found real bugs!

19. Crowd SourcedTesting
Disadvantages
19
•Cost
•Monthly cost US$4000 for 2 regression tests @1 hr
•You could have 200 failed tests: cost US$2000
•Tester Skill
•Testers do not know anything about your application
•The testers are not particularly skilled in testing
•They know how to use a web app
•The tests have to be simple to execute
•Testers sign NDA: how do you enforce that?
•Tests run in parallel
•The tester look only for the answer of each test step

20. Crowd SourcedTesting
Platform Choice
20
•Popular vendor Options
•Rainforest QA (www.rainforestqa.com)
•Applause
•Testbirds
•Lionbridge
•Mob4Hire
•Space is still very young, and vendors have different approaches
•Exploratory testing vs. functional/ regression testing
•Testers from HIT marketplaces vs. fixed test teams
•Some vendors even provide test automation for you

21. Crowd SourcedTesting
Conclusion
21
•CrowdTesting has been a great success for us
•Best of both worlds of test automation and manual testing
•Quick results
•No management overhead
•Easy scalability to new browsers and devices
•Well-maintained test suite
•Few false-positives
•Main drawback: Cost
•Agile QA
•Release the day after the sprint ends
•Regression-tested hot fixes in 4 hours
•Retain Small team
•Very young and dynamic space, lots of changes still happening

22. Crowd SourcedTesting
Contact Us
22
Measure Sales - Reward Success
11Yarra Street, South MelbourneVIC
+61 3 8685 1200
info@performancecentre.c
linkedin.com/company/
www.performancecentre.c