CAPTCHA is an effective and widely used solution for preventing computer programs (i.e., bots) from performing automated but often malicious actions, such as registering thousands of free email accounts or posting advertisement on Web blogs. To make CAPTCHAs robust to automatic character recognition techniques, the text in the tests are often distorted, blurred, and obscure. At the same time, those robust tests may prevent genuine users from telling the text easily and thus distribute the cost of crime prevention among all the users. Thus, we are facing a dilemma, that is, a CAPTCHA should be robust enough so that it cannot be broken by programs, but also needs to be easy enough so that users need not to repeatedly take tests because of wrong guesses. In this article, we attempt to resolve the dilemma by proposing a human computation game for quantifying the usability of CAPTCHAs. In our game, DevilTyper, players try to defeat as many devils as possible by solving CAPTCHAs, and player behavior in completing a CAPTCHA is recorded at the same time. Therefore, we can evaluate CAPTCHAs’ usability by analyzing collected player inputs. Since DevilTyper provides entertainment itself, we conduct a large-scale study for CAPTCHAs’ usability without the resource overhead required by traditional survey-based studies. In addition, we propose a consistent and reliable metric for assessing usability. Our evaluation results show that DevilTyper provides a fun and efficient platform for CAPTCHA designers to assess their CAPTCHA usability and thus improve CAPTCHA design.

1. Chien-Ju Ho1, Chen-Chi Wu2,
Kuan-Ta Chen1, Chin-Kuang Lai2
Presenter: Derec Wu1
1Institute of Information Science, Academia Sinica
2Department of Electrical Engineering, NationalTaiwan University

4.  Acronym for Computer Automated Public
Turing test to tell Computers and Humans
Apart
 Challenge-Response test
 Require users type letters or digits from a
distorted image to distinguish humans from
computers

5.  CAPTCHAs tests must be
 Secure
▪ Hard for computers
▪ Prevent computer programs from performing automated
malicious tasks
 Usable
▪ easy for human beings ?

6. Human Usability
v.s.
Computational Challenges

7.  Determine the difficulty of the CAPTCHA test
for human beings
 Traditional approach
 human survey
▪ cost a lot of money
▪ difficult to scale up

8.  A human computation game for CAPTCHA
usability evaluation
 Players are engaged to solve the problem for
us while having fun themselves
 Lower monetary cost and easier to scale up

9.  Overview
 CAPTCHA
 Why devilTyper
 DevilTyper Design
 Experiment
 Experiment setup
 Results
 Conclusion

10.  Each devil is attached with a CAPTCHA test
 Players are required to solve the test
correctly to win the game
 Player behaviors are recorded and are used to
evaluate the CAPTCHA usability

11.  Players must solve the CAPTCHA before the
devil from the top reaches the bottom
 Get scores by solving CAPTCHAs
 Lose HP if the devil reaches the bottom

12.  High score lists are maintained to encourage
players playing more

13.  http://deviltyper.iis.sinica.edu.tw/

14.  The following player behaviors for solving
each CAPTCHA test are collected
 Finish time
 Rate of typing error
 Rate of giving up the test
 Rate of repeated typing
 Rate of failing to solve the test within time limit

15.  Overview
 CAPTCHA
 Why devilTyper
 DevilTyper Design
 Experiment
 Experiment setup
 Results
 Conclusion

16.  We announced the game in a popular social
network PTT and held a four-week campaign
 Total cost: US$ 30
 Total number of games being played: 6,500
 Total CAPTCHAs being solved: 1,407,055

17.  CAPTCHATypes

18.  The results of different metrics are consistent
* A-F:
different types of CAPTCHAs
*The results are normalized to
0 to 1 for comparisons

19.  The DevilTyper results are consistent with
traditional survey method (MechanicalTurk)
* A-F:
different types of CAPTCHAs
*The results are normalized to
0 to 1 for comparisons
DevilTyper provides an open platform
for evaluating CAPTCHA usability

20. Design factors analysis using
DevilTyper

21. A:
B:
C:
D:
E:
F:
G:
PlainText

22.  Three strategies for text distortion in
CoolCAPTCHA
Character Distance X-AxisWave Y-AxisWave

23.  Three strategies for noise addition in
TgCAPTCHA
LongArcs Noise ShortArcs Noise Short Line Noise

24.  The difficulty of recognizing each character in
differentCAPTCHA types can be determined
“i” is hardly recognizable
in TgCAPTCHA
“i” is easier to recognize
in CoolCAPTCHA
Q V
C
T

25.  We proposed a human computation game,
Deviltyper, for evaluating CAPTCHA usability
 Monetary cost is much lower than traditional
surveys
 Evaluation is easier to scale up
 We show how this open platform can be used
to help the CAPTCHA designers to design
more user-friendlyCAPTCHAs

26. ThankYou 
http://deviltyper.iis.sinica.edu.tw/