1.
How to do Mobile HCI Research in the large? Niels Henze University of Oldenburg Media Informa@cs and Mul@media Systems OFFIS -­‐ Ins@tute for Informa@on Technology Intelligent User Interface Group

2.
… but lets start with a ques@on: Who of you ever par@cipated in a user study?

3.
do you think that any of these guys ever did?

4.
Outline 1. Limita@ons of common studies 2. Into the large 3. Types of studies 4. What is so special? 5. What works for me 6. Wrap up

5.
User studies at MobileHCI 2010 20% acceptance rate 43 short+long papers

6.
User studies at MobileHCI 2010 20% acceptance rate 43 short+long papers subjects per paper hXp://nhenze.net/?p=810

7.
User studies at MobileHCI 2010 20% acceptance rate 43 short+long papers subjects per paper subject’s gender hXp://nhenze.net/?p=810

8.
all with a university degree, recruited in the Ins@tute community students or employees at our university User studies at recruited through flyers, posters and various mailing lists at the university MobileHCI 2010 20% acceptance rate 10 university students and 2 par@cipants 43 short+long papers are marke@ng professionals subjects per paper undergraduate or graduate students at subject’s gender the local university studying a variety of majors o]en a biased sample university students most subjects were students with a background in computer sciences most par@cipants were students studying or working in the University of Glasgow members in a joint research project hXp://nhenze.net/?p=810

9.
Some male students from the lab took part in our study... Small sample size isn’t necessarily an issue for a study Not every study needs a perfect sample of the popula@on Focussing on studies with few subjects prevents finding subtle differences We stew in our own juices if using our own students by default

10.
Some mo@va@on Large numbers are expensive in the lab – 1,000 subjects for an hour -­‐> 10,000€ – 1,000 subjects for an hour -­‐> 6 month – 1,000 subjects from around the world -­‐> impossible Different contexts are hard to address – We have no subway in Oldenburg – Don’t want to pay the flight for my par@cipant – And what are the relevant contexts anyway?

11.
Example of gehng large… Target selec@on on mobile phones thirty right-­‐handed subjects different target loca@ons and sizes [Park2008MobileHCI]

12.
Target selec@on on mobile phones thirty right-­‐handed subjects different target loca@ons and sizes Taps are skewed fixed posture single device Korean students vague results [Park2008MobileHCI]

13.
…same thing in the large game published on the Android Market

14.
…same thing in the large game published on the Android Market we inform the player about the study

15.
…same thing in the large game published on the Android Market we inform the player about the study just looks like an ordinary game

16.
…same thing in the large game published on the Android Market we inform the player about the study just looks like an ordinary game par@cipants get some introduc@on

17.
…same thing in the large game published on the Android Market we inform the player about the study just looks like an ordinary game par@cipants get some introduc@on

18.
…same thing in the large game published on the Android Market we inform the player about the study just looks like an ordinary game par@cipants get some introduc@on they tap the targets

19.
…same thing in the large game published on the Android Market we inform the player about the study just looks like an ordinary game par@cipants get some introduc@on they tap the targets we vary targets’ size and posi@on

20.
…same thing in the large game published on the Android Market we inform the player about the study just looks like an ordinary game par@cipants get some introduc@on they tap the targets we vary targets’ size and posi@on there is even a high score list

21.
published on the Android Market 100,000 installa@ons in three months 120 million touch events more than hundred different devices players from all over the world

22.
[Park2008MobileHCI]

23.
[Henze2011MobileHCI]

24.
Types of work Proof of concept – Showing that an idea/concept/product works – Lots of users, good ra@ngs, posi@ve comments, ... App stores as research tool – Experience report – Ethical and legal issues Inves@ga@ng app-­‐specific aspects – How a specific app is used – Compare different visualiza@ons Observing general aspects – Learn about how people and devices behave – How are apps how, how people touch the screen, ...

25.
Proof of concept

26.
Smule’s iPhone Ocarina music instrument for the iPhone million installa@ons [Wang2009NIME]

27.
Shapewriter developed gesture-­‐based keyboard + notepad qualita@ve feedback from App Store comments [Zhai2009CHI]

28.
App stores as research tool

29.
Into the wild with Hungry Yoshi loca@on based game for the iPhone 94,642 unique downloader inves@gated how to get subjec@ve feedback [McMillan2010Pervasive]

30.
100% 83,68% 81,31% 80% Experience from 60% 54,76% 5 Studies 40% compare amount of collected data 20% experience with collec@ng 7,32% qualita@ve data 0,46% discuss internal and 0% external validity [Henze2011IJMHCI]

31.
Inves@ga@ng app-­‐specific aspects

32.
Ra@ngs for Mobile Applica@ons compare amount of collected data experience with collec@ng qualita@ve data discuss internal and external validity [Girardello2010DSZ]

33.
Compare off-­‐screen visualisa@ons using repeated measures using a tutorial for a map applica@on [Henze2010MobileHCI] [Henze2010MobileHCI]

34.
Compare off-­‐screen visualisa@ons using repeated measures using a tutorial for a map applica@on and using a simple game (which worked much beXer!) [Henze2010MobileHCI] [Henze2010MobileHCI]

35.
Observing general aspects

36.
Falling Asleep with … appazaar [Böhmer2011MobileHCI]

37.
A Study of BaXery Life [Ferreira2011Pervasive]

38.
app stores as a inves@ga@ng app-­‐ inves@ga@ng proof of concept research tool specific aspects general aspects [Wang2009NIME] [McMillan2010RiL] [Girardello2010DSZ] [Hood2011IJTR] [McMillan2010Pervasive] [Zhai2009CHI] [Riccamboni2010IB] [Henze2011MobileHCIa] [Henze2011IJMHCI] [Gilbertson2008CiE] [Kuhn2010MM] [Henze2011MobileHCIb] [Miluzzo2010RiL] [Watzdorf2010LocWeb] [Poppinga2010OMUE] [Yan2011MobiSys] [Ferreira2011Pervasive] [Oliver2010HotPlanet] [Budde2010IoT] [Morrison2010RiL] [Buddharaju2010CHI] [Karpischek2011RiL] [Sahami2011CHI] [Henze2010MobileHCI] [Verkasalo2010MB] [Pielot2011ELV] [Henze2010NordiCHI] [Böhmer2011MobileHCI] [Cramer2010UbiComp] [Morrison2011CHI] [Henderson2009HotPlanet] [Norcie2011ELV] Ethics and legal issues

39.
but what is special about app store studies?

40.
App-­‐based vs. other studies Common con-­‐ Mining exis@ng App-­‐based trolled studies data studies Few par@cipants Many par@cipants Many par@cipants Ar@ficial context Natural context Natural context Defined tasks Defined task No tasks (if needed) Total control over Weak control over No control par@cipants par@cipants Heavily biased Biased to unbiased Unbiased sample sample sample

41.
You have to “sell” your study The study has a goal – Collect informa@on about specific behaviour – Performance for a specific task Users have to install the app on their own will – App needs a purpose – Good ra@ngs, high ranking Find a compromise – Maintain the goals of the study – AXract sufficient par@cipants

42.
Types of apps Applica@ons Games Widgets

43.
100.000 90.000 80.000 Par@cipants 70.000 How do we count the 60.000 number of par@cipant? 50.000 40.000 30.000 20.000 10.000 0 installa@ons opt-­‐in ac@ve users [McMillan2010Pervasive] [Morrison2010RiL]

44.
US Android users US popula@on 60% Par@cipants 50% How do we count the number of par@cipant? 40% A good sample of the 30% popula@on? 20% 10% 0% 18-­‐34 35-­‐44 45-­‐54 55-­‐64 65+ [Nielsen2011] [USCensusBureau2008]

45.
Collec@ng informa@on Objec@ve data – As early as possible [Henze2011IJMHCI] – More than just the task performance • All aspects that affect the results • E.g. device type, local, @me, screen size, resolu@on, ... • In par@cular: a version number – Compromise between permissions and data to collect

46.
Collec@ng informa@on Subjec@ve data – App Store comments can provide informa@on • but usually dont [Henze2011IJMHCI] • Might help to claim an app is great (e.g. [Zhai2009CHI]) • Ra@ngs without baseline are meaningles – Inves@gated how to get subjec@ve feedback [McMillan2010Pervasive] • In-­‐game “tasks” with dynamically loaded ques@ons • Integra@on with Facebook • Interviewed 10 people over VoIP for $25

47.
Realy stupid hope Stupid waste of @me!!! cailan FC the rabbit.... uninstalled Godimus Prime Ready for prime Its ok Stupid waste of @me. erika lance @me boring and dumb. Users don’t care if it’s a Beba research prototype Stupid and offincive to my pet rabbit bayleigh Logan 1 word...... dumb! josue 5 stars if there is a way to turn the music off. Doesnt go to well with slipknot Allen What the hell is this?? Boo! Luci Cullen Girl Examples from one of my games

48.
Ready for prime @me Users don’t care if it’s a research prototype Low quality results in low ra@ngs

49.
Ready for prime @me users don’t care if it’s a research prototype low quality results in low ra@ngs and few install installa@ons

50.
Ethical and legal issues “One should treat others as one would like others to treat oneself” [Flew1979Dic@onary] “Primum non nocere”/”First, do no harm” (Thomas Sydenham)

51.
6.96% 57.28% Informed consent Presenta@on highly affects the conversion rate 67.42% 87.57% [Pielot2011ELV]

52.
Informed consent Presenta@on highly affects the conversion rate Par@cipants arent aware what data is collected [Morrison2011CHI]

53.
Regula@ons Which rules to follow?

54.
“any informa+on rela+ng to an iden+fied or iden+fiable natural person” Regula@ons • Transparency: the persons whose data Which rules to follow? are being collected or accessed have the right to be informed when such data e.g. EU Data Protec@on processing is taking place. Direc@ve • Legi+mate purpose: data can only be collected for specific purposes • Propor+onality : data should be processed in a fashion that is not excessive beyond the purposes for which they were collected [Henderson2009HotPlanet]

55.
… or what works for me

56.
number of installa+ons 400 350 Games vs. Apps Thousands 300 our games are more 250 successful 200 150 100 50 0

57.
games 15,6% Games vs. Apps our games are more successful there are more apps than games apps 84,4% available in the Android Market hXp://www.androlib.com/appstatstype.aspx

58.
Games vs. Apps our games are more successful there are more apps than games players execute the strangest tasks

59.
Games vs. Apps our games are more successful there are more apps than games players execute the strangest tasks widgets and background services are perfect for longitudinal observa@ons

60.
Informing the user provide informa@on in the Market

61.
Informing the user provide informa@on in the Market show a modal dialog at the first start

62.
Informing the user provide informa@on in the Market show a modal dialog at the first start provide more informa@on and a link in an about page

63.
Publishing fancy screenshots and icon (that the first thing someone sees) @tle & descrip@on contain words users search for of course I don’t want to miss a single user prepare a dedicated webpage for each app

64.
Playing with the market weekly updates

65.
Playing with the market weekly updates rate your app as soon as it becomes available (and force everyone you know to do it as well)

66.
Keep it simple focused and specialized studies

67.
Keep it simple focused and specialized studies learning by doing

68.
Keep it simple focused and specialized studies learning by doing release early, o]en, and try it again if it doesn’t work

69.
Logging use hXp to transmit data

70.
Logging use hXp to transmit data store unaggregated measures

71.
Logging use hXp to transmit data store unaggregated measures consider limited resources in total: 392,401 files 27,331,383,646 bytes Examples from one of my games

72.
How to do Mobile HCI Research in Wrap up the large? ethnography, controlled Niels Henze experiments, observa@ons, … can all work in the large University of Oldenburg Media Informa@cs and collect data early, Mul@media Systems release o]en, be flexible OFFIS -­‐ Ins@tute for Informa@on respect ethics, Technology ser Interface Group Intelligent U consider regula@ons

73.
References [Wang2009NIME] Ge Wang: Designing Smule’s iPhone Ocarina. Proc. NIME, 2009. [Zhai2009CHI] Zhai, S., Kristensson, P.O., Gong, P., Greiner, M., Peng, S., Liu, L. Dunnigan, A., Shapewriter on the iPhone: from the laboratory to the real world. Adjunct Proc. CHI, 2009. [Gilbertson2008CiE] Paul Gilbertson, Paul Coulton, Fadi Chehimi, Tamas Vajk: Using Tilt as an Interface to control No BuXon 3-­‐D Mobile Games. ACM Computers in Entertainment, 2008. [Oliver2010HotPlanet] Earl Oliver. The Challenges in Large-­‐Scale Smartphone User Studies. Invited talk @ HotPlanet, 2010. [McMillan2010RiL] Donald McMillan: iPhone So]ware Distribu@on for Mass Par@cipa@on. Proc. Research in the Large Workshop @ UbiComp, 2010. [Miluzzo2010RiL] Emiliano Miluzzo, Nicholas D. Lane, Hong Lu, Andrew T. Campbell: Research in the App Store Era: Experiences from the CenceMe App Deployment on the iPhone. Proc. Research in the Large Workshop @ UbiComp, 2010. [Henze2011IJMHCI] Niels Henze, Mar@n Pielot, Benjamin Poppinga, Torben Schinke, Susanne Boll: My App is an Experiment: Experience from User Studies in Mobile App Stores, accepted by the Interna@onal Journal of Mobile Human Computer Interac@on (IJMHCI), 2011 [McMillan2010Pervasive] Donald McMillan, Alistair Morrison, Owain Brown, Malcolm Hall & MaXhew Chalmers: Further into the Wild: Running Worldwide Trials of Mobile Systems, Proc. Pervasive 2010. [Cramer2010UbiComp] HenrieXe Cramer, Mahas Rost, Nicolas Belloni, Didier Chincholle, Frank Bentley: Research in the Large. Using App Stores, Markets, and Other Wide Distribu@on Channels in Ubicomp Research. Adjunct Proc. Ubicomp, 2010. [Morrison2010RiL] Alistair Morrison, Stuart Reeves, Donald McMillan, MaXhew Chalmers: Experiences of Mass Par@cipa@on in Ubicomp Research, Proc. Research In The Large Workshop at Ubicomp, 2010. [Poppinga2010OMUE] Benjamin Poppinga, Mar@n Pielot, Niels Henze, Susanne Boll: Unsupervised User Observa@on in the App Store: Experiences with the Sensor-­‐based Evalua@on of a Mobile Pedestrian Naviga@on Applica@on. Proc. OMUE in conjunc@on with NordiCHI, 2010.

74.
References [Pielot2011ELV] Mar@n Pielot, Niels Henze, Susanne Boll: Experiments in App Stores – How to Ask Users for their Consent?, Proceedings of the CHI workshop on Ethics, logs & videotape, 2011. [Henderson2009HotPlanet] Tristan Henderson, Fehmi Ben Abdesslem: Scaling Measurement Experiments to Planet-­‐ Scale: Ethical, Regulatory and Cultural Considera@ons. Proc. HotPlanet, 2009. [Morrison2011CHI] Alistair Morrison, Owain Brown, Donald McMillan, MaXhew Chalmers: Informed Consent and Users Ahtudes to Logging in Large Scale Trials. Adjunct Proc. CHI, 2011. [Norcie2011ELV] Greg Norcie: Ethical and Prac@cal Considera@ons For Compensa@on of Crowdsourced Research Par@cipants, Proc. ETHICS, LOGS and VIDEOTAPE @ CHI, 2011. [Girardello2010DSZ] A. Girardello, F. Michahelles, Explicit and Implicit Ra@ngs for Mobile Applica@ons. In 3. Workshop “Digitale Soziale Netze” and der 40. Jahrestagung der Gesellsha] für Informa@k, September 2010, Leipzig. [Riccamboni2010IB] Rodolfo Riccamboni, Alessio Mereu, Chiara Boscarol: Keys to Nature: A test on the iPhone market. Tools for Iden@fying Biodiversity: Progress and Problems, 2010. [Kuhn2010MM] Michael Kuhn, Roger WaXenhofer, Samuel Welten: Social Audio Features for Advanced Music Retrieval Interfaces. Proc. MM, 2010. [Yan2011MobiSys]Bo Yan, Guanling Chen: AppJoy: Personalized Mobile Applica@on Discovery. Proc. MobiSys, 2011. [Budde2010IoT] Andreas Budde, Florian Michahelles: Product Empire -­‐ Serious play with barcodes. Proc. IoT, 2010. [Karpischek2011RiL] Stephan Karpischek, Geron Gilad, Florian Michahelles: Towards a BeXer Understanding of Mobile Shopping Assistants -­‐ A Large Scale Usage Analysis of a Mobile Bargain Finder Applica@on. Workshop on Research in the Large @ UbiComp, 2011. [Henze2010MobileHCI] Niels Henze, Susanne Boll: Push the Study to the App Store: Evalua@ng Off-­‐Screen Visualiza@ons for Maps in the Android Market, Proc. MobileHCI, 2010 [Henze2010NordiCHI] Niels Henze, Benjamin Poppinga, Susanne Boll: Experiments in the Wild: Public Evalua@on of Off-­‐ Screen Visualiza@ons in the Android Market, Proc. NordiCHI, 2010.

75.
References [Hood2011IJTR] Jeffrey Hood, Elizabeth Sall, Billy Charlton: A GPS-­‐based Bicycle Route Choice Model for San Francisco, California. Transporta@on LeXers: The Interna@onal Journal of Transporta@on Research, 2011 [Henze2011MobileHCIa] Niels Henze, Enrico Rukzio, Susanne Boll: 100,000,000 Taps: Analysis and Improvement of Touch Performance in the Large, Proceedings of MobileHCI, 2011 [Henze2011MobileHCIbNiels Henze, Susanne Boll: Release Your App on Sunday Eve: Finding the Best Time to Deploy Apps, Adjunct proceedings of MobileHCI, 2011 [Watzdorf2010LocWeb] Stephan von Watzdorf, Florian Michahelles: Accuracy of Posi@oning Data on Smartphones. Proc. LocWeb, 2010. [Ferreira2011Pervasive] Denzil Ferreira, Anind K. Dey, Vassilis Kostakos: Understanding Human-­‐Smartphone Concerns: A Study of BaXery Life. Proc. Pervasive, 2011. [Buddharaju2010CHI] Pradeep Buddharaju, Yuichi Fujiki, Ioannis Pavlidis, Ergun Akleman: A Novel Way to Conduct Human Studies and Do Some Good. Adcunct Proc. CHI, 2010. [Sahami2011CHI] Alireza Sahami, Michael Rohs, Robert Schleicher, Sven Kratz, Alexander Müller, Albrecht Schmidt: Real-­‐Time Nonverbal Opinion Sharing through Mobile Phones during Sports Events, Proc. CHI 2011. [Verkasalo2010MB] Hannu Verkasalo: Analysis of Smartphone User Behavior, Proc. Ninth Interna@onal Conference on Mobile Business, 2010. [Böhmer2011MobileHCI] MaXhias Böhmer, Brent Hecht, Johannes Schöning, Antonio Krüger, Gernot Bauer: Falling Asleep with Angry Birds, Facebook and Kindle – A Large Scale Study on Mobile Applica@on Usage. Proc. MobileHCI, 2011. [Agarwal2010HotNets] Sharad Agarwal, Ratul Mahajan, Alice Zheng, Victor Bahl: There’s an app for that, but it doesn’t work. Diagnosing Mobile Applica@ons in the Wild. Proc. Hotnets, 2010. [Morrison2010RiL] Alistair Morrison, MaXhew Chalmers: SGVis: Analysis of Mass Par@cipa@on Trial Data. Proc. Research In The Large Workshop at Ubicomp, 2010. [Lane2010CM] Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, Andrew T. Campbell: A Survey of Mobile Phone Sensing. IEEE Communica@ons Magazine, 2010.