Presenter: Niklas Elmqvist BELIV 2010 Workshop http://www.beliv.org/beliv2010/

1. Mutually Linked StudiesBalancing Threats to Internal and Ecological Validity in InfoVis Evaluation Niklas ElmqvistSchool of Electrical & Computer EngineeringPurdue University

2. The Problem of Validity InfoVis evaluation often comes in two flavors: Rigorous + unrealistic Scientific rigor Toy dataset => Lacking in ecological validity Realistic + ad hoc Real dataset Imperfect experimental design => Lacking in internal validity 2

3. Example: The ColorLens 3 [McDonnel & Elmqvist 2009]

4. Example: Evaluating the ColorLens Realistic photographs: Ecologically valid (+) Learning effects (-) Cannot control images (-) Does not “feel” canonical! (-) Perlin Noise: Full control over dataset (+) Eliminate learning (+) Canonical and abstract (+) Not realistic! (-) 4 ?

5. Mutually Linked Studies Why choose one type of study? Do both! Idea: Mutually linked studies 1 x toy study – canonical and exhaustive 1 x realistic study – prove it works Complement each other Recycle much of the mechanics for both Experimental design Tasks Analysis Participants (!) 5 Reduces time and paper space requirements for conducting two studies (they are not separate)!

6. Questions? Acknowledgments: Color Lens co-authors(coming soon to a journal near you)Pierre DragicevicHM Jean-Daniel Fekete 6

7. Validity Primer Several different types of validity… I will discuss the following here: Internal validity Degree to which the outcome is a function of the controlled parameters of experiment Ecological validity Degree to which the results of the experiment can be applied to realistic situations 7