This document summarizes Dr. Max Wilson's experience replicating a 2011 study by Shah and González-Ibáñez on collaborative information seeking using the Coagmento software. Some key challenges included using an updated version of the software, designing tasks that were still relevant, limited resources and timescales, and uncertainty around data processing and analysis methods. While the replication found different results, it provided valuable lessons about issues in replicating studies for both research and teaching purposes.

1. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Teaching HCI Methods: Replicating
a Study of Collaborative Search
Max L.Wilson
Evaluating the synergic effect of collaboration in information seeking.
In SIGIR 2011
by Shah & González-Ibáñez
Replicated Paper:
Wednesday, 8 May 13

2. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Background
• I was teaching HCI Methods in Swansea
• Replicating a study is “a good way to learn”
• I’d just finished teaching the Information Seeking module
- and I’m very interested in Collaborative Information Seeking
Wednesday, 8 May 13

3. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
http://coagmento.org
Wednesday, 8 May 13

4. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Chirag Shah
• Assistant Prof at Rutgers LIS
• Built Coagmento during his PhD
• Now Working with his PhD Student
•
Wednesday, 8 May 13

5. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Collaborative Information Seeking
Wednesday, 8 May 13

6. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study Conditions
relevant information, and using it to compose a report.
6. Participants filled out post-task questionnaires.
The researcher conducting the study communicated with the
participants through the chat-box at different times during the
study instructing them to start/stop the task or fill in a
questionnaire.
3.4 Conditions
To study the difference between individual information seeking
and CIS, as well as to understand how various CIS settings can
affect a collaborative team’s effectiveness in accomplishing an
information-seeking task, we conducted experiments with four
different conditions: single participants, two participants at the
same computer, two participants in the same room but different
computers, and two participants in different rooms with individual
computers.
In order to have a baseline to study the synergic effect of
collaboration, we artificially created pairs of users from C1 (single
users). We generated all possible combinations of pairs in groups
of 5, reaching a total of 49 groups and creating 245 artificial
teams in total. This was done in order to cover all possible pairs of
users while avoiding a given user appearing in more than one
team within the same group of teams.
These five conditions are summarized in Table 1. Setups for four
of these conditions are also depicted in Figure 3. Note that in the
real experiment, those in C5 condition were located in different
rooms separated by walls, and not just a partition. They could not
see or talk to each other directly, and the only communication
channel they had was the text-box provided with the system.
Table 1: Experimental conditions.
Cond. Description
C1 Single participants
C2 Artificial team
C3 Co-located using the same computer
C4 Co-located using different computers
C5 Remotely located
Your report on this topic should address the following issues:
description of how the oil spill took place, reactions by BP as well
as various government and other agencies, impact on economy
and life (people and animals) in the gulf, attempts to fix the
leaking well and to clean the waters, long-term implications and
lessons learned.”
The participants saw this description on the screen (phase 4 in the
study), and were also given a printed copy to refer to during their
session.
Figure 3: Experimental setups for four different conditions.
4. EVALUATION
In order to evaluate the effectiveness of the participants in various
conditions, we employed a number of traditional and non-
traditional evaluation measures, which are presented below. Here
we also describe other useful constructs and definitions that will
n order to have a baseline to study the synergic effect of
ollaboration, we artificially created pairs of users from C1 (single
sers). We generated all possible combinations of pairs in groups
f 5, reaching a total of 49 groups and creating 245 artificial
eams in total. This was done in order to cover all possible pairs of
sers while avoiding a given user appearing in more than one
eam within the same group of teams.
hese five conditions are summarized in Table 1. Setups for four
f these conditions are also depicted in Figure 3. Note that in the
eal experiment, those in C5 condition were located in different
ooms separated by walls, and not just a partition. They could not
ee or talk to each other directly, and the only communication
hannel they had was the text-box provided with the system.
Table 1: Experimental conditions.
Cond. Description
C1 Single participants
C2 Artificial team
C3 Co-located using the same computer
C4 Co-located using different computers
C5 Remotely located
.5 Task
We chose “gulf oil spill” as the topic for this experimentation
nce it was quite popular and relevant at the time the study was
eing conducted. Our preliminary investigations, including a few
ilot runs, indicated that there was a huge amount of material on
his topic, and that the participants would find it interesting and
hallenging enough as an exploratory search task. Each
articipant was given the following task description.
Figure 3: Experimental setups for four differen
4. EVALUATION
In order to evaluate the effectiveness of the particip
conditions, we employed a number of traditio
traditional evaluation measures, which are presente
we also describe other useful constructs and defini
later be used while reporting and discussing the resu
4.1 Universe of webpages
In order to compute quantities such as coverage,
universal set of webpages. Given that the search d
experiments was the open web, we needed a mor
that we could use to compare with. We decided to
of all the webpages visited by all of our participant
other words, the universe of webpages was defined
Wednesday, 8 May 13

7. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
What they found
• Remote collaborators were more independent (less overlap), and
more synergetic than random pairs
• Significant differences between conditions
• Across several measures
- page diversity
- page coverage
- relevance (precision, recall, F-measure)
- page usefulness
• No difference in NasaTLX
Wednesday, 8 May 13

8. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Why was this a good paper to Replicate?
• 1) Coagmento was a downloadable tool
• 2)The study is clearly reported in the paper
• 3) I know Chirag relatively well
• 4)The study used more than the basic metrics
Wednesday, 8 May 13

9. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Many Challenges
• 1) Software
• 2) Data capture
• 3)Task Design
• 4)Team Research Experience
• 5) Financial Support
• 6)Time Scales
• 7) Data Processing
• 8) Data Analysis
Wednesday, 8 May 13

10. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #1
• Coagmento had evolved/improved (after this study)
• BUT - Roberto offered to try to roll-back the software
-This created a small project delay (hard for teaching)
- But meant we were using more comparable software
• BUT - Sadly this process was only semi-successful
SoftwareVersions
Wednesday, 8 May 13

11. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #2
• The raw data was from the servers
• We began to consider recording the screens and manually
creating the logs
• BUT - Roberto offered to create a new server instance
• And provided us with a zip of the data at the end!
Data Capture
Wednesday, 8 May 13

12. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #3
• A leading newspaper has hired your team to create a comprehensive report
on the causes, effects, and consequences of the recent gulf oil spill.As a part of
your contract, you are required to collect all the relevant information from any
available online sources that you can find.
• To prepare ... [prompts to use features of the software]
• Your report on this topic should address the following issues: description of
how the oil spill took place, reactions by BP as well as various government and
other agencies, impact on economy and life (people and animals) in the gulf,
attempts to fix the leaking well and to clean the waters, long-term implications
and lessons learned.
Task Design
Wednesday, 8 May 13

13. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #3
• Temporally irrelevant
• Culturally less relevant?
- less intrinsic motivation for participants
• Should we create a new one, or use the original one?
Task Design
Wednesday, 8 May 13

14. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #3
• A leading newspaper has hired your team to create a comprehensive report
on the causes, effects, and consequences of the recent gulf oil spill.As a part of
your contract, you are required to collect all the relevant information from any
available online sources that you can find.
• To prepare ... [prompts to use features of the software]
• Your report on this topic should address the following issues: description of
how the oil spill took place, reactions by BP as well as various government and
other agencies, impact on economy and life (people and animals) in the gulf,
attempts to fix the leaking well and to clean the waters, long-term implications
and lessons learned.
Task Design
Wednesday, 8 May 13

15. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #3
• A leading newspaper has hired your team to create a comprehensive report
on the causes, effects, and consequences of the Olympic Games.As a part
of your contract, you are required to collect all the relevant information from
any available online sources that you can find.
• To prepare ... [prompts to use features of the software]
• Your report on this topic should address the following issues: Impact on
economy of host countries (people and animals), long-term
implications on the host country, conditions and voting
policy to become hosting nation and the next host country
and their preparations to host the games.
Task Design
Wednesday, 8 May 13

16. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #4
• 6 Novice MSc students, rather than 1 solid PhD student
- each taking several modules at the time
• Potential for high variance between students
• Tried to create anchors - like a fixed script etc
• Not clear how important the variance would be
ResearchTeam
Wednesday, 8 May 13

17. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #5
• Original participants received $15 (x70) each
• And prizes for the best collaborating teams
• We had no budget
- managed £50 of prizes
• Decided the prize for best team being most important to
replicate
Financial Incentives
Wednesday, 8 May 13

18. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #6
• Had to be within a fixed module (a semester)
• They had other modules to work on
• We had some time-slips in setting up the study
• We were only able to run 20 pairs, rather than 30 pairs
Time Limitations
Wednesday, 8 May 13

19. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #7
• This was very interesting
• For example - they removed search result pages
- this can never be an explicit known set
- especially as the task domain was different
• We followed their principles for data processing for analysis
• But we could not be sure we did this the same way
Data Processing
Wednesday, 8 May 13

20. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Challenge #8
• The exact stages of statistical analysis are not always clear
• For example - they used a modified NASATLX
- consequently, the exact analysis was not clear
- in particular, as to whether pair-wise comparisons were made
• Also, as the scales were likert, and the stats reported as ANOVA
- we weren’t sure if ANOVA on Ranks was used
- or a traditional ANOVA
• (also the novice students didnt know the difference)
Data Analysis
Wednesday, 8 May 13

21. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
The Outcome
• Is almost irrelevant - but we did not find the same results
• They found remote collaborators to be more independent, but
more synergetic than random pairs
• There were so many potential reasons though
- smaller sample size
- different task difficulty
- different software performance
- different financial incentives
- novice researchers
Wednesday, 8 May 13

22. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
RepliCHI discussion points
• 1) How should we handle different software versions?
• 2) Should we be using original tasks?
• 3) How to support data processing for future researchers?
• 4) Is there community value from replicating as teaching?
Wednesday, 8 May 13

23. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Many Challenges
• 1) Software
• 2) Data capture
• 3)Task Design
• 4)Team Research Experience
• 5) Financial Support
• 6)Time Scales
• 7) Data Processing
• 8) Data Analysis
Wednesday, 8 May 13

24. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Many Challenges
• 1) Software
• 2) Data capture
• 3)Task Design
• 4)Team Research Experience
• 5) Financial Support
• 6)Time Scales
• 7) Data Processing
• 8) Data Analysis
General Replication
Issues
Wednesday, 8 May 13

25. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Many Challenges
• 1) Software
• 2) Data capture
• 3)Task Design
• 4)Team Research Experience
• 5) Financial Support
• 6)Time Scales
• 7) Data Processing
• 8) Data Analysis
General Replication
Issues
Replication for
Teaching Issues
Wednesday, 8 May 13

26. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Many Challenges
• 1) Software
• 2) Data capture
• 3)Task Design
• 4)Team Research Experience
• 5) Financial Support
• 6)Time Scales
• 7) Data Processing
• 8) Data Analysis
General Replication
Issues
Replication for
Teaching Issues
Publishing Issues
Wednesday, 8 May 13