The availability of social tags has greatly enhanced access to information. Tag clouds have emerged as a new “social” way to find and visualize information, providing both one-click access to information and a snapshot of the “aboutness” of a tagged collection. A range of research projects explored and compared different tag artifacts for information access ranging from regular tag clouds to tag hierarchies. At the same time, there is a lack of user studies that compare the effectiveness of different types of tag-based browsing interfaces from the users point of view. This paper contributes to the research on tag-based information access by presenting a controlled user study that compared three types of tag-based interfaces on two recognized types of search tasks – lookup and exploratory search. Our results demonstrate that tag-based browsing interfaces significantly outperform traditional search interfaces in both performance and user satisfaction. At the same time, the differences between the two types of tag-based browsing interfaces explored in our study are not as clear.

1. Graz University of Technology
Evaluating Tag-Based Information Access
in Image Collections
Christoph Trattner*,
Yiling Lin, Denis Parra, Zhen Yue, Peter Brusilovsky
*Graz University of Technology, Austria
University of Pittsburgh, USA
Christoph Trattner Hypertext 2012
1

2. Graz University of Technology
Tagging Systems
“Tagging gained tremendously in popularity over the
past few years”
Christoph Trattner Hypertext 2012
2

3. Graz University of Technology
Tags
Christoph Trattner Hypertext 2012
3

4. Graz University of Technology
Tags
Christoph Trattner Hypertext 2012
4

5. Graz University of Technology
Tags
Christoph Trattner Hypertext 2012
5

6. Graz University of Technology
Problem Statement
 Not surprisingly, there was a lot of research in the past
few years that for instance investigated the value of
tags for efficient search and information retrieval in
online information systems
 Surprisingly, most of the studies only use information
retrieval or network-theoretic measures and ignore the
user side
 To contribute to this field of research we conducted a
controlled user study
1 2 3
Christoph Trattner Hypertext 2012
6

7. Graz University of Technology
What will be presented?
Are tags useful to be used in search interfaces?
Christoph Trattner Hypertext 2012
7

8. Graz University of Technology
Dataset
 ~ 2,000 images
 ~ 4,200 tags
 ~ 16,000 tag assignments
Interesting Fact:
Tags were generated by
~100 users from Amazon
Mechanical Turk
Christoph Trattner Hypertext 2012
8

9. Graz University of Technology
Interfaces
1 Baseline
2 Tag Cloud Search Interface
Faceted Tag Cloud Search Interface
3
Christoph Trattner Hypertext 2012
9

10. Graz University of Technology
How were the interfaces evaluated?
Christoph Trattner Hypertext 2012
10

11. Graz University of Technology
Evaluation
 Within-subject design, i.e. all of or subjects
evaluated all interfaces during the study.
 Interfaces were counter balanced
Baseline Tag Cloud Faceted Tag Cloud
1 2 3
Christoph Trattner Hypertext 2012
11

12. Graz University of Technology
Evaluation
 2 types of tasks:
 Look-up search task
 9 images with different difficulty level hard
200
Rank Pos
150 medium easy
 Exploratory search task 100
50
 3 tasks with different difficulty level 0
1
118
235
352
469
586
703
820
937
1054
1171
1288
1405
1522
1639
1756
1873
Sample Task:
“Find at least 8 different types of stores/shops in Pittsburgh! Each type of
store/shop should have at least two images from different locations, i.e.
in total you will have to find at least 16 images.”
Christoph Trattner Hypertext 2012
12

13. Graz University of Technology
Evaluation: Look-up Task
Look-up task
Christoph Trattner Hypertext 2012
13

14. Graz University of Technology
Evaluation: Exploratory Search Task
Exploratroy search
task
Christoph Trattner Hypertext 2012
14

15. Graz University of Technology
Evaluation
 All in all, 24 subjects
 Median age 31
 19 reported to be familiar with tagging systems
 All reported to be to use computers more than 5
hours a day
 All of them reported to be familiar with search
engines
 One session took 90 mins
Christoph Trattner Hypertext 2012
15

16. Graz University of Technology
What are the results?
Christoph Trattner Hypertext 2012
16

17. Graz University of Technology
Results: Performance (1/2)
Variables:
Question 1: What interface performs best? • Total Actions
• Search Time
1 2 3
 Look-up: no sign. differences between interfaces
 Exploratory:
Tag Cloud Interface out-performs baseline
Faceted Tag Cloud Interface almost as slow as baseline
Christoph Trattner Hypertext 2012
17

18. Graz University of Technology
Results: Performance (2/2)
Variables:
Question 2: What is the effect of familiarity and difficulty
• Total Actions
on the performance of the interfaces?
• Search Time
1 2 3
 On medium difficultly level…
Tag Cloud Interface out-performs baseline interface in search time
Christoph Trattner Hypertext 2012
18

19. Graz University of Technology
Results: Usage (1/2)
Question 3: How are the interfaces used?
=> Log analysis
Results:
 Search action and click image action used most often
 Add tag action sign. more used in facet
 Show more results sign. less used in tag cloud
Christoph Trattner Hypertext 2012
19

20. Graz University of Technology
Results: Usage
Question 4: Does tag grouping by semantic category affect the usage of
these categories?
50.00%
Baseline
45.00%
Tag Cloud
40.00%
Facet
35.00%
30.00%
25.00%
20.00%
Answer: Yes,
15.00%
We found sign. differences
10.00%
between the baseline and the
5.00%
faceted tag cloud interface
0.00%
who where when what other
Christoph Trattner Hypertext 2012
20

21. Graz University of Technology
Results: Participants„ perception of the
interfaces
Question 5: What was the perception of the particpants regarding the interfaces?
 Post-questionaires after each interface
Scale: 1=very bad….5=very good
Christoph Trattner Hypertext 2012
21

22. Graz University of Technology
Results: Preference and Rating
Question 6: What was the preference of the users?
• Post-questionair was handed out to the subjects with overall 7 questions.
Question 7: How are the interfaces rated?
Scale: 1 = very bad….5=very good
Christoph Trattner Hypertext 2012
22

23. Graz University of Technology
Questions?
 Why did people like the tag cloud interfaces more than the
baseline?
 Why was the tag cloud interface better rated than the
faceted tag cloud interface?
 Why did people recommend the faceted tag cloud interface
even if they rated the tag cloud interface higher?
Christoph Trattner Hypertext 2012
23

24. Graz University of Technology
Results: Comment Analysis (1/3)
 Why did people like the tag cloud interfaces?
“The tag cloud provided more information than
search only interface”
“I like tag cloud because it gives me new
ideas and it is easier to use”
Christoph Trattner Hypertext 2012
24

25. Graz University of Technology
Results: Comment Analysis (1/3)
 Why did people prefer the faceted tag cloud
interface?
“It is easy to find the tags that I needed in faceted
tag cloud”
“I like faceted tag cloud interface, because
the interface is clearer and I always know
where to find the tag”
Christoph Trattner Hypertext 2012
25

26. Graz University of Technology
Results: Comment Analysis (2/3)
 Why did people prefer the tag cloud interface over
the faceted tag cloud interface?
“The facet did not seem to identify tags well”
“I think the categorization was not good, it was
not relevant to the task”
?
Christoph Trattner Hypertext 2012
26

27. Graz University of Technology
Conclusions of this work
 In general, tags are useful in search interfaces
They help the user to find information faster
Less clicks, less search time
They give the users hints
They make the user happier
However, depending on the tag
interface design different results Take home message
can be observed…
Not always the most advanced
interface design is the best
choice…
Christoph Trattner Hypertext 2012
27

28. Graz University of Technology
End of Presentation
Thank you!
Christoph Trattner
ctrattner@iicm.edu
Graz University of Technology, Austria
Christoph Trattner Hypertext 2012
28