1. 2010 CRC PhD Student Conference
Supporting multimodal media recommendation
and annotation using social network analysis
Adam Rae
a.rae@open.ac.uk
Supervisors Stefan Rüger, Suzanne Little, Roelof van Zwol
Department The Knowledge Media Institute
Status Full Time
Probation Viva After
Starting Date October 2007
Research Hypothesis
By analysing and extracting information from the social graphs de-
scribed by both explicit and implicit user interactions, like those
found in online media sharing systems like Flickr1 , it is possible
to augment existing non-social aware recommender systems and
thereby significantly improve their performance.
Large scale web based systems for sharing media continue to tackle the problem
of helping their users find what they are looking for in a timely manner. To do
this, lots of good quality metadata is required to sift through the data collection
to pick out exactly those documents that match the information need of the
user. In the case of finding images from the online photo sharing website Flickr,
this could be from over 4 billion examples. How can we help both the system
and the user in enriching the metadata of the media within the collection in
order to improve the experience for the user and to reduce the burden on the
underlying data handling system? Can modelling users, by themselves and
within the context of the wider online community help? Can this modeling be
used to improve recommender systems that improve the experience and reduce
cognitive burden on users?
Existing approaches tend to treat multimedia in the same way they have
dealt with text documents in the past, specifically by treating the textual meta-
data associated with an image as a text document, but this ignores the inherently
different nature of the data the system is handling. Images are visual data, and
while they can be described well by textual metadata, they cannot be described
completely by it. Also, the user cannot be ignored in the retrieval process, and
learning more about a user provides information to the system to tailor results to
their specific requirements. Users interact online, and these interactions form a
1 http://www.flickr.com/
Page 91 of 125

2. 2010 CRC PhD Student Conference
new type of data that has yet to be fully explored nor exploited when modelling
users.
The work presented here combines the mining of social graphs that occur in
Flickr with visual content and metadata analysis to provide better personalised
photo recommender mechanisms and the following experiment and its analysis
are a major component in my overall thesis.
Interaction Scenario
In order to address this research question, multiple experiments have been car-
ried out, one of which I present here:
Envisage an incoming stream of photos made available to a user. In
systems of a scale similar to Flickr, this could be thousands of im-
ages per second. Can a system that uses cues from the social, visual
and semantic aspects of these images perform better than one that
uses the more traditional approach of using only semantic informa-
tion, according to specifically defined objective metrics? How does
performance vary between users?
An experiment was carried out that mines data from the social communities in
Flickr, from the visual content of images and from the text based metadata and
uses a machines learning mechanism to merge these signals together to form a
classifier that, given a candidate image and prospective viewing user, decides
whether the user would label that image as a ‘Favourite’2 - see Figure 1.
Related Work
The significant influence that our peers can have on our behaviour online has
been studied by researchers such as Lerman and Jones[3], and the particular
interaction that occurs between users and visual media in particular in the
work of Nov et al.[4]and Kern et al[2]. Their insights into the importance of
understanding more about a user in order to best fulfil their information need
supports the hypothesis that this kind of information can be usefully exploited
to improve systems that try to match that need to a data set supported by social
interaction. Here I extend their ideas by incorporating this valuable social data
into a complementary multimodal framework that takes advantage of multiple
types of data.
The use of social interaction features in the work of Sigurbjörnsson and van
Zwol[7] and Garg and Weber[1] inspired my more comprehensive feature set
and its analysis. Their findings that aggregating data generated from online
communities is valuable when suggesting tags is important and I believe also
transfers to recommendation in general as well as to the specific task of recom-
mending images. In fact, I demonstrated this in previous work on social media
tag suggestion[6].
I use some of the human perception based visual features outlined in the
work of San Pedro and Siersdorfer[5], as these have been shown to work well
in similar experimental scenarios and cover a range of visual classes. I extend
them further with a selection of other high performing visual features.
2A binary label Flickr users can use to annotate an image they like.
Page 92 of 125

3. 2010 CRC PhD Student Conference
Incoming stream of previously unseen candidate images
Textual Social Visual
User information User information
Feature Extraction
User A User B
Has tagged beaches before Member of urban animals group
Trained Classifier
Potential Favourite Images
Potential Favourite Images for User B
for User A
Figure 1: Diagram of the image classification system used with Flickr data.
Experimental Work
400 users of varying levels of social activity were selected from Flickr and their
‘Favourite’ labelled images collected. This resulted in a collection of hundreds
of thousands of images. To train my classifier, these images were treated as
positive examples of relevant images. I generated a variety of negative example
sets to reflect realistic system scenarios. For all photo examples we extracted
visual and semantic features, and social features that described the user, the
owner of the photo, any connection between them as well as other behaviour
metrics. We then tested our classifier using previously unseen examples and
measured the performance of the system with a particular emphasis on the
information retrieval metric of precision at 5 and 10 to reflect our envisaged use
case scenario.
Results
An extract of the results from the experiment are shown in Table 1. They can
be summarised thus:
• It is possible to achieve high levels of precision in selecting our positive
examples, especially by using social features. This performance increase
is statistically significantly higher than the baseline Textual run. These
social signals evidently play a significant rôle when a user labels an image
a ‘Favourite’ and can be usefully exploited to help them.
• The value of individual types of features is complex, but complementary.
The combined systems tend to perform better than the individual ones.
• It is far easier to classifier photos that are not ‘Favourites’ than those that
are, as shown by the high negative values. This can be used to narrow
down the search space for relevant images by removing those that are
obviously not going to interest the user, thus reduing load on both the
user and the system.
Page 93 of 125

4. 2010 CRC PhD Student Conference
System Accuracy + Prec. + Rec. - Prec. - Rec.
Textual 0.87 0.48 0.18 0.88 0.97
Visual 0.88 1.00 0.09 0.88 1.00
Social 0.92 0.80 0.56 0.94 0.98
Textual+Visual 0.88 0.62 0.27 0.90 0.97
Textual+Social 0.92 0.77 0.62 0.94 0.97
Visual+Social 0.93 0.89 0.56 0.94 0.99
Text+Vis.+Soc. 0.93 0.84 0.62 0.94 0.98
Table 1: Accuracy, precison and recall for various combinations of features using
the experiments most realistic scenario data set. Photos labelled as ‘Favourites’
are positive examples, and those that are not are negative examples. Higher
numbers are better.
• As is typical in this style of information retrieval experiment, we can trade-
off between precision and recall depending on our requirements. As we are
interested in high precision in this particular experiment, we see that the
combination of the Visual+Social and Text+Visual+Social runs
give good precision without sacrificing too much recall.
References
[1] Nikhil Garg and Ingmar Weber. Personalized, interactive tag recommenda-
tion for flickr. In Proceedings of the 2008 ACM Conference on Recommender
Systems, pages 67–74, Lausanne, Switzerland, October 2008. ACM.
[2] R. Kern, M. Granitzer, and V. Pammer. Extending folksonomies for image
tagging. In Workshop on Image Analysis for Multimedia Interactive Services,
2008, pages 126–129, May 2008.
[3] Kristina Lerman and Laurie Jones. Social browsing on flickr. In Proceedings
of ICWSM, December 2007.
[4] Oded Nov, Mor Naaman, and Chen Ye. What drives content tagging: the
case of photos on flickr. In Proceeding of the twenty-sixth annual SIGCHI
conference on Human factors in computing systems, pages 1097–1100, Flo-
rence, Italy, 2008. ACM.
[5] Jose San Pedro and Stefan Siersdorfer. Ranking and classifying attractive-
ness of photos in folksonomies. In WWW, Madrid, Spain, April 2009.
[6] Adam Rae, Roelof van Zwol, and Börkur Sigurbjörnsson. Improving tag
recommendation using social networks. In 9th International conference on
Adaptivity, Personalization and Fusion of Heterogeneous Information, April
2010.
[7] Roelof van Zwol. Flickr: Who is looking? In IEEE/WIC/ACM Inter-
national Conference on Web Intelligence, pages 184–190, Washington, DC,
USA, 2007. IEEE Computer Society.
Page 94 of 125

5. 2010 CRC PhD Student Conference
The effect of Feedback on the Motivation of Software
Engineers
Rien Sach
r.j.sach@open.ac.uk
Supervisors Helen Sharp
Marian Petre
Department/Institute Computing
Status Fulltime
Probation viva After
Starting date October 2009
Motivation is reported as having an effect on crucial aspects of software engineering
such as productivity (Procaccino and Verner 2005), software quality (Boehm 1981),
and a project’s overall success (Frangos 1997). Feedback is a key factor in the most
commonly used theory in reports published on the motivation of software engineers
(Hall et al. 2009), and it is important that we gain a greater understanding of the effect
it has on the motivation of software engineers.
My research is grounded in the question “What are the effects of feedback on the
motivation of software engineers?”, and focuses on feedback conveyed in human
interactions. I believe that before I can focus my question further I will need to begin
some preliminary work to identify how feedback occurs, what types of feedback
occur, and the possible impact of this feedback.
Motivation can be understood in different ways. For example, as a manager you might
consider motivation as something you must maintain in your employees to ensure
they complete work for you as quickly as possible. As an employee you might
consider motivation as the drive that keeps you focused on a task, or it might simply
be what pushes you to get up in the morning and go to work.
Herzberg (1987) describes motivation as “a function of growth from getting intrinsic
rewards out of interesting and challenging work”. That’s quite a nice definition; and
according to Herzberg motivation is intrinsic to one’s self. Ryan and Deci (2000)
describe intrinsic motivation as “the doing of activity for its inherent satisfaction
rather than for some separable consequence” (Page 60).
Herzberg (1987) defines extrinsic factors as movement and distinguishes it from
motivation, stating that “Movement is a function of fear of punishment or failure to
get extrinsic rewards”. Ryan and Deci (2000) state that “Extrinsic motivation is a
construct that pertains whenever an activity is done in order to attain some separable
outcome”.
There are 8 core motivational theories (Hall et al. 2009) and some of the theories
focus on motivation as a “a sequence or process of related activities” (Hall et al. 2009)
called process theories, while others focus on motivation “at a single point in time”
(Couger and Zawacki 1980) called content theories.
Page 95 of 125

6. 2010 CRC PhD Student Conference
As reported in a systematic literature review conducted by Beecham et al (2007), and
their published review of the use of theory inside this review in 2009 (Hall et al 2009),
the three most popular theories used in studies of motivation in Software Engineering
were Hackman and Oldman’s Job Characteristics Theory (68%), Herzberg’s
Motivational Hygiene Theory (41%), and Maslow’s Theory of Needs (21%)1.
Hackman and Oldman’s Job Characteristics Theory focuses on the physical job, and
suggests five characteristics (skill variety, task identity, task significance, autonomy,
and feedback) that lead to three psychological states which in turn lead to higher
internal motivation and higher quality work. Herzberg’s Hygiene Theory suggests that
the only true motivation is intrinsic motivation, and this leads to job satisfaction,
where extrinsic factors are only useful in avoiding job dissatisfaction.
One of the five key job characteristics in Hackman and Oldman’s theory is feedback.
Feedback is not explicitly mentioned in Herzberg’s Motivational Hygiene Theory, but
he notes that it is a part of job enrichment, which he states is “key to designing work
that motivates employees” (Herzberg 1987). However this is a managerial view point.
Software Engineers are considered to be current practitioners working on active
software projects within the industry. This includes programmers, analysts, testers,
and designers who actively work and produce software for real projects in the real
world.
From a management perspective, gaining a greater understanding of what motives
employees could prove invaluable in increasing productivity and software quality, and
from an individual perspective the prospect of being given feedback that motivates
you and makes your job more enjoyable and improves the quality of your work
experience could lead to a more successful and enjoyable work life.
My proposed research is divided into stages. In the first stage I plan to conduct
interviews and diary studies to identify the types of feedback in software engineering
and how feedback is experienced by software engineers. I then plan to conduct
additional studies to identify what impact this feedback has on software engineers and
how that impact is evident. Finally, I plan to observe software engineers at work to
see feedback in context, and to compare those observations to the information
gathered during the first two stages.
At the end of my PhD I hope to accomplish research that leads to a greater
understanding of what feedback is inside software engineering and how it is given or
received. Subsequently I wish to gain an understanding of how this feedback alters the
motivation of software engineers and how this manifests as something such as
behaviour, productivity or attitude.
1
The percentages are a representative of how many of 92 papers the theories were found to be
explicitly used in. There can be multiple theories used in any one paper, and the 92 papers were part of
a systematic literature review conducted by Hall et al (2007) sampling over 500 players.
Page 96 of 125

7. 2010 CRC PhD Student Conference
References
B.W. Boehm, Software Engineering Economics, Prentice-Hall, 1981.
COUGER, J. D. AND ZAWACKI, R. A. 1980. Motivating and Managing Computer Personnel.
John Wiley & Sons.
S.A. Frangos, “Motivated Humans for Reliable Software Products,” Microprocessors and
Microsystems, vol. 21, no. 10, 1997, pp. 605–610.
Frederick Herzberg, One More Time: How Do You Motivate Employees? (Harvard Business
School Press, 1987).
J. Procaccino and J.M. Verner, “What Do Software Practitioners Really Think about Project
Success: An Exploratory Study,” J. Systems and Software, vol. 78, no. 2, 2005, pp. 194–203.
Richard M. Ryan and Edward L. Deci, “Intrinsic and Extrinsic Motivations: Classic Definitions
and New Directions,” Contemporary Educational Psychology 25, no. 1 (January 2000): 54-67.
Tracy Hall et al., “A systematic review of theory use in studies investigating the motivations of
software engineers,” ACM Trans. Softw. Eng. Methodol. 18, no. 3 (2009): 1-29.
Sarah Beecham et al., “Motivation in Software Engineering: A systematic literature review,”
Information and Software Technology 50, no. 9-10 (August 2008): 860-878.
Page 97 of 125