This document summarizes the findings from interviews with 5 developers about their practices for involving users during software evolution. It finds that user feedback comes from various scattered sources and is not systematically collected. Developers analyze feedback manually to prioritize tasks, but face challenges from the large quantity and quality of natural language feedback. They require tools to better consolidate, structure, analyze and track large amounts of user feedback to aid decision making during software evolution.

1. User
Involvement
in
Software
Evolution
Practice:
A
Case
Study
Dennis
Pagano
(@dennispagano),
Bernd
Bruegge
Technische
Universität
München

2. Foundations
• User
Involvement
• Aims
at
maximizing
system
usefulness
and
usability
by
understanding
users’
needs
and
expecta0ons
• So3ware
Evolu7on
• So<ware
has
been
delivered
(this
is
a
simplified
view)
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
2

3. D.
Edsger
Dijkstra
D.
Edsger
Dijkstra
Why
User
Involvement?
«The
no7on
of
’user’
cannot
be
precisely
defined,
and
therefore
has
no
place
in
computer
science
or
so>ware
engineering.»
A.
Alexander
G.
Bell
B.
Michael
Jackson
C.
Christopher
Columbus
That
was
in
1979.
Of
course,
the
rest
is
history.
3
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013

4. In?luential
Changes
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
4
• From
programmers
to
arbitrary
persons
• From
few
vicYms
to
many
paramount
stakeholders
• More
demanding
Users
• From
expensive
scienYfic
programs
to
cheap
apps
• Running
on
new
devices
• Delivered
via
applicaYon
distribuYon
pla[orms
So3ware
• Users
and
use
context
unknown
upfront
• Increasing
distance
between
users
and
developers
➡ User
involvement
changed:
from
user-­‐centered
design
to
con7nuous
interpreta7on
of
post-­‐deployment
user
feedback

5. Study
Goal
and
Questions
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
5
Understand
current
prac7ce
of
user
involvement
during
so3ware
evolu7on
Workflow:
Why
and
how
do
developers
work
with
user
feedback,
and
which
problems
arise?
2
Requirements:
What
do
developers
expect
from
tool
support
for
user
involvement?
3
SeIng:
How,
when,
and
where
may
users
provide
feedback?
1

6. Research
Method
and
Data
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
6
• Semi-­‐structured,
open
ended
interviews
• Audio
recording
and
manual
transcripYon
• Coding
and
building
hypotheses
• Five
developers
from
small
and
medium
sized
companies
(SMEs)
• Focus:
Large
number
of
applicaYon
users
(>
10,000)
• Different
audiences
(consumers
and
professionals)
• Different
plaOorms
(desktop
and
mobile)
Method
Data

7. Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
7
1
User
Involvement
Sefng
Results

8. User
Involvement
Setting
• Four
main
user
feedback
arYfacts:
Error
reports,
feature
requests,
feedback
on
exis7ng
features,
ra7ngs
• User
feedback
scaQered:
email,
applicaYon
distribuYon
pla[orms,
integrated
feedback
mechanisms
etc.
• Users
inten7onally
select
feedback
channel:
The
more
cri7cal
the
feedback,
the
more
public
the
channel
• Users
not
systema7cally
involved:
No
agreed
pracYce
how
to
provide
nor
how
to
gather
user
feedback
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
8
1

9. 2
Results
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
9
User
Involvement
Workflow

10. Bene?its
of
User
Feedback
1. Allows
for
conYnuous
assessment
of
product
acceptance
(real-­‐world
usage
data)
2. Helps
to
improve
so3ware
quality
3. PosiYve
raYngs
and
user
experience
convey
trust:
adver7sement
and
marke7ng
4. Helps
idenYfying
missing
features,
BUT
developers
need
to
assess
how
many
users
will
benefit
from
new
feature
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
10
2

11. User
Feedback
Analysis
• Developers
analyze
user
feedback
to
create
priori7zed
tasks
fifng
into
current
product
roadmap
• Feedback
importance
depends
on
frequency
of
its
occurrence
!"#$"%&'(#)"&%
*+,#-),%&./$0"%1""23-)4&%
5&&"&&%./2.6.2(-0%7#.'#.,8%
*&9:-,"%.:7-),%
;0-&&.18%1""23-)4%
!"#$%&'()*+,&'
!"#$%&',-*#.('
/(&+'0&&123,.'
!"#$%"%&'()!*%+#)
4+")+"56'
7893,5'
,"-%.#)!*%+#)
:++)+'+&9)+5''#'0&35*+&'+&;*&(5'
</&"#,%./%)'/6"/9'/-0%
,''0&%-/2%='#4>'=&%
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
11
2

12. 1. Natural
language
content,
low
feedback
quality,
contradictory
feedback
2. Communica7on
gaps
in
user
feedback
channels
disconnect
users
and
developers
Problems
and
Challenges
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
12
2
➡ Main
problem:
manual
analysis
process
• Developers
read
feedback
mulYple
Ymes
• Manual
esYmaYon
of
how
many
users
are
affected
➡ Feedback
quan7ty
amplifies
these
problems

13. 3
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
13
Tool
Requirements
Results

14. Tool
Requirements
• Consolidate,
structure,
analyze,
and
track
user
feedback
• Group
duplicate
or
similar
feedback,
capture
feedback
type,
and
associated
feature
• Measure
frequency
of
user
feedback
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
14
3
• Collect
feedback
and
analyze
later
• Less
feedback
and
professional
end-­‐users
Re-­‐ac7ve
approach
• RecommendaYon
of
exisYng
feedback
• High
volume
of
non-­‐
confidenYal
feedback
Pro-­‐ac7ve
approach

15. Implications
of
Findings
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013
15
Researchers
1. Extract
informa7on
from
natural
language
user
feedback
(microtext
understanding,
island
parsing)
2. Elicit
context
informa7on
for
analysis
process
Companies
1. Consolidate,
structure,
analyze,
and
track
user
feedback
2. IdenYfy
similar
reports,
feedback
type,
and
affected
feature
3. Assess
feedback
impact
(frequency)
and
reporYng
user
Tool
Developers
1. Offer
fewer,
suitable,
bidirec7onal
feedback
channels
2. Educate
users
how
to
provide
helpful
feedback

16. Feel
free
to
contact
us
for
feedback…
16
Dennis
Pagano
TU
Munich
pagano@cs.tum.edu
@dennispagano
Bernd
Bruegge
TU
Munich
bruegge@cs.tum.edu
Dennis
Pagano
-­‐
TU
Munich
-­‐
May
2013