Luca Sabatucci Ahab's Leg Dilemma - on the Design of a Controlled Experiment Empire 2011, Trento, Italy

1. Ahab's Leg Dilemma:
on the Design of a
Controlled Experiment
Luca Sabatucci
Mariano Ceccato
Alessandro Marchetto
Angelo Susi

2. The Ahab’s Leg dilemma
• When changing media (or communication
style) we need to add details to a story, to
keep the story engaging
Umberto Eco
1956 Ray Bradbury & John Huston.

3. The Ahab’s Leg dilemma
• The peg leg is fundamental for the story
– Deciding which leg is a peg one has no bearing on it
– When the peg-leg is instantiated, this decision may
generate of lot of consequences
1930 Warner Bros. 1956 Ray Bradbury & John Huston.
Directed by Lloyd Bacon.

4. Ahab’s Leg in RE
• Often, narrative scenarios are used to validate requirements
with stakeholders in focus groups
• Narrative scenarios are derived from requirements (change of
communication style)
• Details must be added during translation to instantiate generic
requirements into a concrete spatial-temporal context
• Stakeholders might be distracted by irrelevant details

5. An example from our experience
Requirement:
the system communicates with caregivers with low and high priority signals
The camera PDA displays
identifies Fall on that an
the event staircase unknown person
and sends a is fallen in the
signal to staircase
caregiver’s
PDA
Maria falls on the staircase
• The focus group was proceeding well until a nurse commented on
the PDA (Ahab’s Leg)
– PDA is intrusive (to carry around, battery…) and it would change
working practices
– Lively discussion on less intrusive devices
– This was pointless, because the kind of device was not yet decided
Credits: picture (c) By Chiara Leonardi

6. Mandatory Vs Optional
• Some are mandatory to make the story concrete and believable
(e.g., PDA)
– Concreteness is important for stakeholders to envisage functionalities
• Some are optional and choreographic, needed just to increase the
stakeholders engagement (e.g., name of the patient)
• Does mandatory and optional ALs affect in the same way
requirement validation sessions?
The camera PDA displays
identifies Fall on that an
the event staircase unknown person
and sends a is fallen in the
signal to staircase
caregiver’s
PDA
Maria falls on the staircase
Credits: picture (c) By Chiara Leonardi

7. Cardinality
• Optional ALs can be removed
– Many: scenarios very concrete but with the risk of high distraction
– Few: scenarios very abstract, difficult to present to stakeholders
• Influence of personal and contextual background:
– Melville did not specify if all the member of the crew had two legs. But the
reader assumes it based on his/her common knowledge of the real world
– Abstract scenarios make stakeholders mentally complete missing details
• The initial scenario is corrupted
• No possibility to control ALs by the analyst
The camera PDA displays
identifies Fall on that an
the event staircase unknown person
and sends a is fallen in the
signal to staircase
caregiver’s
PDA
Maria falls on the staircase

8. Stakeholder awareness
• Not realistic to compare scenario with and
without ALs (the latter does not make sense)
• If the discussion is moderated by a facilitator,
he/she could highlight irrelevant details to avoid
spending time in discussing about them
– Risk of attracting even more attention on them
– Just mention that there are more and less important
details (with some example)
• Exploiting stakeholder awareness is probably the
more realistic approach and analyst would take,
to limit distraction.

9. The plan
• The role of Ahab’s Leg has been observer during a real
project validation session
• We conjecture that this is not due to the specific project,
but the problem is more general
• Test this conjecture in a controlled and repeatable in-lab
experiment
– We control/measure all the relevant variables
– We change just one variable and we study the effect

10. Research questions
• Ahab’s Legs are often unavoidable
• Not a problem, unless they divert the
attention from important aspects of the story.
RQ1: What is the actual impact of Ahab’s Legs on the
distraction of stakeholders during a requirement
validation sessions?
RQ2: Is there a reliable way to reduce their impact on
the distraction?

11. Context of the experiment
• Subjects: Computer science master
students.
– Background on software and
requirement engineering
– Some actual development experience
• Objects: 2 software system found
on the internet
– MyBanking: home banking application for
mobile devices, designed to replace credit
cards and cash.
– MyShopping: augmented reality
application for mobile devices that display
information on items pointed by the
camera.

12. Hypothesis formulation
• H0: explicitly mentioning what are the over-
specified details (Ahab’s Legs) in application
scenarios does not significantly reduce the
distraction in a requirement validation
session.
• HA: explicitly mentioning what are the over-
specified details (Ahab’s Legs) in application
scenarios significantly reduces the distraction
in a requirement validation session.

13. Variable selection
• Independent variable: explicitly telling that there are
details irrelevant for the discussion (Ahab’s Legs) or
without such explicit mention.
• Dependent variable: distraction observed during the
requirement validation phase.
– The stakeholder comment addresses a topic that is not
part of the requirement (e.g., Ahab’s Leg) distractioni =1
– The comment requires to fix/reconsider a requirement
distractioni =0
– Disrtaction = distractioni

14. Co-factors that we measure
• Learning effect between the two labs
• System for which requirements are validated
• Subjects’ academic merit as the average of exam
score
• Subjects’ background measured as they attended
relevant courses
• Previous subjects’ experience
– On requirement validation
– On industrial software development

15. Experimental design
Group1 Group2 Group3 Group4
Lab1 MyBanking AL MyBanking No MyShopping AL MyShopping No
Lab2 MyShopping No MyShopping AL MyBanking No MyBanking AL
• Fill the profiling pre-questionnaire
• Lab 1
– Read the description of the first application
– For each of the 4 scenarios
• Read a scenario
• Write a comment/question for the scenario
• Lab 2
– Read the description of the first application
– For each of the 4 scenarios
• Read a scenario
• Write a comment/question for the scenario
• Fill the feedback questionnaire.

16. Missing aspects?
• Other strategies to control the influence of
Ahab’s Leg dilemma?
• Trade-off between distraction and level of
abstraction?
• What is the influence of subject background?

17. Conclusions

18. Questions?