My Talk at ASE conference

1. Scalable Product Line Configuration:
A Straw to Break the Camel’s Back
Abdel Salam Sayyad
Joseph Ingram
Tim Menzies
Hany Ammar

2. Sound bites
Real Software Product Lines…
… are big, complex
… representative feature models are now available for research
… scale up or die !
What does it take?
Multi-objective problem formulation
An optimizer that relies heavily on user preferences
Respect domain constraints, and...
… A Straw to Break the Camel’s Back
2

3. Roadmap
① State of the Art
② Experiment & Results
③ Conclusion

4. Roadmap
① State of the Art
② Experiment & Results
③ Conclusion

5. Feature Models
• Feature models = a
lightweight method for
defining a space of options
• De facto standard for
modeling variability
Cross-Tree Constraints
Size ? 10 Features, 8 Rules
Cross-Tree Constraints
5

6. Feature Model Repository
Size ? 290 Features, 421 Rules
cardinal = 2.26x1049 *Pohl ‘11+
6

7. Feature Models of Real Software Projects
Size ? 6888 Features; 344,000 Rules
cardinal? Forgeddaboutit!
7

8. Size isn’t all…
Property
Size
SPLOT
Significantly smaller
Linux
Significantly larger
Constraints
Feature Groups
Leaves
Significantly less
High ratio
Deeper
Significantly more
Low ratio
Shallower
T. Berger, S. She, R. Lotufo, A. Wasowski, and K. Czarnecki, "A Study of Variability
Models and Languages in the Systems Software Domain," IEEE Tran Soft Eng. 2013
8

9. A Multi-Objective Formulation
Multi-objective optimization = navigating competing concerns
– Success criteria = choose features that achieve the user preferences!
Suppose each feature had the following metrics:
1. Boolean
2. Integer
3. Real
USED_BEFORE?
DEFECTS
COST
Show me the space of “best options” according to the objectives:
1.
2.
3.
4.
5.
That satisfies most domain constraints (0 ≤ #violations ≤ 100%)
That offers most features
That we have used most before
Using features with least known defects
Using features with least cost
9

10. No single “optimum” solution
The Pareto Front
Higher-level
Decision Making
The Chosen Solution
10

11. State of the Art
Features
Linux (LVAT)
6888
544
Single-objective
Johansen
‘11
Henard
‘12
Multi-objective
Sayyad ’13b
White ‘07, ‘08, 09a, 09b,
Shi ‘10, Guo ‘11
SPLOT
290
9
Pohl ‘11
Benavides
‘05
LopezHerrejon
‘11
Sayyad
’13a
Olaechea
‘13
Objectives
11

12. Roadmap
① State of the Art
② Experiment & Results
③ Conclusion

13. Two Algorithms
1) NSGA-II [Deb et al. ‘02+
Non-dominated Sorting
Genetic Algorithm
Binary Dominance
2) IBEA *Zitzler and Kunzli ‘04]
Indicator-Based Evolutionary
Algorithm
Continuous Dominance
13

14. 7 Feature Models
14

15. Feature Fixing
• Look for mandatory or dead features.
• Fix those feature in the evolution.
• Skip rules with only those features.
15

16. Experiment
Low Crossover & Mutation rates
Stopping Criteria
16

17. HV
= Hypervolume of dominated region
%Correct = % of solutions that are fully correct (out of 300 solutions)
17

18. Comparing %Correct
i.e. % of solutions that are fully correct (out of 300 solutions)
NSGA-II No FF
100%
NSGA-II with FF
80%
IBEA No FF
IBEA with FF
60%
40%
20%
0%
ToyBox
axTLS
eCos
FreeBSD
Fiasco
uClinux
Linux X86
18

19. How to crack the nut?
• What if we told IBEA what good solutions look
like?
• Good solution? … A correct solution, with:
• Minimal skeleton?
• Maximum features?
• Then what?
• Seed the initial population with a pre-calculated
“good” solution.
19

20. • How to generate good (correct) solutions?
• Z3 SMT (Satisfiability Module Theory) Solver
• Two-objective IBEA, min violations, max features
• Z3: super fast, low on selected features
• 2-obj IBEA: slow, but feature-rich
20

21. Correct solutions after 30 minutes for
Linux Kernel feature model
21

22. Correct solutions after 30 minutes for
Linux Kernel feature model
… A Straw to Break the Camel’s Back
22

23. More recent result
• added the “PULL” trick (give more objective weight
to constraint violations)
23

24. Why is this interesting?
• 20 correct solutions to choose from within 2 minutes
24

25. Do I have the user’s attention?
• 2 seconds…
• to stay focused
• 10 seconds…
• to stay on task
• few minutes?
• To provide
timely input
at a meeting
25

26. Roadmap
① State of the Art
② Experiment & Results
③ Conclusion

27. Sound bites
Scalability
Method innovation is key
Problem Formulation
Give the best (comprehensive yet
concise) picture to the user.
IBEA with Feature Fixing
Continuous dominance
Respect domain constraints
Acknowledgment
This research work was
funded by the Qatar
National Research Fund
(QNRF) under the National
Priorities Research Program
(NPRP)
Grant No.: 09-1205-2-470.
Enter Z3?
One day we will build that bridge!
27