On#the#Value#of#User#Preferences#in#
Search4Based#So7ware#Engineering:##
A#Case#Study#in#So7ware#Product#Lines##
#
Tim#Menzies,##
Abdel#Salam#Sayyad,#
Hany#Ammar#
#
WVU,#USA#
Nov’12##
#

Sound#bites#
• The#new#age#of#the#app#
#
• Stop#Nnkering#with#small#stuff#
#
• Enough#with#the#usual#suspects:##
– NSGA4II,#SPEA2,#etc#
#
• If#preferences#maSer#
– Then#the##best#opNmizer#understands#preferences#the#best#
2#

Roadmap#
① Feature(based,SE,
② Algorithms,
③ IBEA,
④ Tree,muta<on,
⑤ Conclusion,
#

Roadmap#
① Feature(based,SE,
② Algorithms,
③ IBEA,
④ Tree,muta<on,
⑤ Conclusion,
#

WELCOME,TO,,
THE,NEW,WORLDE,,
,
5#

The#Nmes,#they#are#a#changing#
Olde,worlde:,, New,worlde:,,
product(based,SE, app(based,SE,,
e.g.#Microso7#office# • E.g.#Apple#app#store#
6#

The#Nmes,#they#are#a#changing#
Olde,worlde:,, New,worlde:,,
product(based,SE, app(based,SE,,
• Vendors#tried#to#retain#their# • Smart#phones#and#tablet4
user#base#via#some# based#so7ware#
complete#ecologies# #
# • Users#choosing#many#
numbers#of#small#apps#from#
• One#so7ware#soluNon#for# different#vendors,##
all#user#needs###(e.g.# – each#performing#a#specific#
Microso7#Office).## small#task.#
# #
• Large,#complex,#so7ware# • Vendors#must#quickly#and#
plaorms,## conNnually#reconfigure#apps##
– To#retain#and#extend#their#
– very#slow#to#change.## customer#base.#
7#

Feature–oriented#domain#analysis#
• Feature#maps#=#a#
lightweight#method#for#
defining#a#space#of##opNons#
• Product4line#configuraNons#
• Defacto#standard#for#
modeling#variability##
8#

Original#FODA#paper#:#2700+#citaNons#
Half#since#
2007#
9#

hSp://www.splot4research.org/#
200+#models,##plus#an#instance#generator# 10#

LINUX#kernel#=#6000+#features#
86%#declare#constraints#of#some#sort,##
Most#features#refer#to#244#other#features.##
#
11#

Need#for#beSer#automaNon#
• Such#complexity#needs#automated#support#
– especially##feature#models##combined#with#
• #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.##
Search#for#valid#products:##
9#state#of#the#art#theorem#
provers##[Pohl,#ASE’11]#
#
Bad##
scalability#
And#these#were#
“simple”#models#
12#

Diving##deeper#
• Much#prior#work#explored##
Nny#objecNve#spaces##
– Two#or#three#objecNves#
– Or,#higher#(but#only#for#small#models)#
#
#
#
• So7ware#engineering#=#navigaNng#compeNng#concerns#
1. That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)# The#
usual#
2. That#offers#most#features# suspects#
3. Build#“stuff”#In#least#Nme#
4. That#we#have#used#most#before#
13#
5. Using#features#with#least#known#defects#

Roadmap#
① Feature(based,SE,
② Algorithms,
③ IBEA,
④ Tree,muta<on,
⑤ Conclusion,
#

MOEA=#MulN4objecNve###
evoluNonary#algorithms#######################
• Repeat#Nll#happy#or#exhausted#
– SelecNon#(cull#the#herd)#
– Cross4over#(the#rude#bit)#
– MutaNon#(stochasNc#jiggle)#
15#

Some#MOEA#ApplicaNon#
Domain Application Types
Control gas pipeline, pole balancing, missile evasion, pursuit
Design semiconductor layout, aircraft design, keyboard
configuration, communication networks
Scheduling manufacturing, facility scheduling, resource allocation
Robotics trajectory planning
Machine Learning designing neural networks, improving classification
algorithms, classifier systems
Signal Processing filter design
Game Playing poker, checkers, prisoner’s dilemma
Combinatorial set covering, travelling salesman, routing, bin packing,
Optimization graph colouring and partitioning

MOEA#for#Search4based#SE#
TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams#
Requirements## #Bagnall,#Mansouri,#Zhang#
Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd##
Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe#
Heap#allocaNon #Cohen,#Kooi,#Srisa4an##
Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer##
SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani##
Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS#
Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,##
## # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,#
## # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins#
Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7#
Model#checking #Alba,#Chicano,#Godefroid#
Probing # # #Cohen,#Elbaum##
UIOs# # # #Derderian,#Guo,#Hierons# So#study#FODA4#
Comprehension #Gold,#Li,#Mahdavi# to#learn#how#to#
Protocols# # #Alba,#Clark,#Jacob,#Troya# improve#these#
Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo# tasks.#
Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis#
17#

Much#increased#interest##
in#Search4based#SE#
18#

The#Pareto#FronNer#
• Mutants#=#<D,O>#=#<decisions,#objecNves>#
– E.g.#car#
• Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels#
• ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)#
– E.g.#learning#formula#
• Decisions:#what#variables#and#constants#to#use#
• ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may#
complete)#
• Pareto#fronNer:#select#the#non4dominated#mutants#
– X#dominates#Y##
• if#for#all#objecNves,##X#is#never#any#worse##than#Y#
• If#for#one#objecNve,#X#beSer#than#Y#
19#

Once#you#know#fronNer#
•#Select#from#here#
Issues:# •#Ignore#here#
1) Spread#
2) Hypervolume#
3) ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)# 20#

The#usual#suspects:#
##=#NSGA4II###
#=#SPEA2#
In#this#case,#NSGA4II#
gets#more#spread#
Combines#N#objecNves##
to#one##with#some##
weighNng#scheme#
21#

Some#details#on#the#usual#suspects#
NSGA(II,, SPEA2,,
• Is#a#geneNc#algorithm# • Is#a#geneNc#algorithm#
# #
• Changes#the#definiNon#of#
• Non4dominated#sort# “dominaNon”#
– HeurisNc#way#to#fast#group# – SPEA#(version#1)#scored#mutants#by#
mutants#into#bands# how#many#others#they#dominated#
# – Got#confused#by#overlaps#in#the#
dominaNon#sets#
• Crowd#pruning##via#approximate# #
hypercube#around#each#mutant:# • SPEA#(version#2):#
O(Onlogn)# – Adds#a#“local#density#factor”#to#the#
dominaNon#weight#
3# – Mutants#in#dense#areas#valued##
2# more#
##
1# • SPEA2#beSer#than#SPEA1#
2# 22#

Any#number#of#opNmizaNons##
to#tradiNonal#GAs#
• The#history#of#MOEAs#in#the#last# • DifferenNal#evoluNon#(Storn#1996)#
15#years#is# – Mutate#by#interpolaNons#between#
– OpNmize#via#hybrid#GA#+#other# exisNng#mutants#
search#method#
# – For#x#in#mutants##
####y#=#any1#+#extrapolate(any3#–#any2)#
• Local#search:## ####if#y#dominates#x#then#x#=#y#
– before#select,#do#a#liSle#simulated# #
annealing#on#X%#of#the#populaNon#
# • Cellular#automata,#
• ScaSer#search#(Glover’s#next# • #Ant#colony#opNmizaNon,####
generaNon#tabu#search)#
– Includes#a#liSle#local#search#
• Bayesian#staNsNcs#to#bias#the#
# mutaNon,##
• ParNcle#swam#opNmizaNon# • Etc##
– May#do#as#well#as#scaSer#search##(Yin# • etc#
and#Glover#2007)#
23#

Roadmap#
① Feature(based,SE,
② Algorithms,
③ IBEA,
④ Tree,muta<on,
⑤ Conclusion,
#

Three#groups#of#Algorithms#
DominaNon# DominaNon#
Is#a#binary#
PSO# Is#a#conNnuous#
concept# concept#
DE# ScaSer## IBEA#
Spea2# search# Aggressive#
exploraNon#
Indicator4based## of#preference##
SA# evoluNonary# space#
Nsga4II# algorithms#
mocell#
Z3#
SMT#solvers#
25#

IBEA#
• Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences#
• I(x1,x2):#
– Least#adjust#objecNve#scores#such#that#x1#dominates#x2#
• Repeat#Nll#just#a#few#le7#
– Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else#
# K=#
# 0.05#
#
#
#
#
– Sort#all#instances#by#F#
– Delete#worst#
• Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors#
##
26#

Case#studies#
Data#from#hSp://www.splot4research.org/#
Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/##
Cross4tree#
constraints#
27#

4#studies:#
Bi,#tri,#quad,#five4#objecNves#
So7ware#engineering#=#navigaNng#compeNng#concerns#
1. That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#
100%)#
2. That#offers#most#features#
3. Build#“stuff”#In#least#Nme#
4. That#we#have#used#most#before#
5. Using#features#with#least##
known#defects#
#
#
Binary#objecNves#=#1,2#
Tri4objecNve#########=#1,2,3#
Quad4objecNve####=#1,2,3,4#
Five4objecNve######=#1,2,3,4,5#
28#

#
HV#############=#hypervolume#of#dominated#region#
Spread######=#coverage#of#fronNer#
%#correct#=#%constraints#saNsfied#
29#

#
HV#############=#hypervolume#of#dominated#region#
Spread######=#coverage#of#fronNer#
%#correct#=#%constraints#saNsfied#
Comment,1:,all#about#the#same#for#the#24objecNve#problem# 30#

#
HV#############=#hypervolume#of#dominated#region#
Spread######=#coverage#of#fronNer#
%#correct#=#%constraints#saNsfied#
Comment,2:,E4shop#is#a#nasty#problem:#needs#50M#evals# 31#

#
HV#############=#hypervolume#of#dominated#region#
Spread######=#coverage#of#fronNer#
%#correct#=#%constraints#saNsfied#
Comment,3:,IBEA#has#no#spread#operators,#but#gets#best#spread#32#

#
HV#############=#hypervolume#of#dominated#region#
Spread######=#coverage#of#fronNer#
%#correct#=#%constraints#saNsfied#
Comment,4:,IBEA#has#no#HV#operators,#but#usually#gets#best#HV#33#

#
HV#############=#hypervolume#of#dominated#region#
Spread######=#coverage#of#fronNer#
%#correct#=#%constraints#saNsfied#
Comment,5:,All#the#non4IBEA#algorithms#are#very#similar# 34#

#
HV#############=#hypervolume#of#dominated#region#
Spread######=#coverage#of#fronNer#
%#correct#=#%constraints#saNsfied#
Comment,6:,IBEA#does#much,#much##beSer#on#constraints# 35#

Why#is#this#interesNng?#
Other,MOEAs, IBEA,
• The#usual#suspects#are#widely,# • Rather#stupid#on#those#
uncriNcally#used#in#many#MOEA#
applicaNons# internal#tricks#
– E.g.#especially#NSGA4II#and#SPEA2# – Just#does#a#ye#olde#crossover#
• Focused#on#internal#algorithmic# mutate#GA#
tricks# – Plus:#aggressive#exploraNon#
– Techniques#for# of#the#preference#space#
• #improving#spread##
• Improving#HV# • And#the#net#effect#of#all#
• Avoid#overlaps#in#cross4over#of#
dominated#space# those#differences#
• etc# – BeSer#spreads#
• And#the#net#effect#of#all#those# – BeSer#HV#
differences?#
– Not#much# – Fewer#constraint#violaNons#
Conclusion:#
preference#is#power# 36#

Roadmap#
① Feature(based,SE,
② Algorithms,
③ IBEA,
④ Tree,muta<on,
⑤ Conclusion,
#

What#about#non4MOEA#soluNons?#
DominaNon# DominaNon#
Is#a#binary#
PSO# Is#a#conNnuous#
concept# concept#
DE# ScaSer## IBEA#
Spea2# search# Aggressive#
exploraNon#
Indicator4based## of#preference##
SA# evoluNonary# space#
Nsga4II# algorithms#
mocell#
Z3#
SMT#solvers#
38#

Ethan’s#complaint#
• So7ware#engineer##
designs#are#o7en##
nested#hierarchical##
constraints#
• Ethan#Jackson,#Microso7,##
advocate#for#the#Z3#SMT#solver:#
– Why#mutate#at#random,#then#check#for#
constraint?#
– BeSer#to#drive#the#mutaNons#by#the#
constraints?#
39#

Dump#MOEAs?##
Move#to#more#logical#forms?#
Pro:, Con:,
move,to,,say,,SMT,solvers, stay,with,MOEA,
• Next#generaNon#of# • ImplementaNon#complexity#
algorithm# less#
– The#next#big#thing# • More#tools#
• BeSer#to#drive#the# • Easier#modificaNon#and#
mutaNons#by#the# experimentaNon#
constraints?# • Models#can#be#expressed#
anyway#you#like#
40#

Tree#mutaNon#
• Work#in#progress#
• Simple#adaptaNon#of#current#MOEAs#for#systems#
of#hierarchical#constraints#
#
if#rand(0,1)<#mutaNon_probability:###
#########Don't#mutate#if#you're#violaNng#one#of#the#rules:#
# #1)#if#deselecNng#root#feature#
# #2)#if#selecNng#feature#whose##
##########################parent#is#not#selected#
# #3)#if#deselecNng#feature#that#
#########################another#selected#feature#requires#
# #4)#if#group#cardinality#violaNon#
#else:#
# #flip#this#bit# # # #####
# # #if#selecNng#(turning#on)#a#feature#then#
# # # #turn#on#children### IBEA#
# # #else#if#deselecNng#(turning#off)# stabilizes#70#
#######################################feature#then:# Nmes#faster#
# # ######### #########turn#off#all#children# 41#

Tree#mutaNon#preserves##
domain#constraints#
So#what##
case#for#
SMT?#
42#

Roadmap#
① Feature(based,SE,
② Algorithms,
③ IBEA,
④ Tree,muta<on,
⑤ Conclusion,
#

Sound#bites#
• The#new#age#of#the#app#
– In,this,new,worlde:,,use,FODA,
(feature(oriented,domain,analysis),
• Stop#Nnkering#with#small#stuff##
– Many,MOEAs,have,strikingly,,
similar,performance,
• Enough#with#the#usual#suspects:##
– NSGA4II,#SPEA2,#etc#
– Too,much,uncri<cal,applica<on,of,these,algorithms,
• If#preferences#maSer#
– Then#the##best#opNmizer#understands##preferences#the#best#
– IBEA:,aggressive,preference,explora<on,
– Tree,muta<on:,respect,your,domain,
44#