This whitepaper presents the effectiveness of pairwise test design technique in providing optimized test coverage for testing the variable interactions of a printer driver. A case study has been conducted to present an elaborate comparison with the legacy approach.

1. AdoptingthePairwiseTestDesignTechniqueto
OptimizePrinterDriverTestCoverage

2. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Abstract
Abbreviations
MarketTrendsandChallenges
Solution
CaseStudy
BestPractices
CommonIssuesCommonIssues
Conclusion
Reference
3
3
3
4
5
7
88
9
10
TableofContents

3. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Testingtheinteractionsbetweenprinterdrivervariableshasalwaysbeenachallenge.Consideringthat
testingallthepossiblecombinationsofdrivervariableinteractionsisquiteimpractical,thereisalwaysa
demandforoptimizingthetestcoverage.Pairwisetesting,whichisaproventechniqueinthesoftwaretesting
industry,canbeadoptedtoprovideoptimizedtestcoverageforprinterdrivertesting.
AAcasestudyhasbeenconductedtoexploretheeffectivenessofusingthepairwisetestdesigntechnique
againstthelegacyapproachthatusedrandomization,usingoperationalprofiles.Theinputdatacontained
around61variablesresultinginapproximately2800pairwisetestcasestocompletelytestallvariableinterac-
tions.Thetestcasesgeneratedbythepairwiseapproachandtheteststhatweregeneratedusingthelegacy
approachwereexecutedontheproductundertestanditwasobservedthatthepairwisetestscenarioswere
abletodetect19%moreofdefectsthantheonesthatweredetectedbythelegacytestcaseswithanexecu-
tioneffortsavingsof70%.
Foranyapplicationinthecurrentworld,duetothedistinctvarietiesofusergroups,thereisaneedforaccom-
modatingmoreinputvariablesinasoftwaresystem.Withthisincrease,thecomplexitymultipliesintesting
theinteractionsbetweenallthesevariables.Withonly10variableseachwith2statesperse,thetotaltestsce-
nariostotestallthepossiblecombinationswillbe210
=1024testcases.Ifjustonemorevariableisaddedto
thesoftwaresystem with2states,thetotaltestscenariosbecome2048!
Inaprinterdriver,therearemorethan60inputvariablesleadingtoanunimaginablenumberoftestcombi-
nations;acompletecoverageofwhich,isnearlyimpossibletoprovide.Consideringthecompetitivemarket
Sl.No
1
2
3
4
5
6
77
DRE
TTM
NASA
OS
PDL
PICT
OATSOATS
DefectRemovalEfficiency
TimeToMarket
OperatingSystem
PageDescriptionLanguage
OrthogonalArrayTestStrategy
TheNationalAeronauticsandSpace
Administration,USA
PairwiseIndependentCombinatorial
Testingtoolfrom Microsoft
FullFormAcronyms
Abstract
Abbreviations
MarketTrendsandChallenges
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |3

4. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
environmentandtheneedforareducedtimetomarket(TTM),thetestteamsarechallengedtocomeupwith
variousstrategiestooptimizetestingwithoutcompromisingonthequalityofthedeliverables.Thismay
includereducingtheoverallnumberoftestcasestoaccelerateproducttimetomarket
Sometimes,testteamschooseoption(2)whichcanefficientlyunveilamorenumberofdefectsbutnotmost
ofthem.Withthisoption,providingaquantitativenumberintermsoftestcoverageisachallengeaswell.
Mostoften,option(1)ischosenbymanytestteamswhichbringsineffectivetestcoverage,leadingtopoor
DefectRemovalEfficiency(DRE).ApoorDREmeansmorefielddefectsandlesssatisfiedcustomers.
Byrandomlypickingasubsetoftestsfrom theoverallcountwithoutadoptingastrategy(pureran-
domizationorguessing)
Byrandomlypickingasubsetoftestsfrom theoverallcountusingoperationalprofiles(morefocus
oncustomerusagelikepapersizes/papertypes,highestcustomerbaselikeWindows8.1,Mac10.9,
etc.)or
Bychoosingastatistically-basedtestdesigntechnique.
Thesolutiontothecombinatorialtestingproblem statedinthepreviouspageistoadoptthePairwiseTest
DesignTechniquefortestingtheprinterdriver’svariableinteractions.
PairwisePairwisetestingisaneconomicalalternativetotestingallthepossiblecombinationsofasetofvariables.Itis
aneffectivetestcasegenerationtechniquebasedonthehistoricaldatathatmostfaultsarecausedbyinterac-
tionsoftwoactions.Inpairwisetesting,testcasesaredesignedtoexecuteallthepossiblediscretecombina-
tionsofeachpairofinputparameters.Pairwise-generatedtestsuitescoverallthecombinationsoftwo;and
arethereforemuchsmallerthanexhaustiveones,yetstillveryeffectiveinfindingdefects.Pairwisetestingis
alsoreferredtoasall-pairstestingor2-waytesting.
Graph1showsthecumulativepercentageofsoftwaredefectsdetectedbythen-tupleinteractionswhereitis
apparentthatformedicaldevicesoftwareandNASAapplications,pairwiseinteractionsareabletodetect
morethan90%ofsoftwarefailures.Higher-orderinteractions(3-way,4-way,etc.)canstillbeadopted,ifpre-
ferred,butthebenefit-costratiowillbeconsiderablyless.
Graph1.Errordetectionratesforinteractionstrengths1to6
Cumulativepercentof
softwarefailures
Interactions
MediacalDevices
Browser
Sensor
NASA
NetworkSecurity
Solution
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |4

5. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
TheprocessflowforthepairwisetestgenerationgoesasshowninFigure1.
Intermsofprinterdrivertesting,thekeysuggestionistocombinethetwoformsofCombinatorialTesting–
ConfigurationTesting(testinginteractionsofconfigurationrelatedvariablessuchastheOS,Applications,
PDL,andConnectivity,etc.)andInputParameterTesting(testinginteractionsoftheprinterdriverfeatures).
Acasestudywasconductedwherealaunchedprinterdriverversionwaschosen,andtheeffectivenessofthe
pairwisetestingtechniquewasstudiedagainstthelegacytestapproachthatwasadoptedwhilelaunchingthe
driver.Thelegacyapproachusedrandomizationusingoperationalprofiles.Thedetailedprocessflowofthe
casestudyisshowninFigure2.
Theinputdatacontainedaround61variableswithstatesrangingfrom 1tomorethan40resultinginapproxi-
mately2800testcasestocompletelytestallvariableinteractions.PICT(PairwiseIndependentCombinatorial
Testingtool)wasusedtogeneratethepairwisescenarios.
PICTisafreecommandlinetoolfrom Microsoftthattakesamodelfileasinputandgeneratesacompactset
ofparametervaluechoicesthatrepresentthetestcasestogetcomprehensivecombinatorialcoverageofthe
inputparameters.PICTallowsuserstofeedconstraintswhilegeneratingthepairwisescenarios.Therewere
morethan75constraintsfedintoPICTforthiscasestudy.
Figure2.Detailedprocessflowofthecasestudy
Figure1.Processflowforthepairwise
testgeneration
CaseStudy
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |5

6. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Table1representsasampleoftheinputdatausedforgeneratingpairwisescenarios.
Table2representsasampleofthepairwisedatageneratedforthecasestudy.
Variables
States
Letter(8.5x11”)
Legal(8.5x14”)
Statement(5.5x8.5”)
Executive(7.25x10.5”)
8.5”x13”
AutomaticallySelect
PrinterDefaultType
Plain
HolePunched
Transparency
AutomaticallySelect
White
Blue
Yellow
Green
1SidedPrint
2SidedPrint
2Sidedprint,Flip
onShortEdge
MediaSize MediaType MediaColor 2-SidedPrinting
Variables
Total#oftestcases
Resource(s)Utilized
Duration(InManDays)
EffortSavingsThroughPairwiseStrategy
Uniquedefectdidentified(excl.invalid&duplicate
defects)
%ofdefectsdetectedadditionallybyPairwise
scenories
~5000
6
396
59 70
70%
19%
~2800
3
114
LegacyApproach MediaType
Testcase#
No63/4Envelop
B5(176x250mm)
New
A4(210x297mm)
Letter(8.5x11”)
B5(176x250mm)
1
2
3
4
5
6
Letterhead
Plain
Custom Type2
Custom Type1
Custom Type5
HolePunched
Clear(Transparency)
Gray
Goldenrod
Custom Color3
Pink
Custom Color1
1SidedPrint
2SidedPrint
1SidedPrint
2SidedPrintFliponShortEdge
2SidedPrint
2SidedPrintFliponShortEdge
MediaSize MediaType MediaColor 2-SidedPrinting
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |6
Table1.Sampleofinputdata
Table2.Sampleofpairwisedata
Table3.ComparisondatabetweenthePairwiseapproachandtheLegacyapproach

7. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
From Table3,itisapparentthatthepairwiseapproachfound19% moredefectsthanthelegacyapproach
withasavingsinexecutioneffortof70%.Itwasalsofoundthatthepairwisescenarioswereabletounveilall
thedefectsthatthelegacytestapproachdiscovered,leavingNOleakages.
Table4showsthekeydifferencesbetweenthelegacyapproachandthepairwiseapproach.
AnotherapproachintheindustrythatattemptstosolvetheproblemdefinedinthispaperisOrthogonalArray
TestStrategy(OATS).Orthogonalarraysaretwodimensionalarraysofnumberswhichpossesstheinteresting
qualitythatbychoosinganytwocolumnsinthearrayyoureceiveanevendistributionofallthepair-wise
combinationsofvaluesinthearray.OATSusestheconceptoforthogonalarraystodesigntestsforsoftware
systems.
Asampleorthogonalarray[L9(34)]hasbeenshowninTable5forreference.
BestPractices
LegacyApproach
Randomized
Subjective
Coveragecannotbequantitativelymeasuredasthe
dataisrandomized
Cannotbeautomatedeasily
Optimized
Objective
Coveragecanbequantitativelymeasured
Greatcandidateforautomation!
PairwiseApproach
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |7
Table4.Pairwiseapproachv/sLegacyapproach

8. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Unfortunately,OATSbecomesineffectivewhenthesystem undertestincludesconstraintsbecausethereis
nowayconstraintscanbefedintoorthogonalarrayswhilegeneratingtestscenarios;thiscausesOATSto
generatetestswithinvalidpairs.
AnotherdisadvantagewithOATSisthattheOrthogonalArrayscontainallpairsanequalnumberoftimes.
Thatmeans,usingorthogonalarrayswillresultinmorenumberoftestscomparedtothepairwisestrategy
becausetheyincludepairsequally.Thepairwisestrategycoversallthepairsinalessernumberofteststhan
OATS.
Thechallengewiththepairwiseteststrategyisthattherequirements(includingtheconstraints)shouldbe
accurateandcomplete;rightthefirsttime.TherequirementsshouldNOTchangeoncethepairwisescenarios
aregenerated;theentirepairwisematrixwillhavetobere-generatedifanyonerequirementchangespost
thepairwisetestgeneration.Thisisbecauseeachrowitem inthepairwisematrixisunique,attemptingto
coverasmanyuniquepairsaspossiblewiththeleastnumberoftests.
AAdedicatedeffortisalsorequiredforgeneratingpairwisescenariosforeveryproductundertest,evenfor
similarproducts;evenaslightchangeintherequirementsbetweentwoproductswillcallforseparate
pairwisematrices.
Anotherchallengeconcernstheselectionoftherighttooltobeused.Therearemanyfreeandpaidtools
availableinthemarketandoneneedstoclearlydefinethetoolevaluationcriteriaforpickinguptherighttool.
Theevaluationcriteriashouldconsider,butnotbelimitedto:
CommonIssues
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |8
ControlFactors
Expt.No
1
2
3
4
5
6
77
8
9
1
1
1
2
2
2
33
3
3
1
2
3
1
2
3
11
2
3
1
2
3
2
3
1
33
1
2
1
2
3
3
1
2
22
3
1
A B C D
Table5.SampleOrthogonalArray

9. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Thebudgetallottedforthetestactivity,ifapaidtoolistobeused.
Toolsupport;mostofthefreetoolswillNOThaveanytoolsupportprovidedalongwiththem.
Thenumberofinputvariablesandstates.
Thecomplexityoftheconstraintsthatcomewiththeinputdataandtheabilityofthetooltoaccom-
modatetheseconstraintswhilegeneratingpairwisescenarios.
Easeofuseintermsoffeedingthevariablesandconstraintsintothetool.
SupportinimportingthegeneratedpairwisetestsintoaTestManagementtoolfortestexecution
management.
Thepairwisetestdesigntechniqueisveryeffectiveinprovidingoptimizedtestcoveragefortestingthe
variableinteractionsofaprinterdriver.Withjust2800testscenariosderivedusingthepairwisetestdesign
techniqueascomparedtoapproximately5000testcasesusingthelegacyapproach,thepairwisetechnique
wasabletodetect19%moreofdefectsthantheonesthatweredetectedthroughthelegacyapproachwitha
testexecutioneffortsavingsof70%.
TheThepairwisetestdesigntechniquealsoenablesthetestteamstomeasuretestcoverageinaquantitativeway
asopposedtousingotherstrategieslikepurerandomization,guessingand/orrandomizationusingopera-
tionalprofiles.
Adoptingthepairwisetestdesigntechniquefortestingthevariableinteractionsoftheprinterdriversignifi-
cantlyreducestheexecutioneffort,detectsmoredefectswithlessnumberoftests,andprovidesuswitha
quantitativewayofmeasuringtestcoverage.Thus,thepairwisetechniqueprovesitselftobethebestcombi-
natorialtestdesigntechniquewithareducednumberoftestsandanimproveddefectdetectionrate.
Conclusion
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |9

10. Formoredetailscontact:ers.info@hcl.com
Followusontwitter:http://twitter.com/hclersand
Ourbloghttp://www.hcltech.com/blogs/engineering-and-rd-services
Visitourwebsite:http://www.hcltech.com/engineering-rd-services
Hello,I’m from HCL’sEngineeringandR&DServices.Weenabletechnologyledorganizationstogotomarketwithinnovativeproducts
andsolutions.Wepatnerwithourcustomersinbuildingworldclassproductsandcreatingassociatedsolutiondeliveryecosystems
to help bringmarketleadership.Wedevelop engineeringproducts,solutionsand platformsacrossAerospaceand Defense,
Automotive,ConsumerElectronics,Software,Online,IndustrialManufacturing,MedicalDevices,NetworkingandTelecom,Office
Automation,SemiconductorandServers&Storageforourcustomers.
ThiswhitepaperispublishedbyHCLEngineeringandR&DServices.
Theviewsandopinionsinthisarticleareforinformationalpurposesonlyandshouldnotbeconsideredasasubstituteforprofessional
businessadvice.TheusehereinofanytrademarksisnotanassertionofownershipofsuchtrademarksbyHCLnorintendedtoimply
anyassociationbetweenHCLandlawfulownersofsuchtrademarks.
FormoreinformationaboutHCLEngineeringandR&DServices,
Pleasevisithttp://www.hcltech.com/engineering-rd-services
Copyright@ HCCopyright@ HCLTechnologies
Allrightsreserved.
SwadishAaryan
HCLEngineeringandR&DServices
Reference
AuthorInfo
1)http://www.pairwise.org/
2)PairwiseTestingintheRealWorld:
PracticalExtensionstoTest-CaseScenarios.(http://msdn.microsoft.com/en-us/library/cc150619.aspx#Pairwise_topic1)
3)JamesBachandPatrickJ.Schroeder,PairwiseTesting:ABestPracticeThatIsn’t
4)JacekCzerwonka,PICT3.3User’sGuide
5)PracticalCombinatorialTesting.SP800-142.(Report).Natl.Inst.ofStandardsandTechnology.2010.
AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |10