Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
AdoptingthePairwiseTestDesignTechniqueto
OptimizePrinterDriverTestCoverage
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Abstract
...
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Testingth...
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
environme...
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Theproces...
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Table1rep...
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
From Tabl...
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Unfortuna...
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Thebudget...
Formoredetailscontact:ers.info@hcl.com
Followusontwitter:http://twitter.com/hclersand
Ourbloghttp://www.hcltech.com/blogs/...
Nächste SlideShare
Wird geladen in …5
×

Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage

617 Aufrufe

Veröffentlicht am

This whitepaper presents the effectiveness of pairwise test design technique in providing optimized test coverage for testing the variable interactions of a printer driver. A case study has been conducted to present an elaborate comparison with the legacy approach.

Veröffentlicht in: Business
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage

  1. 1. AdoptingthePairwiseTestDesignTechniqueto OptimizePrinterDriverTestCoverage
  2. 2. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. Abstract Abbreviations MarketTrendsandChallenges Solution CaseStudy BestPractices CommonIssuesCommonIssues Conclusion Reference 3 3 3 4 5 7 88 9 10 TableofContents
  3. 3. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. Testingtheinteractionsbetweenprinterdrivervariableshasalwaysbeenachallenge.Consideringthat testingallthepossiblecombinationsofdrivervariableinteractionsisquiteimpractical,thereisalwaysa demandforoptimizingthetestcoverage.Pairwisetesting,whichisaproventechniqueinthesoftwaretesting industry,canbeadoptedtoprovideoptimizedtestcoverageforprinterdrivertesting. AAcasestudyhasbeenconductedtoexploretheeffectivenessofusingthepairwisetestdesigntechnique againstthelegacyapproachthatusedrandomization,usingoperationalprofiles.Theinputdatacontained around61variablesresultinginapproximately2800pairwisetestcasestocompletelytestallvariableinterac- tions.Thetestcasesgeneratedbythepairwiseapproachandtheteststhatweregeneratedusingthelegacy approachwereexecutedontheproductundertestanditwasobservedthatthepairwisetestscenarioswere abletodetect19%moreofdefectsthantheonesthatweredetectedbythelegacytestcaseswithanexecu- tioneffortsavingsof70%. Foranyapplicationinthecurrentworld,duetothedistinctvarietiesofusergroups,thereisaneedforaccom- modatingmoreinputvariablesinasoftwaresystem.Withthisincrease,thecomplexitymultipliesintesting theinteractionsbetweenallthesevariables.Withonly10variableseachwith2statesperse,thetotaltestsce- nariostotestallthepossiblecombinationswillbe210 =1024testcases.Ifjustonemorevariableisaddedto thesoftwaresystem with2states,thetotaltestscenariosbecome2048! Inaprinterdriver,therearemorethan60inputvariablesleadingtoanunimaginablenumberoftestcombi- nations;acompletecoverageofwhich,isnearlyimpossibletoprovide.Consideringthecompetitivemarket Sl.No 1 2 3 4 5 6 77 DRE TTM NASA OS PDL PICT OATSOATS DefectRemovalEfficiency TimeToMarket OperatingSystem PageDescriptionLanguage OrthogonalArrayTestStrategy TheNationalAeronauticsandSpace Administration,USA PairwiseIndependentCombinatorial Testingtoolfrom Microsoft FullFormAcronyms Abstract Abbreviations MarketTrendsandChallenges AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |3
  4. 4. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. environmentandtheneedforareducedtimetomarket(TTM),thetestteamsarechallengedtocomeupwith variousstrategiestooptimizetestingwithoutcompromisingonthequalityofthedeliverables.Thismay includereducingtheoverallnumberoftestcasestoaccelerateproducttimetomarket Sometimes,testteamschooseoption(2)whichcanefficientlyunveilamorenumberofdefectsbutnotmost ofthem.Withthisoption,providingaquantitativenumberintermsoftestcoverageisachallengeaswell. Mostoften,option(1)ischosenbymanytestteamswhichbringsineffectivetestcoverage,leadingtopoor DefectRemovalEfficiency(DRE).ApoorDREmeansmorefielddefectsandlesssatisfiedcustomers. Byrandomlypickingasubsetoftestsfrom theoverallcountwithoutadoptingastrategy(pureran- domizationorguessing) Byrandomlypickingasubsetoftestsfrom theoverallcountusingoperationalprofiles(morefocus oncustomerusagelikepapersizes/papertypes,highestcustomerbaselikeWindows8.1,Mac10.9, etc.)or Bychoosingastatistically-basedtestdesigntechnique. Thesolutiontothecombinatorialtestingproblem statedinthepreviouspageistoadoptthePairwiseTest DesignTechniquefortestingtheprinterdriver’svariableinteractions. PairwisePairwisetestingisaneconomicalalternativetotestingallthepossiblecombinationsofasetofvariables.Itis aneffectivetestcasegenerationtechniquebasedonthehistoricaldatathatmostfaultsarecausedbyinterac- tionsoftwoactions.Inpairwisetesting,testcasesaredesignedtoexecuteallthepossiblediscretecombina- tionsofeachpairofinputparameters.Pairwise-generatedtestsuitescoverallthecombinationsoftwo;and arethereforemuchsmallerthanexhaustiveones,yetstillveryeffectiveinfindingdefects.Pairwisetestingis alsoreferredtoasall-pairstestingor2-waytesting. Graph1showsthecumulativepercentageofsoftwaredefectsdetectedbythen-tupleinteractionswhereitis apparentthatformedicaldevicesoftwareandNASAapplications,pairwiseinteractionsareabletodetect morethan90%ofsoftwarefailures.Higher-orderinteractions(3-way,4-way,etc.)canstillbeadopted,ifpre- ferred,butthebenefit-costratiowillbeconsiderablyless. Graph1.Errordetectionratesforinteractionstrengths1to6 Cumulativepercentof softwarefailures Interactions MediacalDevices Browser Sensor NASA NetworkSecurity Solution AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |4
  5. 5. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. TheprocessflowforthepairwisetestgenerationgoesasshowninFigure1. Intermsofprinterdrivertesting,thekeysuggestionistocombinethetwoformsofCombinatorialTesting– ConfigurationTesting(testinginteractionsofconfigurationrelatedvariablessuchastheOS,Applications, PDL,andConnectivity,etc.)andInputParameterTesting(testinginteractionsoftheprinterdriverfeatures). Acasestudywasconductedwherealaunchedprinterdriverversionwaschosen,andtheeffectivenessofthe pairwisetestingtechniquewasstudiedagainstthelegacytestapproachthatwasadoptedwhilelaunchingthe driver.Thelegacyapproachusedrandomizationusingoperationalprofiles.Thedetailedprocessflowofthe casestudyisshowninFigure2. Theinputdatacontainedaround61variableswithstatesrangingfrom 1tomorethan40resultinginapproxi- mately2800testcasestocompletelytestallvariableinteractions.PICT(PairwiseIndependentCombinatorial Testingtool)wasusedtogeneratethepairwisescenarios. PICTisafreecommandlinetoolfrom Microsoftthattakesamodelfileasinputandgeneratesacompactset ofparametervaluechoicesthatrepresentthetestcasestogetcomprehensivecombinatorialcoverageofthe inputparameters.PICTallowsuserstofeedconstraintswhilegeneratingthepairwisescenarios.Therewere morethan75constraintsfedintoPICTforthiscasestudy. Figure2.Detailedprocessflowofthecasestudy Figure1.Processflowforthepairwise testgeneration CaseStudy AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |5
  6. 6. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. Table1representsasampleoftheinputdatausedforgeneratingpairwisescenarios. Table2representsasampleofthepairwisedatageneratedforthecasestudy. Variables States Letter(8.5x11”) Legal(8.5x14”) Statement(5.5x8.5”) Executive(7.25x10.5”) 8.5”x13” AutomaticallySelect PrinterDefaultType Plain HolePunched Transparency AutomaticallySelect White Blue Yellow Green 1SidedPrint 2SidedPrint 2Sidedprint,Flip onShortEdge MediaSize MediaType MediaColor 2-SidedPrinting Variables Total#oftestcases Resource(s)Utilized Duration(InManDays) EffortSavingsThroughPairwiseStrategy Uniquedefectdidentified(excl.invalid&duplicate defects) %ofdefectsdetectedadditionallybyPairwise scenories ~5000 6 396 59 70 70% 19% ~2800 3 114 LegacyApproach MediaType Testcase# No63/4Envelop B5(176x250mm) New A4(210x297mm) Letter(8.5x11”) B5(176x250mm) 1 2 3 4 5 6 Letterhead Plain Custom Type2 Custom Type1 Custom Type5 HolePunched Clear(Transparency) Gray Goldenrod Custom Color3 Pink Custom Color1 1SidedPrint 2SidedPrint 1SidedPrint 2SidedPrintFliponShortEdge 2SidedPrint 2SidedPrintFliponShortEdge MediaSize MediaType MediaColor 2-SidedPrinting AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |6 Table1.Sampleofinputdata Table2.Sampleofpairwisedata Table3.ComparisondatabetweenthePairwiseapproachandtheLegacyapproach
  7. 7. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. From Table3,itisapparentthatthepairwiseapproachfound19% moredefectsthanthelegacyapproach withasavingsinexecutioneffortof70%.Itwasalsofoundthatthepairwisescenarioswereabletounveilall thedefectsthatthelegacytestapproachdiscovered,leavingNOleakages. Table4showsthekeydifferencesbetweenthelegacyapproachandthepairwiseapproach. AnotherapproachintheindustrythatattemptstosolvetheproblemdefinedinthispaperisOrthogonalArray TestStrategy(OATS).Orthogonalarraysaretwodimensionalarraysofnumberswhichpossesstheinteresting qualitythatbychoosinganytwocolumnsinthearrayyoureceiveanevendistributionofallthepair-wise combinationsofvaluesinthearray.OATSusestheconceptoforthogonalarraystodesigntestsforsoftware systems. Asampleorthogonalarray[L9(34)]hasbeenshowninTable5forreference. BestPractices LegacyApproach Randomized Subjective Coveragecannotbequantitativelymeasuredasthe dataisrandomized Cannotbeautomatedeasily Optimized Objective Coveragecanbequantitativelymeasured Greatcandidateforautomation! PairwiseApproach AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |7 Table4.Pairwiseapproachv/sLegacyapproach
  8. 8. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. Unfortunately,OATSbecomesineffectivewhenthesystem undertestincludesconstraintsbecausethereis nowayconstraintscanbefedintoorthogonalarrayswhilegeneratingtestscenarios;thiscausesOATSto generatetestswithinvalidpairs. AnotherdisadvantagewithOATSisthattheOrthogonalArrayscontainallpairsanequalnumberoftimes. Thatmeans,usingorthogonalarrayswillresultinmorenumberoftestscomparedtothepairwisestrategy becausetheyincludepairsequally.Thepairwisestrategycoversallthepairsinalessernumberofteststhan OATS. Thechallengewiththepairwiseteststrategyisthattherequirements(includingtheconstraints)shouldbe accurateandcomplete;rightthefirsttime.TherequirementsshouldNOTchangeoncethepairwisescenarios aregenerated;theentirepairwisematrixwillhavetobere-generatedifanyonerequirementchangespost thepairwisetestgeneration.Thisisbecauseeachrowitem inthepairwisematrixisunique,attemptingto coverasmanyuniquepairsaspossiblewiththeleastnumberoftests. AAdedicatedeffortisalsorequiredforgeneratingpairwisescenariosforeveryproductundertest,evenfor similarproducts;evenaslightchangeintherequirementsbetweentwoproductswillcallforseparate pairwisematrices. Anotherchallengeconcernstheselectionoftherighttooltobeused.Therearemanyfreeandpaidtools availableinthemarketandoneneedstoclearlydefinethetoolevaluationcriteriaforpickinguptherighttool. Theevaluationcriteriashouldconsider,butnotbelimitedto: CommonIssues AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |8 ControlFactors Expt.No 1 2 3 4 5 6 77 8 9 1 1 1 2 2 2 33 3 3 1 2 3 1 2 3 11 2 3 1 2 3 2 3 1 33 1 2 1 2 3 3 1 2 22 3 1 A B C D Table5.SampleOrthogonalArray
  9. 9. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved. Thebudgetallottedforthetestactivity,ifapaidtoolistobeused. Toolsupport;mostofthefreetoolswillNOThaveanytoolsupportprovidedalongwiththem. Thenumberofinputvariablesandstates. Thecomplexityoftheconstraintsthatcomewiththeinputdataandtheabilityofthetooltoaccom- modatetheseconstraintswhilegeneratingpairwisescenarios. Easeofuseintermsoffeedingthevariablesandconstraintsintothetool. SupportinimportingthegeneratedpairwisetestsintoaTestManagementtoolfortestexecution management. Thepairwisetestdesigntechniqueisveryeffectiveinprovidingoptimizedtestcoveragefortestingthe variableinteractionsofaprinterdriver.Withjust2800testscenariosderivedusingthepairwisetestdesign techniqueascomparedtoapproximately5000testcasesusingthelegacyapproach,thepairwisetechnique wasabletodetect19%moreofdefectsthantheonesthatweredetectedthroughthelegacyapproachwitha testexecutioneffortsavingsof70%. TheThepairwisetestdesigntechniquealsoenablesthetestteamstomeasuretestcoverageinaquantitativeway asopposedtousingotherstrategieslikepurerandomization,guessingand/orrandomizationusingopera- tionalprofiles. Adoptingthepairwisetestdesigntechniquefortestingthevariableinteractionsoftheprinterdriversignifi- cantlyreducestheexecutioneffort,detectsmoredefectswithlessnumberoftests,andprovidesuswitha quantitativewayofmeasuringtestcoverage.Thus,thepairwisetechniqueprovesitselftobethebestcombi- natorialtestdesigntechniquewithareducednumberoftestsandanimproveddefectdetectionrate. Conclusion AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |9
  10. 10. Formoredetailscontact:ers.info@hcl.com Followusontwitter:http://twitter.com/hclersand Ourbloghttp://www.hcltech.com/blogs/engineering-and-rd-services Visitourwebsite:http://www.hcltech.com/engineering-rd-services Hello,I’m from HCL’sEngineeringandR&DServices.Weenabletechnologyledorganizationstogotomarketwithinnovativeproducts andsolutions.Wepatnerwithourcustomersinbuildingworldclassproductsandcreatingassociatedsolutiondeliveryecosystems to help bringmarketleadership.Wedevelop engineeringproducts,solutionsand platformsacrossAerospaceand Defense, Automotive,ConsumerElectronics,Software,Online,IndustrialManufacturing,MedicalDevices,NetworkingandTelecom,Office Automation,SemiconductorandServers&Storageforourcustomers. ThiswhitepaperispublishedbyHCLEngineeringandR&DServices. Theviewsandopinionsinthisarticleareforinformationalpurposesonlyandshouldnotbeconsideredasasubstituteforprofessional businessadvice.TheusehereinofanytrademarksisnotanassertionofownershipofsuchtrademarksbyHCLnorintendedtoimply anyassociationbetweenHCLandlawfulownersofsuchtrademarks. FormoreinformationaboutHCLEngineeringandR&DServices, Pleasevisithttp://www.hcltech.com/engineering-rd-services Copyright@ HCCopyright@ HCLTechnologies Allrightsreserved. SwadishAaryan HCLEngineeringandR&DServices Reference AuthorInfo 1)http://www.pairwise.org/ 2)PairwiseTestingintheRealWorld: PracticalExtensionstoTest-CaseScenarios.(http://msdn.microsoft.com/en-us/library/cc150619.aspx#Pairwise_topic1) 3)JamesBachandPatrickJ.Schroeder,PairwiseTesting:ABestPracticeThatIsn’t 4)JacekCzerwonka,PICT3.3User’sGuide 5)PracticalCombinatorialTesting.SP800-142.(Report).Natl.Inst.ofStandardsandTechnology.2010. AdoptingthePairwiseTestDesignTechniquetoOptimizePrinterDriverTestCoverage |10

×