This summarizes a document describing an analogy-based defect prediction model using case-based reasoning. The model retrieves similar past cases from a case base of software modules to predict the number of defects in new modules based on their metrics. It pre-processes data from public NASA repositories, selects relevant metrics as features, and evaluates the model on a test set, achieving predictive performance comparable to other methods. The goal is to identify high-risk modules early to improve software quality.
13. Model Scalability: CBR systems are scalable to very large case libraries, and fast retrieval of cases continues to be of practical advantage. References<br />[Admodt et al. ‘94] <br />Aamodt A., Plaza E., Case-Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches, Artificial Intelligence Communications, vol. 7 (1), 1994. pp. 39-52.<br />[Aha ‘91] <br />Aha, D.W., Case-based learning algorithms, DARPA Case-Based Reasoning Workshop, Morgan Kaufmann, 1991.<br />[Catal et al. ‘08] <br />Catal C., Diri B., A systematic review of software fault prediction studies, Expert Systems with Applications, vol. 36, 2008, pp. 7346-7354.<br />[El Emam et al. ‘01] <br />El Emam K., Benlarbi S., Goel N., Rai S. N., Comparing case-based reasoning classifiers for predicting high risk software components, The Journal of Systems and Software, vol. 55, 2001, pp. 301-320.<br /> [Khoshgoftaar et al. ‘97] <br />Khoshgoftaar T. M., Ganesan K., Allen E. B., Ross F. D., Munikoti R., Goel N., Nandi A., Predicting fault-prone modules with case-based reasoning, 8th International Symposium on Software Reliability Engineering, IEEE Computer Society, 1997, pp. 27–35.<br />[Khoshgoftaar et al. ‘03] <br />Khoshgoftaar T.M., Seliya N., Analogy-based practical classification rules for software quality estimation, Empirical Software Engineering, vol 8, 2003, pp. 325–350.<br />[Kohavi ‘97] <br />Kohavi R., John G.H., Wrappers for feature selection for machine learning, Artificial Intelligence, 1997, pp. 273-324.<br />[Larose ‘05] <br />Daniel T. Larose, DISCOVERINGKNOWLEDGE IN DATA; an Introduction to Data Mining, Wiley, New Jersey, 2005, Chapter 7: Neural Networks, pp. 128-147. <br />[Li et al. ‘08] <br />Jingzhu Li, Ruhe G., Software effort estimation by analogy using attribute selection based on rough set analysis, International Journal of Software Engineering and Knowledge Engineering, Vol. 18 (1), 2008, pp. 1-23.<br />[Little et al. ‘02] <br />Little R. J. A., Rubin D.B., Statistical Analysis with Missing Data, 2nd edition, John Wiley & Sons, New York, 2002.<br />[Mahar et al. ‘06]<br />Mahar K., El Deraa A., software project estimation model using function point analysis with cbr support, IADIS International Conference Applied Computing, 2006, pp. 508-512.<br />[URL 1] URL: http://mdp.ivv.nasa.gov, Last Accessed on 4/12/2009.<br />[URL 2] URL: http://promisedata.org/, Last Accessed on 4/12/2009.<br />[URL 3] URL: http://en.wikipedia.org/wiki/Main_Page, Last Accessed on 4/12/2009.<br />
![Analogy Based Defect Prediction Model<br />Elham Paikari<br />Department of Electrical and Computer Engineering<br />Schulich School of Engineering<br />University of Calgary<br />epaikari@ucalgary.ca<br />Introduction<br />Software metrics-based quality prediction models can be effective tool for identifying the number of defects of the modules. The use of such models prior to each release planning or re-planning of the system, or even deployment of that can considerably improve system quality. A defect prediction model is calibrated using metrics from a past release or similar project, and is then applied to modules currently under development. Subsequently, a timely prediction of which modules needs more effort to remove the defects, can be obtained. <br />A software defect prediction model is composed of independent variables (software metrics) measured and collected during the software life-cycle, and a dependent variable (number of defects) recorded later in the life cycle. Initially, the model is calibrated based upon (training data) the independent variables and dependent variable of a previously developed system release or similar project. The calibrated model can then be used to predict the number of defects of software modules currently under development. Thus, if one supplies the software metrics for a module, one can predict what the number of defects is. Subsequently, the project management can decide which modules will be targeted for quality improvement activities, avoiding resources wasted on testing and improving the entire system. <br />The evaluative studies of CBR classifiers have been promising, showing that their predictive performance is as good as or better than other types of classifiers. However, CBR can be instantiated in different ways by varying its parameters, it is not clear which combination of parameters provides the best performance. But in this paper we evaluate the performance of CBR with a new way of attribute weighting.<br />Most notably advantage of a CBR in our context is that the detailed characterization of the similar cases can help one interpret the automatic results. In principle, as well, CBR would provide a straightforward approach for dealing with missing values. However, in the context of quality prediction using product metrics there are rarely missing values.<br />When evaluating the predictive performance of CBR, one first constructs a case base of previous components where the source code metrics and the defects are known. A different data set is then used as the test set. This can be achieved in a number of different ways, such as a holdout sample, cross-validation, bootstrapping, multiple releases, or random subsets. For our study we use random subsets.<br />In addition to constructing a CBR model, a sensitivity analysis with neural network is done to assign the weights of the parameters and a brief relative comparison of different techniques is also presented. <br />Defect Prediction<br />A similarity-based comparison principle for defect prediction improve software quality and testing efficiency by constructing predictive models from software attributes to enable a timely identification of number of defects of modules. <br />The delivery of large software development project is a complex undertaking. Large software projects are filled with uncertainty related to changing requirements, sufficiency of the architecture, and quality of the delivered source code. Improper estimations in this process result in unexpected cost, reduced delivered scope and late deliveries. Project managers have to take into account all the resource constraints, adjusting requirements, quality and timelines. They also have to anticipate and resolve bottlenecks arising from differing level of progress in different teams. One of these bottlenecks can be related to number of the defects in each module.<br />One of the most cost-effective strategies for software quality improvement is an approach of identifying, before system tests and operations, software modules that are likely to have more defects [Khoshgoftaar et al. ‘97]. Such identification can then be followed by targeting quality improvement on such high-risk modules. If faulty modules are detected or identified in a timely fashion during the development life-cycle, the number of defects likely to be detected during operations can be reduced. For example, if software quality estimation models are designed and utilized prior to system deployment, the number of defects discovered by customers may reduce significantly. Therefore, the outcome of a timely quality improvement would be a more reliable and easy to maintain system [Khoshgoftaar et al. ‘01].<br />The most important goal of constructing this defect prediction model is having a good estimation for Project Planning. It can be important for investigation of the ability to deliver the product successfully as predefined with a certain level of quality. Even before delivery, during the different releases of the software, for designing of high-level plans for intermediate targets of quality, with this model, re-planning can be done based on updated information with prediction.<br />Exploration of process improvement opportunities is another aspect of usefulness of this prediction model to estimate the number of defects, and then the risk analysis for the product would be more accurate. In this model, inputs are software metrics collected earlier in development, and output is a prediction of the number of faults that will be discovered later in development or during operations. Then it is worthy to use techniques for early prediction of software quality based on software complexity metrics because it can considerably reduce the likelihood of defects discovered during operations.<br />Case Based Reasoning<br />Reasoning by re-using past cases is a powerful and frequently applied way to solve problems for humans. This claim is also supported by results from cognitive psychological research. Part of the foundation for the case-based approach, is its psychological plausibility. Case-based reasoning and analogy are sometimes used as synonyms. Case-based reasoning can be considered a form of intra-domain analogy. <br />In CBR terminology, a case usually denotes a problem situation. A previously experienced situation, which has been captured and learned in a way that it can be reused in the solving of future problems, is referred to as a past case, previous case, stored case, or retained case. Correspondingly, a new case or unsolved case is the description of a new problem to be solved. Case-based reasoning is - in effect - a cyclic and integrated process of solving a problem, learning from this experience, solving a new problem, etc. <br />Case-based reasoning (CBR), broadly construed, is the process of solving new problems based on the solutions of similar past problems. An auto mechanic who fixes an engine by recalling another car that exhibited similar symptoms is using case-based reasoning. A lawyer who advocates a particular outcome in a trial based on legal precedents or a judge who creates case law is using case-based reasoning. So, too, an engineer copying working elements of nature (practicing biomimicry), is treating nature as a database of solutions to problems. Case-based reasoning is a prominent kind of analogy making.<br />It has been argued that case-based reasoning is not only a powerful method for computer reasoning, but also a pervasive behaviour in everyday human problem solving; or, more radically, that all reasoning is based on past cases personally experienced. This view is related to prototype theory, which is most deeply explored in cognitive science.<br />Process Steps<br />Case-based reasoning has been formalized for purposes of computer reasoning as a four-step process — sometimes referred to as the R4 model — that combine to make a cyclical process [Admodt et al. ‘94]:<br />Retrieve: Given a target problem, retrieve cases from memory that is relevant to solving it. A case consists of a problem, its solution, and, typically, annotations about how the solution was derived. <br />Reuse: Map the solution from the previous case to the target problem. This may involve adapting the solution as needed to fit the new situation. <br />Revise: Having mapped the previous solution to the target situation, test the new solution in the real world (or a simulation) and, if necessary, revise. <br />Retain: After the solution has been successfully adapted to the target problem, store the resulting experience as a new case in memory. <br />Figure 1 illustrates this cycle diagrammatically. Central is the case-base, which is a repository of completed cases, in other words the memory. When a new problem arises it must be codified in terms of the feature vector (or problem description) that is then the basis for retrieving similar cases from the case-base.<br />Figure 1 -The CBR Process [Mahar et al. ‘06]<br />Clearly, the greater the degree of overlap of features, the more effective the similarity measures and case retrieval. Ideally the feature vectors should be identical since CBR does not deal easily with missing values, although of course there are many data imputation techniques that might be explored [Little et al. ‘02]. Measuring similarity lies at the heart of CBR and many different measures have been proposed. Irrespective of the measure, the objective is to rank cases in decreasing order of similarity to the target and utilise the known solutions of the nearest k cases. <br />Solutions derived from the retrieved cases can then be adapted to better fit the target case either by rules, a human expert or by a simple statistical procedure such as a weighted mean. In the latter case the system is often referred to as k-nearest neighbour (k-NN) technique. Once the target case has been completed and the true solution known it can be retained in the case-base. In this way the case-base grows over time and new knowledge is added. Of course it is important not to neglect the maintenance of the case-base over time so as to prevent degradation in relevance and consistency.<br />Analogy-Based Reasoning<br />This term is sometimes used, as a synonym to case-based reasoning, to describe the typical case-based approach. However, it is also often used to characterize methods that solve new problems based on past cases from a different domain, while typical case-based methods focus on indexing and matching strategies for single-domain cases. Research on analogy reasoning is therefore a subfield concerned with mechanisms for identification and utilization of cross-domain analogies. The major focus of this study is on the reuse of a past case, what is called the mapping problem: Finding a way to transfer, or map, the solution of an identified analogue (called source or base) to the present problem (called target).<br />Why CBR?<br />CBR is argued to offer a number of advantages over many other knowledge management techniques, which are all important in defect prediction in software development, which is related to risk and reliability, too.<br />1. Avoids many of problems associated with knowledge elicitation and codification.<br />2. Only needs to address those problems that actually occur, whilst generative (i.e. algorithmic) systems must handle all possible problems.<br />3. Handles failed cases, which enable users to identify potentially high risk situations.<br />4. Copes with poorly understood domains (for example, many aspects of software engineering) since solutions are based upon what has actually happened as opposed to hypothesised models.<br />5. Supports better collaboration with users who are often more willing to accept solutions from analogy based systems since these are derived from a form of reasoning akin to human problem solving. <br />CBR for Defect Prediction<br />It has long been recognised that a major contribution to successful software engineering is the ability to be able to make effective predictions particularly in the realms of costs and quality. Consequently there has been significant research activity in this area, much of which has focused on defect prediction. This problem is characterised by an absence of theory, inconsistency and uncertainty that make it well suited to CBR approaches.<br />Data<br />For defect prediction, metrics property has method-level, class-level, component-level, file-level, process-level, and quantitative-level categories. Method-level metrics are widely used for software fault prediction problem. Halstead McCabe metrics have been proposed in 1970s but they are still the most popular method-level metrics. Method-level metrics can be collected for programs developed with structured programming or object-oriented programming paradigms because source code developed with these paradigms include methods. When method-level metrics are used, predicted number of defects of modules is helpful during system testing or field tests. Widely used method- level metrics are shown in REF _Ref247957045 Table 1. Distribution of metrics which have been used after year 2005 is shown in REF _Ref247957287 Figure 2.<br />Table 1 - Widely used method-level metrics<br />Figure 2 - Distribution of methods after year 2005<br />There are well established research communities in defect and effort prediction. A good model should be a generalization of real-world data. Collecting data from real world software engineering projects is problematic. Because software projects are notoriously difficult to control and corporations are often reluctant to expose their own software development record to public scrutiny. PROMISE (PRedictOr Models In Software Engineering) and The MDP (Metrics Data Program) are the most comprehensive Data Repositories and databases that store problem data, product data and metrics data [URL 1], [URL 2].<br />Software management decisions should be based on well-understood and well-supported predictive models. Then primary goal of the repository is to provide project non-specific data to the software community. And for this research we need datasets which include number of defects, too. <br />The other important aspect about data in this project is the metrics utilized for prediction. Metrics should be selected from different phases of Development. Because we will use the predicted number of defects after a complete lifecycle, defects, as the result of all of these phases would be matter. <br />Distribution of datasets <br />One of the most important problems for software defect prediction studies is the usage of non-public (private) datasets. Several companies developed defect prediction models using proprietary data, but it is not possible to compare results of such studies with results of other models because their datasets cannot be reached. Because machine learning researchers had similar problems in 1990s, they created a repository called UCI (University of California Irvine). Similar to UCI, Software Engineers have PROMISE repository which has several public datasets since 2005. NASA fault prediction datasets locate in PROMISE with a format (ARFF) that makes it possible to use these datasets directly from an open source machine learning tool called WEKA.<br />All the Research works have been divided into four categories regarding to their types of datasets: public, private, partial, and unknown. Public datasets mostly locate in PROMISE and NASA MDP (metrics data program) repositories and they are distributed freely. Private datasets belong to private companies and they are not distributed as public datasets. Partial datasets are datasets which have been created using data from open source projects and have not been distributed to community. These partial datasets do not locate in public repositories such as PROMISE. If there is no information about the dataset, the type of dataset is called ‘‘unknown” [Catal et al. ‘08]. Doubtless, the underlying characteristics of the problem data set are likely to exert a strong influence upon the relative effectiveness of different prediction systems.<br />From the available data in the NASA repository, which includes the number of defects, there are four types of data attribute. A list of related fields are shown in REF _Ref247890811 Table 2.<br />Table 2 - Available Data Fields from related Datasets<br />First type is coming from McCabe software metrics. Cyclomatic complexity (or conditional complexity) is software metric (measurement). It was developed by Thomas J. McCabe Sr. in 1976 and is used to measure the complexity of a program. It directly measures the number of linearly independent paths through a program's source code. The concept, although not the method, is somewhat similar to that of general test complexity measured by the Flesch-Kincaid Readability Test.<br />Mathematically, the cyclomatic complexity of a structured program is defined with reference to a directed graph containing the basic blocks of the program, with an edge between two basic blocks if control may pass from the first to the second (the control flow graph of the program). The complexity is then defined as [URL 3]:<br />M=E-N+2p<br />Where:<br />,[object Object]](https://image.slidesharecdn.com/analogy-based-defect-prediction-model-elham-paikari-department-of1392/85/Analogy-Based-Defect-Prediction-Model-Elham-Paikari-Department-of-1-320.jpg)
