Managing The Business Risk Of Fraud
•Download as PPT, PDF•
7 likes•1,331 views
This document provides an agenda for a presentation on managing the business risk of fraud using sampling and data mining techniques. The presentation will cover understanding fraud risk management frameworks, performing statistical sampling for audits, using data mining and analytics to introduce efficiency to the audit process without losing effectiveness, and applying various sampling and analytic techniques like regression analysis. The objectives are to understand fraud risk management, apply sampling methods, and use data mining to reduce audit costs and effectively detect fraud.
![PWC Global Survey – Nov, 2009 “ Economic crime in a downturn ” ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (16)
Viewers also liked
Viewers also liked (11)
Similar to Managing The Business Risk Of Fraud
Similar to Managing The Business Risk Of Fraud (20)
Recently uploaded
Recently uploaded (20)
Managing The Business Risk Of Fraud
- 1. Managing the Business Risk of Fraud using Sampling and Data Mining Fall 2009 Mike Blakley Presented to:
- 13. Statistics based “ roach ” hunting Many frauds coulda/woulda/shoulda been detected with analytics
- 22. Scatter plot with prediction and confidence intervals
- 23. Cost of six types of AIDS drugs Example 2 Objective 1a
- 25. Pareto Chart Objective 1
- 26. Typical Prescription Patterns Example 2 Objective 1a
- 27. Prescriptions by Dr. X Example 2 Objective 1a
- 29. Revenue trends Example 3 Objective 1a
- 30. Dental Billings Example 3 Objective 1a
- 53. Peeling the Onion Objective 1c Population as Whole Possible Error Conditions Fraud Items
- 54. Fraud Pattern Detection Objective 1d Market Basket Stratification Trend Line Holiday Day of Week Duplicates Univariate Gaps Benford’s Law Round Numbers Target Group
- 68. Law of large numbers
- 71. Central limit theorem Classic measure Mean of a sufficiently large number of random samples will be approximately normally distributed.
- 76. Industrial revolution Manufacturing Engineering Excitement!
- 77. Student’s T William Gosset - 1908 Guinness Brewery
- 82. Attribute Sampling Using RAT-STATS Unrestricted populations
- 94. Variable Sampling Using RAT-STATS Unrestricted populations
- 107. Attribute Sampling Using RAT-STATS Stratified populations
- 118. Variable Sampling Using RAT-STATS Stratified populations
- 137. Yellow book standards Standards regarding statistical sampling and IT
- 144. Trend Busters Does the pattern make sense? 7 - Trends
- 149. Trend Lines by Account - Example Results Generally the trend is gently sloping up, but two accounts (43870 and 54630) are different . 7 – Trends 1.89 -0.560 56 54630 2.36 0.340 23 43870 0.36 1.010 32 43517 0.85 1.023 27 32451 4.3 1.070 17 43517 0.87 1.230 18 32451 Std Err Slope N Account
- 150. Scatter plot with prediction and confidence intervals
- 163. Compute Cumulative Amount for each Objective 2d
- 169. Kolmogorov-Smirnov Objective 2d - KS
- 174. It’s that time! Session Break!
- 176. Trapping anomalies Objective 3
- 177. Fraud Pattern Detection Objective 3 Market Basket Stratification Trend Line Holiday Day of Week Duplicates Univariate Gaps Benford’s Law Round Numbers Target Group
- 178. Outliers / Variability Outliers are amounts which are significantly different from the rest of the population 1 - Outliers
- 181. Example Results Two providers (3478421 and 2356721) had significantly more variability in the amounts of their claims than all the rest. 1 - Outliers 18.54 2,311 5463122 23.25 3,421 3546789 87.23 4,536 2356721 342.23 3,243 3478421 Coeff Var N Provider
- 183. Unusual stratification patterns Do you know how your data looks? 2 - Stratification
- 187. Units of Service Stratified - Example Results Two providers (2735211 and 4562134) are shown to be much different from the overall population (as measured by Chi Square). 2 – Stratification 0.4632 298 2,503 4237869 0.5231 342 3,410 4321089 0.7453 5,234 8,913 4562134 0.8453 7,453 6,011 2735211 D-stat Chi Sq N Provider
- 192. Day of Week - Example Results Provider 2735211 only provided service for Medicaid on Tuesdays. Provider 4562134 was closed on Thursdays and Fridays. 3 – Day of Week 0.2189 56 7,905 4237869 0.321 87 5,162 4321089 0.8472 7,746 5,182 4562134 0.9802 12,435 5,404 2735211 D-stat Chi Sq N Provider
- 194. Round Numbers It’s about…. Estimates! 4 – Round Numbers
- 197. Round Numbers in Journal Entries - Example Results Two accounts, 2735211 and 4562134 have significantly more round number postings than any other posting account in the journal entries. 4 – Round Numbers 0.2189 546 9,549 4237869 0.321 768 8,318 4321089 0.97023 35,324 833 4562134 0.9802 54,637 4,136 2735211 D-stat Chi Sq N Account
- 199. Made up Numbers Curb stoning Imaginary numbers Benford’s Law 5 – Made up numbers
- 204. Invoice Amounts tested with Benford’s law - Example Results During tests of invoices by store, two stores, 324 and 563 have significantly more differences than any other store as measured by Benford’s Law. 5 – Made Up Numbers 0.2189 312 74 217 0.321 476 23 432 0.97023 4,735 89 563 0.9802 5,234 79 324 D-stat Chi Sq Hi Digit Store
- 212. Numeric Sequence Gaps What’s there is interesting, what’s not there is critical … 8 - Gaps
- 216. Gap Testing - Example Results Four check numbers are missing . 8 – Gaps 1 17548 17546 2 12526 12523 1 10791 10789 Missing End Start
- 218. Duplicates Why is there more than one? Same, Same, Same, and Same, Same, Different 9 - Duplicates
- 223. Possible Duplicates - Example Results Five invoices may be duplicates . 9 - Duplicates 2 1,500.00 2/12/2007 17546 2 2,010.37 8/31/2007 10245 4 3,544.78 6/15/2007 10245 Count Invoice Amount Invoice Date Vendor
- 225. Date Checking If we’re closed, why is there … Adjusting journal entry? Receiving report? Payment issued? 10 - Dates
- 234. Holiday Counts - Example Results Two employees (10245 and 32325) were “ off the chart ” in terms of expense amounts incurred on a Federal Holiday . 10 – Dates 0.2189 312 34 24135 0.321 476 18 17546 0.97023 4,735 23 32325 0.9802 5,234 37 10245 D-stat Chi Sq N Employee Number
- 255. Questions?
Editor's Notes
- Good morning.
- As I mentioned before, my overall objective is to provide each of you sufficient information to determine if the approach I’m going to suggest might make sense in your organization. To do that, I have five specific objectives. First I want you to understand why a systematic approach is suggested and how such an approach might be implemented. I have also borrowed heavily from other statistical approaches and concepts. Even though it isn’t necessary for you to understand the down and dirty details of the specific concepts, I want to at least outline them in order that you have some comfort in what we are discussing. Then I’ going to suggest 10 specific tools and techniques which I believe can and should be put into place for the detection of fraud patterns. Finally I’m going to go over a little bit how Excel can be used and also to provide a general picture of how I think systematic fraud pattern detection fits into the overall scheme of what auditors are doing.
- This morning I’m going to describe some of the types of fraud patterns which can be detected using digital analysis, as well as describe a methodical approach for fraud pattern detection. With each type of pattern I will mention a few examples of fraud patterns, as well as how this could fit into continuous monitoring and the type pf business analytics which are possible. As you know, Excel is widely used by auditors, so I will also try to mention a little bit about how excel can be used to assist in this process.
- I want to start with a practical application and illustration as to how the concepts I’m going to be discussing this morning can be applied in a real world situation. This chart is derived from information in the Medicaid system which is a government program to provide medical services to the poor. The data shown here is based loosely on an actual situation which came to light. The chart here shows total annual expenditures for six types of drugs which were prescribed to Aids patients. These amounts are statewide totals for these specific drugs.
- If you look into the details of drug number 4 you find out that it is used to treat Aids wasting syndrome, which is a side effect of AIDS where a significant weight loss occurs. It turns out that this particular drug has an off label use – i.e. a use which has not been approved by the FDA. In this case this drug was also used by body builders to build up their body weight. The statistics on this particular physician was that he had prescribed over $11 million worth of this drug – all paid by taxpayers. The amount that this one physician prescribed accounted for 12% of the entire prescriptions for this type of drug for the state. One of the analytical approaches that we’re going to be discussing this morning would hone in on this type of situation even if the auditor was not aware of any of the details.
- Now if you were to look at six specific physicians or physician groups and look at their prescription pattern for these six Aids drugs, you would likely see a pattern like this. In other words, their pattern of prescription doesn’t vary that much from that of the population as a whole.
- But if you were to look at one specific provider, we’ll call him Dr. X, you see a very different pattern. Look at drug number four. You see that Dr. X prescribes a very large dollar amount in comparison to the population as a whole. There is also some difference for the other five drugs – as you can see he prescribes a lesser dollar amount for these five drugs when compared with the population as a whole.
- If you look into the details of drug number 4 you find out that it is used to treat Aids wasting syndrome, which is a side effect of AIDS where a significant weight loss occurs. It turns out that this particular drug has an off label use – i.e. a use which has not been approved by the FDA. In this case this drug was also used by body builders to build up their body weight. The statistics on this particular physician was that he had prescribed over $11 million worth of this drug – all paid by taxpayers. The amount that this one physician prescribed accounted for 12% of the entire prescriptions for this type of drug for the state. One of the analytical approaches that we’re going to be discussing this morning would hone in on this type of situation even if the auditor was not aware of any of the details.
- This next situation is also based on an actual case. This chart shows the billing trends overall for dentists in this particular state. As you can see there is a fairly gently sloping upwards trend of about 7% for the three years 2001 through 2003. The important point to note here is the fairly gently sloping trend overall.
- For this same period, had you looked at the trends for two dentists in particular you would see a very different trend – this is almost an explosion in billings. One dentists went from zero to over 4 million dollars per year within a three year period. Another dentist had billings shoot up from about $100,000 per year to over 1.5 million. The dentist with the billings over 4 million exceeded that of any of the pother 50,000 dentists and the closes one was a group of dentists which billed about $1 million per year less than this dentist. What this dentists was doing was giving music CDs to any kids who would provide a valid Medicaid id. It’s not sure if they received all the services billed. For example she billed for as many as 900 procedures per day which is quite a bit for a single dentists. One of the techniques I’m going to talk about today is the use of trends to identify and isolate potential fraud patterns. I believe that the approach I’m going to discuss here today would have identified all these patterns as possible fraud areas.
- This morning I’m going to speak about one approach to the identification of fraud patterns using computers. There are many possible approaches and I think that auditors should have as many tools in the toolkit as feasible. The approach I’m going to suggest is based loosely on the Guidance paper which is a joint effort of the IIA, AICPA and the ACFE, which is titled managing the Business Risk of Fraud: A Practical Guide. The guidance paper came out last November and I’m told that the final version will be available later this month. This guide lays out five principles for the management of fraud, one of which is fraud detection. The paper also recommends that a coordinated investigation approach be used for fraud detection.
- This guidance paper is an excellent document, very comprehensive and fairly easy to read. It resulted from a joint effort by the IIA, AICPA and ACFE, and there should be a final version out by the end of this month. One of the most important sections of the guide was section 5 – which is dedicated to the discussion of fraud detection.
- In the section on fraud detection, the guide mentions five areas of emphasis. All of these are important, but the topic we’re going to be discussing today is proactive fraud detection. Obviously if you can detect possible fraud or errors early you’re going to get a bigger payoff than if you detect it later on.
- The guidance paper defines and describes proactive fraud detection as the process of using data analysis to identify anomalies, trends and other risk indicators. Notice that the key term here is data analysis. If you have a relatively low volume of transactions it is possible to simply eyeball them and possibly spot some unusual patterns. But once you start to get into the thousands or hundreds of thousands of transactions, this is no longer feasible. You need to have some sort of data analysis being performed as part of an automated process.
- The guide mentions specific areas which should be considered as part of the identification process. These include journal entries, relationships between data elements, use of benford’s law and performance of continuous monitoring. I’m going to include each of these areas in today’s discussion, as well as a host of other areas.
- The guidance paper notes that data analysis improves the ability to detect fraud due to various factors. For example, with data analysis it may be possible to identify hidden relationships, spotlight suspicious transactions and in general make an assessment of the effectiveness of internal controls. Probably one of the reasons for this is the ability to look at 100% of the transactions instead of just selecting a sample. It is also very feasible to look at literally millions of transactions of potential error situations, even using a desktop PC. I’m going to touch very briefly on techniques for having a desktop system process very heavy loads of transactions.
- One of the key objectives of fraud pattern identification is to focus or identify fraud or error items that may exist in a population. Obviously in most populations, the number of these items may be relatively small in comparison to that of the population as a whole. Also, fraud items are just one component of error items which are also generally of interest to the auditor. The techniques I’m going to discuss this morning will apply to both errors and fraud items.
- This morning I’m also going to focus on ten metrics or assessment techniques that I believe the auditor can and should apply as part of the fraud pattern detection process. Later in this session we’re going to go into detail on each of these ten areas, which I call the “Auditors Top 10”. Just briefly, each of these metrics can provide insight into the detection of unusual patterns or suspicious transactions.
- I want to now go briefly over the basics of digital analysis. Because in order to determine if digital analysis might be suitable in your organization, you will need an overview of how it might benefit your organization.
- Within audit organizations, digital analysis has traditionally been performed only by IT auditors or IT specialists. Some software tools provide the ability for audit generalists to also perform digital analysis. However, it has been estimated that in the typical audit organization, only about 5 – 10% of the auditors will make any significant use of computer assisted audit software. Today there is a growing trend for businesses to make more and more use of business analytics. This trend has also spilled over into the audit area.
- The basic functions of digital analysis software is to classify, quantify and compare both numeric and non-numeric data. These are the building blocks upon which more complex analyzes are built. For example classification, might include data stratification, ageing, frequency distributions etc. Quantification might look at outliers, averages, etc. Comparisons can include such items as actual versus expected or reasonableness testing.
- The basic concept of the approach I’m going to discuss today is to assess fraud risk based strictly on quantification. Basically, software can and should do a lot of the “leg work” involved in obtaining the measurements. Then various statistical measures can be used to quantify the differences. Examples of types of statistical measures include Chi Square, Kolmogorov Smirnov and the d-statistic. We’ll get into each of these later.
- What are the advantages of using digital analysis? First, it can be automated, which means it becomes feasible to handle large data populations. The process I’m going to describe will produce objective, quantifiable metrics, and it isn’t necessary that the auditor have in-depth knowledge of the data being audited going in. This process is also adaptable to continuous monitoring and can also be used to produce business analytics. Business analytics will be a natural by product of the process. But probably on one of the biggest advantages of digital analysis is simply that 100% testing becomes possible. Also, as part of the 100% testing, generally a range of tests can be performed, including “impossible” conditions
- But digital analysis does have some disadvantages, although I believe that often they can be overcome. Generally one of the largest costs will simply be the auditors time in developing and establishing an audit system. And of course there may be software licensing costs unless you are already have some software that can be used for this purpose. You often hear that a particular piece of software is powerful yet doesn’t have a significant learning curve. I tend to be skeptical about that, and I think that you need to be able to devote some tie resources to learning any particular powerful software package. Generally digital analysis will also require some specialized knowledge as well.
- Historically auditors have used data analysis as a one off each time a particular area is being audited. In other words, an audit area may be on an audit cycle and when the area is audited, data analysis procedures will be developed specifically for that area and intended to test weaknesses in a specific area. When the audit is concluded then the software procedures are put back on the shelf until the area is looked at again. However, the trend is to go towards continuous monitoring, i.e. have a process in place that looks at all sorts of transactions on an ongoing basis. In order for continuous monitoring to be comprehensive, it needs to be performed on a computer schedule, whether that is done on a mainframe or all the way down to a server or desktop. I have seen organizations start out by putting some of their processing on desktop systems which can be run overnight or on weekends. By doing this it is possible to perform a substantial amount of processing without making a large investment. Also, this can be a cost effective way to perform a significant amount of analysis without making a large investment.
- Digital analysis can be used beneficially in almost any organization that has data already available in digital format. Digital analysis becomes practically essential if the volume of data is large and or the data structure is complex. Digital analysis is also essential if there is a potential for fraud and an organized approach must be taken to its identification. I have worked for the State of North Carolina in their Medicaid program which as you know is a government program to provide medical services for the poor. This program has a large volume of transactions, something over $8 billion a year, for 1 ½ million participants and services provided by over 50,000 medical service providers. North Carolina has a dedicated staff of over 60 devoted to detecting and handling fraud waste and abuse in the Medicaid system.
- So our first objective was to describe to you the how and why of digital analysis in order to provide an overview of the process sufficient that you might be able to decide whether digital analysis might make sense in your organization and some of the alternatives that are available. In the next section I’m going to talk about how to use a methodical, objective approach using various statistical techniques.
- Upon completing this module on attribute sampling you will understand what the term attribute sampling is and when to apply it. You will understand the meaning of the term “unrestricted populations”. You will have an overview of how the process is done using RAT-STATS. Also, although it is not necessary to understand the computations behind calculating sample size, you will see the formula used in Excel.
- Often during an audit, one of the audit objectives is to determine that compliance with a policy. It is expected that there may be instances of non-compliance with the policy and the auditor will generally establish what are the upper limits of acceptable rates of non-compliance. Generally this will depend upon how critical the need for compliance is, as well as other factors such as mitigating controls. What the auditor needs to determine is an estimate for the rate of compliance within the population as a whole, but without examining every item in the population, as often this may be either difficult, costly or simply unachievable. Common types of attributes which might be tested by the auditor include items such as signatures, approvals, attachment of receipts or other supporting documents, test footing of amounts, etc.
- Rather than an examination of 100% of the population being tested, which is often neither practical nor possible in all cases, the recommended approach is to take a statistically valid random sample and then extrapolate the results of the test to the entire population using statistics. Because the number of transactions or items examined is generally small in comparison with the overall population, this process is economical and efficient. The first step is the determination of the required sample size. On the next slide we will go over the sampling process as a whole.
- The required sample size depends upon what the specific sampling objective is. There are two key elements to be taken into consideration. The first is the confidence level expressed as percentage between 0 and 99.999%. The confidence level provides the auditor a quantified measure of the likelihood that the range result from the test in fact actually contains the “true” population error rate. For example, let’s suppose that each paid invoice is supposed to have attached an original invoice which has been signed by a person reviewing the invoice to indicate their approval for payment. Further lets suppose that there are exactly 1,000 such invoices and that in fact there are exactly 20 invoices out of these 1,000 invoices which do not have the required approval. In other words a 2% error rate. Suppose that the auditor selects a sample and that based on the audit sample the population error rate is estimated at being between 1% and 5%. In this case, the conclusion is in fact correct, because the population error rate is actually 2%. But what if, but pure chance, the audit sample has an error rate of 4% and the auditor determines that the population error rate is between 3% and 5%. This is in fact an invalid conclusion. So by the selection of a confidence level, the auditor has some degree of assurance that the sample conclusions are valid. For example, if the auditor selects a confidence level of 95%, then on average, the audit conclusion will be correct in 19 out of 20 samples (i.e. 95%). There is only a five percent chance that the conclusion is incorrect. The precision, on the other hand is the desired range within which the auditor wishes to draw a conclusion. For example if the auditor expects that the error rate will be 3% but could actually range from 1% to 5%, then the precision amount will be 4%, the difference between the high value of 5% and the low value of 1%. Once the auditor has determined the required sample size, a set of random numbers will be generated in order to select the sample. The sample will be pulled and the items tested. Then the results of the test, which will be a count of the number of instances where an error or other attribute was identified, and then the statistical results of the sample can be computed.
- To do all of this in RAT-STATS, the starting point is for the auditor to first develop the sampling objectives, i.e. specify what the expected error rate is, what the desired level of confidence is and what precision level is required. This information, together with the overall sample size is then entered into RAT-STATS, which computes the required sample size. Often the auditor will wish to specify that a variety of confidence levels be assumed in order that the auditor can determine which of a range of sample sizes seems practical. Based on this information, the auditor can then generate random numbers for the purpose of sample selection, then pull the sample and test the items. Once all the tests have been performed, the results can be summarized and entered into RAT-STATS which will then make the statistical computations as to what the population attribute rate is.
- So for the fist step, the audit needs to determine four items; the first is the size of the population, i.e. the number of items to be tested. The second is the expected error rate. This amount could be based on the results obtained in a prior audit. If the expected error rate is unknown, then the most conservative value, 50%, can be entered. The required confidence level should then be specified. Depending upon how critical the audit test is and whether or not there are mitigating or other compensating controls, this amount can range from as low as 50% or less up to 99.99% or more, all based upon the auditor's judgment. The required precision is also a judgmental value and is determined almost entirely using the auditors judgment. Note that there is an inverse relationship between the confidence level and the precision, assuming that the sample size is kept constant. As the required confidence level increases, so too does the precision level. Similarly, as the required confidence level decreases, so too will the precision level achieved.
- Obtaining random numbers using RAT-STARS is done by specifying various parameters, such as starting number, ending number, quantity of random numbers etc. All of this information is entered into RAT-STATS which can select the numbers randomly and output the results in a variety of formats. These formats include producing a text file, creating an excel workbook, creating a Microsoft Access database, creating a print file or writing the results directly to a printer.
- Once all the random numbers have been obtained it is then possible to pull a sample using the random numbers generated. For example, in the demonstration coming up, you will see that the invoices to be tested are contained on rows in an excel workbook. Each row has a corresponding row number which might range from 8 to 5000, with 8 being the first row with invoice information.
- Generally the most time consuming part of the process will be in the actual test of the item or document. This generally involves obtaining and reviewing the document for the attribute being tested. The auditor should keep a tally or worksheet of all the items tested and the e results of the tests.
- Once all the testing has been completed, the auditor will summarize the results, i.e. count the number of instances the attribute was being tested and enter this information into RAT-STATS. RAT-STATS can then compute the results using statistics and can output the results in a variety of formats such as an excel workbook, Microsoft Access database, text file, print file or directly to a printer.
- In the next session we will show an actual example of usage of the RAT-STATS software to make a random selection from a population of 5,000 invoices and arrive at a statistically valid conclusion as to the error or attribute rate in the population. In this workbook, the results of each attribute test are stored in the worksheet. This is done to expedite the testing, as the results of each test are already recorded for every item (invoice) in the population.
- Upon completing this module on attribute sampling you will understand what the term attribute sampling is and when to apply it. You will understand the meaning of the term “unrestricted populations”. You will have an overview of how the process is done using RAT-STATS. Also, although it is not necessary to understand the computations behind calculating sample size, you will see the formula used in Excel.
- By variable is meant a numeric amount which can vary depending upon a record that is examined. For example, the auditor may be examining a collection of invoices, each of which has an invoice total amount. The invoice total amount would be referred to as a variable for the purposes of sampling. Often an audit objective is to determine the total value of a group of amounts, such as a collection of invoices. If the number of invoices is relatively small, it may be possible for the auditor to examine all of the transactions. However, the more typical case is that the number of transactions or amounts is simply too large to be examined on a 100% basis.
- Rather than an examination of 100% of the population being tested, which is often neither practical nor possible in all cases, the recommended approach is to take a statistically valid random sample and then extrapolate the results of the test to the entire population using statistics. Because the number of transactions or items examined is generally small in comparison with the overall population, this process is economical and efficient. The first step is the determination of the required sample size. On the next slide we will go over the sampling process as a whole.
- The required sample size depends upon what the specific sampling objective is. There are two key elements to be taken into consideration. The first is the confidence level expressed as percentage between 0 and 99.999%. The confidence level provides the auditor a quantified measure of the likelihood that the range resulting from the test in fact actually contains the “true” population value. For example, let’s assume that the auditor is attempting to quantify the amount of numeric errors contained in a population of invoices. The amount of the numeric error is determined by extending each line item on the invoice, then footing the amount and subtracting the amount shown on the invoice. If the invoice calculation is mathematically correct, then this amount will be zero, otherwise it will be some amount, either positive or negative. RAT-STATS uses the terms examined values to refer to the amounts being tested and audited values to be the amounts that the auditor has determined are in fact correct.
- To do all of this in RAT-STATS, the starting point is for the auditor to first develop the sampling objectives, i.e. specify what the expected error rate is, what the desired level of confidence is and what precision level is required. This information, together with the overall sample size is then entered into RAT-STATS, which computes the required sample size. Often the auditor will wish to specify that a variety of confidence levels be assumed in order that the auditor can determine which of a range of sample sizes seems practical. Based on this information, the auditor can then generate random numbers for the purpose of sample selection, then pull the sample and test the items. Once all the tests have been performed, the results can be summarized and entered into RAT-STATS which will then make the statistical computations as to what the population attribute rate is.
- Sampling parameters can be determined using various methods. One method is a probe sample where RAT-STATS is used to read the data values, compute the average and standard deviation which are used to develop the required sample size. This same calculation can also be done using Excel formula or using other statistical software. The sample mean and standard deviation can also be computed using Microsoft Access.
- To develop sampling parameters, the audit needs to determine three items; the first is the size of the population, i.e. the number of items to be tested. The second is the average item value. And the last is the standard deviation, which is a measure of the variability of the data. RAT-STATS makes the assumption that any errors will have a zero value and this assumption is therefore conservative if the amounts are simply overstated or misstated.
- Obtaining random numbers using RAT-STARS is done by specifying various parameters, such as starting number, ending number, quantity of random numbers etc. All of this information is entered into RAT-STATS which can select the numbers randomly and output the results in a variety of formats. These formats include producing a text file, creating an excel workbook, creating a Microsoft Access database, creating a print file or writing the results directly to a printer.
- Once all the random numbers have been obtained it is then possible to pull a sample using the random numbers generated. For example, in the demonstration coming up, you will see that the invoices to be tested are contained on rows in an excel workbook. Each row has a corresponding row number which might range from 2 to 5002, with 2 being the first row with invoice information.
- Generally the most time consuming part of the process will be in the actual test of the item or document. This generally involves obtaining and reviewing the document for the amount being tested. The auditor should keep a tally or worksheet of all the items tested and thee results of the tests. This data will then be put into a file and read into RAT-STATS in order to make the computation as to the overall population values.
- Once all the testing has been completed, the auditor will summarize the results, i.e. enter both the examined value and the audited value into RAT-STATS. RAT-STATS can then compute the results using statistics and can output the results in a variety of formats such as an excel workbook, Microsoft Access database, text file, print file or directly to a printer.
- In the next session we will show an actual example of usage of the RAT-STATS software to make a random selection from a population of 5,000 invoices and arrive at a statistically valid conclusion as to the total value or amount of a population. In this workbook, both the examined values as well as the audited values are stored in the worksheet. This is done to expedite the testing, as the results of each test are already recorded for every item (invoice) in the population.
- Upon completing this module on stratified sampling of attributes you will understand what the term stratified sampling is and when to apply it. You will have an overview of how the process is done using RAT-STATS.
- Stratified sampling begins with dividing the population being tested into “strata” which are collections of items though tto have similar characteristics for the purpose of attribute sampling. For example, suppose that compliance with invoice authorization procedures was though to vary somewhat, but generally be consistent within any specific department. For example, some departments might have consistently high rates of compliance with the procedure and others not. In such as case, it may be more efficient to divide the population into strata in order to able to use a smaller overall sample size. The reason that a smaller sample size can be used is the assumption that there will tend to be some consistency within each strata tested.
- The starting point for stratified attribute sampling is obviously the division of the population to be tested into strata. The selection of strata is entirely up to the auditor performing the test and could consist of any of a variety of criteria, such as range of dollar amount, geographical location, sales region, etc. Once the strata have been selected, then the remainder of the process is no different than that of an unrestricted population.
- To do all of this in RAT-STATS, the starting point is for the auditor to first develop the sampling objectives, i.e. specify what the expected error rate is, what the desired level of confidence is and what precision level is required. This information, together with the overall sample size is then entered into RAT-STATS, which computes the required sample size. Often the auditor will wish to specify that a variety of confidence levels be assumed in order that the auditor can determine which of a range of sample sizes seems practical. Based on this information, the auditor can then generate random numbers for the purpose of sample selection, then pull the sample and test the items. Once all the tests have been performed, the results can be summarized and entered into RAT-STATS which will then make the statistical computations as to what the population attribute rate is.
- So for the fist step, the audit needs to determine four items; the first is the size of the population, i.e. the number of items to be tested. The second is the expected error rate. This amount could be based on the results obtained in a prior audit. If the expected error rate is unknown, then the most conservative value, 50%, can be entered. The required confidence level should then be specified. Depending upon how critical the audit test is and whether or not there are mitigating or other compensating controls, this amount can range from as low as 50% or less up to 99.99% or more, all based upon the auditor's judgment. The required precision is also a judgmental value and is determined almost entirely using the auditors judgment. Note that there is an inverse relationship between the confidence level and the precision, assuming that the sample size is kept constant. As the required confidence level increases, so too does the precision level. Similarly, as the required confidence level decreases, so too will the precision level achieved.
- Obtaining random numbers using RAT-STARS is done by specifying various parameters, such as starting number, ending number, quantity of random numbers etc. All of this information is entered into RAT-STATS which can select the numbers randomly and output the results in a variety of formats. These formats include producing a text file, creating an excel workbook, creating a Microsoft Access database, creating a print file or writing the results directly to a printer.
- Once all the random numbers have been obtained it is then possible to pull a sample using the random numbers generated. For example, in the demonstration coming up, you will see that the invoices to be tested are contained on rows in an excel workbook. Each row has a corresponding row number which might range from 8 to 5000, with 8 being the first row with invoice information.
- Generally the most time consuming part of the process will be in the actual test of the item or document. This generally involves obtaining and reviewing the document for the attribute being tested. The auditor should keep a tally or worksheet of all the items tested and the e results of the tests.
- Once all the testing has been completed, the auditor will summarize the results, i.e. count the number of instances the attribute was being tested and enter this information into RAT-STATS. RAT-STATS can then compute the results using statistics and can output the results in a variety of formats such as an excel workbook, Microsoft Access database, text file, print file or directly to a printer.
- In the next session we will show an actual example of usage of the RAT-STATS software to make a random selection from a population of 5,000 invoices and arrive at a statistically valid conclusion as to the error or attribute rate in the population. In this workbook, the results of each attribute test are stored in the worksheet. This is done to expedite the testing, as the results of each test are already recorded for every item (invoice) in the population.
- Upon completing this module on attribute sampling you will understand what the term attribute sampling is and when to apply it. You will understand the meaning of the term “unrestricted populations”. You will have an overview of how the process is done using RAT-STATS. Also, although it is not necessary to understand the computations behind calculating sample size, you will see the formula used in Excel.
- In stratified variable sampling, the auditor divides the population being tested into strata or groups with the hope that the amounts within each group will have lesser variability. The less variability encountered, then the more precise the sampling results obtained will be. In certain populations, there may be logical divisions which should result in reduced variability. For example, transactions by specified geographical location or store may have a tendency to be more uniform than if they were examined on a national or international basis.
- The process for taking a sample based upon a stratified population is very similar to that of taking a sample that is not stratified. The only obvious difference is that the population must first be stratified using some criteria specified by the auditor. The stratification may be based on examined value ranges, geographical location, etc. The required sample size depends upon what the specific sampling objective is. There are two key elements to be taken into consideration. The first is the confidence level expressed as percentage between 0 and 99.999%. The confidence level provides the auditor a quantified measure of the likelihood that the range resulting from the test in fact actually contains the “true” population value. For example, let’s assume that the auditor is attempting to quantify the amount of numeric errors contained in a population of invoices. The amount of the numeric error is determined by extending each line item on the invoice, then footing the amount and subtracting the amount shown on the invoice. If the invoice calculation is mathematically correct, then this amount will be zero, otherwise it will be some amount, either positive or negative. RAT-STATS uses the terms examined values to refer to the amounts being tested and audited values to be the amounts that the auditor has determined are in fact correct.
- To do all of this in RAT-STATS, the starting point is for the auditor to first develop the sampling objectives, i.e. specify what the expected error rate is, what the desired level of confidence is and what precision level is required. This information, together with the overall sample size is then entered into RAT-STATS, which computes the required sample size. Often the auditor will wish to specify that a variety of confidence levels be assumed in order that the auditor can determine which of a range of sample sizes seems practical. Based on this information, the auditor can then generate random numbers for the purpose of sample selection, then pull the sample and test the items. Once all the tests have been performed, the results can be summarized and entered into RAT-STATS which will then make the statistical computations as to what the population attribute rate is.
- Sampling parameters can be determined using various methods. One method is a probe sample where RAT-STATS is used to read the data values, compute the average and standard deviation which are used to develop the required sample size. This same calculation can also be done using Excel formula or using other statistical software. The sample mean and standard deviation can also be computed using Microsoft Access.
- To develop sampling parameters, the audit needs to determine four items; the first is the number of strata, i.e. the number of groups the population has been subdivided into. The second is the size of the population within each of the strata, i.e. the number of items to be tested. The third is the average item value. And the last is the standard deviation, which is a measure of the variability of the data. RAT-STATS makes the assumption that any errors will have a zero value and this assumption is therefore conservative if the amounts are simply overstated or misstated.
- Obtaining random numbers using RAT-STARS is done by specifying various parameters, such as starting number, ending number, quantity of random numbers etc. Random numbers to be used in RAT-STATS for stratified sampling need to be obtained using a “multi-stage” random sample. All of this information is entered into RAT-STATS which can select the numbers randomly and output the results in a variety of formats. These formats include producing a text file, creating an excel workbook, creating a Microsoft Access database, creating a print file or writing the results directly to a printer.
- Once all the random numbers have been obtained it is then possible to pull a sample using the random numbers generated. For example, in the demonstration coming up, you will see that the invoices to be tested are contained on rows in an excel workbook. Each row has a corresponding row number which might range from 2 to 5002, with 2 being the first row with invoice information.
- Generally the most time consuming part of the process will be in the actual test of the item or document. This generally involves obtaining and reviewing the document for the amount being tested. The auditor should keep a tally or worksheet of all the items tested and thee results of the tests. This data will then be put into a file and read into RAT-STATS in order to make the computation as to the overall population values.
- Once all the testing has been completed, the auditor will summarize the results, i.e. enter both the examined value and the audited value into RAT-STATS. RAT-STATS can then compute the results using statistics and can output the results in a variety of formats such as an excel workbook, Microsoft Access database, text file, print file or directly to a printer.
- In the next session we will show an actual example of usage of the RAT-STATS software to make a random selection from a population of 5,000 invoices and arrive at a statistically valid conclusion as to the total value or amount of a population. In this workbook, both the examined values as well as the audited values are stored in the worksheet. This is done to expedite the testing, as the results of each test are already recorded for every item (invoice) in the population.
- So our first objective was to describe to you the how and why of digital analysis in order to provide an overview of the process sufficient that you might be able to decide whether digital analysis might make sense in your organization and some of the alternatives that are available. In the next section I’m going to talk about how to use a methodical, objective approach using various statistical techniques.
- So our first objective was to describe to you the how and why of digital analysis in order to provide an overview of the process sufficient that you might be able to decide whether digital analysis might make sense in your organization and some of the alternatives that are available. In the next section I’m going to talk about how to use a methodical, objective approach using various statistical techniques.
- So our first objective was to describe to you the how and why of digital analysis in order to provide an overview of the process sufficient that you might be able to decide whether digital analysis might make sense in your organization and some of the alternatives that are available. In the next section I’m going to talk about how to use a methodical, objective approach using various statistical techniques.
- The next area I’d like to talk about is trends. This a often a very critical part of many types of analysis, both for errors and for fraud. You’ll recall that earlier I spoke of the example of the two dentists who went from 0 to multi-millions per year in billings within a three year period.
- Here is a very simple example of a hypothetical company, ACME technology. Their sales are steadily increasing over the 18 month period shown. However, their count of employees is also steadily decreasing. This could either mean that they are becoming increasingly efficient or else that something needs to be looked at. Here you have two trend lines which should be associated or correlated, yet one is going up and the other is going down.
- There is a mathematical technique which can determine linear trends among a group of data. This will result in the development of a trend line and also a measure as to just how good the fit is. This technique is particularly adept at identifying spikes and situations where trend lines differ, i.e. one trend is up yet another trend is down.
- Generally the type of errors that this procedure can detect are anomalies based upon trend lines and their slope. Often trends should be generally correlated. In the example we talked about before, dental billings were increasing at about 7% per year, yet for two dentists the trend lines were sky rocketing. This procedure can also identify key punch errors which were not caught where amounts are an order of magnitude larger than they should be (extra digit).
- Many variables in a population have a linear relationship. For example, assuming that all items are taxable and the sales tax rate does not vary, then the invoice amount excluding tax should have a linear relationship with the sales tax – i.e. the sales tax rate. Although this is a simple example, there are many audit applications for linear regression tests.
- Linear trends are generally identified based upon best fit. The straight line that results in the fewest overall differences between actual and projected. This technique will also measure the extent to which items differ from the expected amount, which is referred to as standard errors. If the population is separated into groups and each group is measured using best fit, it then becomes possible to sort the results based upon slope, variability etc in order to identify those elements which have the most difference from the overall population.
- Here is an example of a sort based upon slope. Note that the top four elements show a fairly flat or gently upwards slope. However, account 43870 has a much lower slope, and account 54630 is even negative. It is also possible to look at standard error as well.
- So our first objective was to describe to you the how and why of digital analysis in order to provide an overview of the process sufficient that you might be able to decide whether digital analysis might make sense in your organization and some of the alternatives that are available. In the next section I’m going to talk about how to use a methodical, objective approach using various statistical techniques.
- At its very basic, the detection of any fraud pattern involves a process which can take a population of transactions and separate the insignificant many from the significant few. This process should be able to help in the identification of errors, some of which may be intentional errors – i.e. fraud. The approach discussed this morning is based largely on quantifiable tests and doesn’t necessarily rely on the auditors conception (or preconception) of any specific attributes of the data.
- At its most basic, this is a “shotgun” approach. The basic idea is pretty simple. You begin by selecting a metric, such as invoice amount. There are a host of possible metrics and we’re going to discuss them in more detail coming up. Once you decide on the metric, you measure the population as a whole. Then you divide the population up into logical groups. These groups could be vendors, store locations, warehouses, operating divisions, employees, etc. Anything that makes logical sense from an organizational perspective. You then assess the same metric for each of the groups that you have identified. For each group, you compare the metric for the group with that of the population overall. This comparison can be done using any of a variety of statistical measures. Once all groups have been assessed, it is then possible to sort the results by the metrics in order to determine which, if any, of the groups is significantly different than the population as a whole. For these groups, the auditor then makes further inquiries to determine why the group is so different. There may very well be a valid explanation, but it is also possible that an error exists. In the extreme situations, there can be intentional errors being made.
- In any error situation, something is different than expected. Here is a chart of the classic digital analysis test for Benford’s law. We’ll get into the details of benford’s law later, but for now notice that the graphic charts expected versus actual for nine digits. The expected values are charted in dark blue and the actual values are charted in light blue. The largest difference between actual and expected is charted in yellow with the light yellow being the actual and the yellow being expected.
- This is a common theme in various pattern patterns – something is different. And the approach suggested here is a structured, methodical approach which is designed specifically to identify any groups within a population where something is different, and provide a quantifiable amount as to just how different it is. Obviously in any population, each group will have its differences. Where the auditor may need to be concerned, however, is where the differences are very large, especially where those differences exist in just one or two groups.
- So how are the differences measured? Generally there are two major classes of measurement types. The measurement may be based solely upon transaction counts, (for example as done for benford’s law) or it may be based upon transaction amounts (such as outliers, which are amounts which are significantly different than the population as a whole).
- I want to say a few words about the use of statistics here. First, we are not using statistics in the traditional sense of making some sort of estimate. Our use of statistics is strictly for developing a measure which will then be used in a judgmental fashion to determine if further audit testing sees appropriate, in other words we are using the statistics to rank groups within a population. Also, keep in mind that it is generally not necessary to have specific knowledge of the statistical computations because most of the software packages will already do the number crunching for you.
- The comparison of each group with the population as a whole can be based either on counts or amounts. Either way, a measure of difference will be obtained which can then be used to rank the groups. Often the highest differences will also have the greatest possibility for error or fraud. Note that this technique is also used by various taxing authorities such as the IRS. They call it the discriminate difference function.
- Here we see a rather simple example of a means to stratify data in a population. Both charts are for the stratification of a population of values which range from 0 to 100. These amounts are then separated into five groups, the first one being values between 0 and 20, the second between 20 and 40 , etc. The chart on the right contains the sums of the values instead of the counts. Thus it is possible to see that the chart on the right will tend to be skewed to the right, due to the larger values being summed.
- Now we provide a hypothetical comparison between values in a population and values in a group. These particular charts are different from the ones before, and show values for a period of twelve months in a calendar year. The chart on the left shows the values for the population as a whole and the chart at the right shows the values for a particular group. The next step will be to convert these amounts shown into cumulative percentages.
- Here we take a chart of counts by month (shown on the left) and convert the values into cumulative percentages. This is a relatively simple process. We start with the first value on the left and divide it by the total for all months to arrive at a percentage. We do the same for the next month, and then add that percentage to the prior value and so on and so forth. The last value in the process will have a value of 100%. By expressing all the values as cumulative percentages, we are now able to compare the two populations on an apples to apples basis, regardless of the actual counts or totals.
- We now need to be able to answer the question, are the two histograms actually different. Obviously there will always be differences, but the question is are these differences just due to chance or not. There are a variety of ways that two populations can be compared. Here we will focus on just two statistics. The first is the chi squared statistic and the second is the Kolmogorov-Smirnov test. Both of these tests will provide a measure of difference between two distributions. Now I want to again emphasize that we are using these statistical tests not to compute any sort of estimate, but simply to rank a collection of groups as to the extent to which they differ from the population as a whole. From a purely statistical point of view, the Chi Square test can only be applied in certain circumstances, and the K-S test can only be used on continuous distributions, but we are ignoring those limitations here.
- As a refresher, the Chi square test is a classic test on count values contained in a table of rows and columns. This test can be used to make a statistically valid conclusion as to whether the difference in the values in the columns is statistically significant. As I mentioned before, there are some limitations on when it can be used. However here we are not trying to answer the question of whether the data difference has statistical significance, but just to rank the groups within the population as to the extent of their difference from the population as a whole.
- I’m not going to go through the computations involved in computing the chi squared statistic. There are some links at the end of this presentation for anyone curious or anyone who wishes a refresher on the topic. The links are from the physics department at a well known university. Just so that you are aware of some of the terminology used, the results of the computation yield a chi squared value. This value, along with the degrees of freedom which is the lesser of the number of columns or rows minus 1 can then be used to come up with a “p-statistic” which will quantify the likelihood that differences are due to chance. All things being equal, a large chi square value will indicate that the differences are not due to chance, that is, there is indeed a statistically significant difference between them. The computations involved are somewhat intensive, but are relatively easy to perform on a computer. If your software package doesn’t provide this functionality, there are some excellent public domain packages available. One such package is the Cephes math library.
- I want to mention very briefly two Russian mathematicians who developed a means to formally compare any two distributions. Their names are Andrey Kolmogorov and Vladimir Smirnov. Generally their names are shortened just to K-S for obvious reasons. They came up with a rather simply to understand metric, but one which is easy to illustrate. The next slide shows a specific example of how the metric is computed. This metric is referred to as the d-statistic or difference statistic.
- To compute the statistic there is a four step process involved. For each element measured, whether count or sum, compute the amount expressed as a percentage. Step two is to then repeat this process for all of the elements measured and compute the cumulative percentage amount. Now compare the two populations and for each corresponding element determine the difference amount as an absolute value, i.e. ignore the sign. Now from all of the computed values, identify the largest one. This is the d-statistic. If your software doesn’t perform this type of computation there are open source software packages mentioned in wikipedia and also available within the Cephes math library.
- Here is a graphic which shows two distributions being compared and also shows the d-statistic in red as the largest difference between the two distributions.
- We’re going to show how data can be classified using ten attributes which are shown here. We’ll take each of these one at a time and go through the process to be used.
- The ten metrics that we are going to discuss are shown here. Note that there can be other metrics as well, I am just providing ten as a starting point. However, I believe that these metrics alone can cover a good bit of the ground for fraud pattern detection.
- So for this second part we have provided an overview of the statistical basis for quantifying differences between two populations. One method makes use of the Chi square test and the other uses the K-S test. Generally either test is going to provide comparable results, as we are simply trying to rank the groups within the population as to the extent of their differences, not make any sort of statistical estimate or computation. In the next section we’re going to talk about specific metrics which can be used by the auditor.
- The whole objective of all of these computations is to identify or trap anomalies. This is done by using one of more of the ten metric which we are now going to discuss in detail. Just as an aside, this photograph was taken by my daughter at a location near the University of North Carolina at Wilmington.
- This morning I’m also going to focus on ten metrics or assessment techniques that I believe the auditor can and should apply as part of the fraud pattern detection process. Later in this session we’re going to go into detail on each of these ten areas, which I call the “Auditors Top 10”. Just briefly, each of these metrics can provide insight into the detection of unusual patterns or suspicious transactions.
- Outliers are values which are much different from the rest of the population. Often auditors will want to look at the top 10, top 50 etc. To be considered an outlier, the value should differ from the population average by a significant amount. The variability of a population can be measured as the coefficient of variation which is simply the population standard deviation divided by the population average. The stand deviation, as you may recall is simply a measure of the degree to which values within the population are dispersed.
- In almost any audit step, often the very first thing to be done is to tie in the population totals to a general ledger or other control account. Obviously there is no value in performing tests on a population whose total amount has not been determined as equal to a control account. In doing these types of tests, it is possible to obtain as a by product various pieces of information such as chart of histogram values, the top 10, 20 values as well as the population statistics such as average, standard deviation, coefficient of variation etc.
- Once you have obtained the basic statistics for the population as a whole, it is often also desirable to obtain these same statistics by logical group, e.g. store number, warehouse location, vendor number etc. If all these statistics are available, it is then possible to sort by statistics such as coefficient of variation, minimum, maximum etc. depending upon what type of population you have and which items will be of the most audit interest.
- In this chart we show the results of looking at coefficients of variation for claims submitted by each medical provider. You will notice that there are three columns. The first is the identifier for the provider, the second shows a count of the number of items which make up that providers statistics and the third column is the coefficient of variation. The values here were obtained by getting the statistics by provider and then sorting these in descending order by coefficient of variation. We see here that two providers stand out fro the population as whole in that they have significantly higher coefficients of variation. This could be due to various factors, but this is a starting point for the auditor to investigate why the values are so high for these two providers.
- The next metric is stratification. Stratification is simply a process to classify values into buckets. For example, if I know that most inventory values in a warehouse will be under $100, I may want to specify most of the bucket values as being between 0 and 100, and then also include a few much larger buckets in order to pick up any large items.
- Once you’ve done a stratification, the results can be shown as a histogram. For each of the buckets you identify, you can place either counts or totals into each of the buckets.
- You would begin by dividing the population into logical groups such as store numbers, warehouse location, vendor number, etc. Then develop the strata values to be used to perform the stratification. Generally this will be based on experience, but it is also possible to use a trial and error method until the results you get for the population as a whole make sense to you as an auditor. Once both the population as a whole has been stratified using the strata values selected, the same stratification procedure is then used for each of the groups making up the population. Each of these stratification procedures will result in a series of values, the number of which will depend upon the number of strata selected. Each of these sets of numbers can in turn, be compared with the population as a whole. The comparison can be based on visual comparison, which is generally only suitable if a relatively small number of groups have been chosen, or else it can be done programmatically using either Chi square for counts for K-S for either counts or amounts. These programmatic comparisons will then yield measures of difference, whether they are chi square, d-statistic or other. The results can then be sorted in descending order for use by the auditor to identify which groups are the most different.
- Some of the types of errors which may be detected by this process include transactions which are well out of the ordinary in terns of either amount or count. Up-coding of insurance claims is the unethical practice where a service that was provided is classified as a higher value service in order to receive a larger insurance payment. If a group of providers will tend to have claims within a certain range, then it may be possible to detect upcoding simply by looking at the distribution of amounts within a stratification. This same process can also detect any situation where there are skewed groupings, i.e. values clustered within ranges which are different from the patterns found in the population as a whole. Again, remember that testing based on stratification can be performed either on counts or amounts. Generally, counts are the most frequently used metric, although amount totals can be used as well. The use of amount totals will make the comparisons much more sensitive, because sums and totals can result in a very skewed distribution, i.e. heavily weighted to the larger values.
- So to recap the process here, there are five steps. First the population is stratified as a whole. This can be based on experience or more often it is a repetitive process which is performed by the auditor until they are satisfied with the stratification values – i.e. they make sense from an audit perspective. The next step is to then perform this same test for each group, e.g. vendor. Then the stratification results for each group are compared with the population as a whole and a measure of the difference is obtained. This measure can be either Chi square, K-S etc. The final step is to then sort the measures in descending order and look at perhaps the top few to determine why the measure is so different from the population as a whole.
- This chart shows the results of the comparison of strata. Here there are two providers who have very different billing patterns than the others as measured by the specific strata selected. These first two providers have very significantly larger chi squared values than the others. What this says is that there is practically no chance that the difference in the billing patterns between these two providers and the rest is due purely to chance. You will also notice that they have a fairly high d-statistic as well. These are two providers that the auditor may want to look at to see exactly why their billing pattern is so different from the rest.
- The next metric I want to talk about is day of the week. In other words how are transactions distributed during the week – Monday through Saturday. Does the pattern found on week days match up? How do the transactions shown on weekends versus weekdays appear?
- Some business have most or all of their transactions Monday through Friday, others, e.g. retail may have a heavy concentration on weekends. Also, their may be a peak period, which might be early in the week or late. Generally this pattern will be somewhat unique business to business, but the actual pattern may be considered a signature of the business.
- This procedure can identify a variety of situations such as unusually high or low activity on one or more days of the week. There was a case where a dentist had unusually low activity except for Tuesday. Turns out he was handling Medicaid patients separately from the rest, which is not an allowable practice. But this pattern would show up readily based upon an analysis of day of the week. Other dentists had unusually low activity on Friday – turns out they are closed for that day.
- How does this procedure work? First you start by classifying the entire population of transactions. Obviously you need to be looking at a particular column which is a date column. This could be transaction date, sales date, receiving report date, invoice date, etc. Then for each day of the week you assign a number from 1 – 7. It isn’t important which date you start with, e.g. Monday might be 1, or Saturday might be 1, just as long as you are consistent. Then you would prepare a histogram for the population as a whole. Next you split up the population into groups such as store numbers, vendor numbers, employee numbers, general ledger account numbers, etc. And you do the same for each of these groups, i.e. prepare a histogram. Then you compare each of the histograms for the group with that of the population as a whole and measure the difference using either a chi square test or K-S test. You would then sort the results obtained in descending order, e.g. chi square or d-stat and look at the top two or three depending upon how different they are than the rest. You may also want to exclude certain groups from testing if they have too few observations, otherwise these will tend to skew the results.
- Here’s an example of what your test results might look like. In this example, you will see that the top two lines are very different from the rest as measured by chi square and the d-statistic. In this case, the provider with the highest difference was the one who only provided service to Medicaid patients on Tuesdays. The second provider shown, 4562134, was almost always closed on Thursday and Fridays, which is not the general practice.
- The next area I want to talk about is round numbers, one of the tests specifically mentioned in the IIA paper on the management of fraud risk.
- Round numbers almost always involve estimates. By round number what I mean is a number that has one or more zeros to the left of the decimal point and the numbers to the right of the decimal point are zero. An example of a round number is 5,000. Although these amounts can happen on accounting transactions, e.g. an actual invoice amount for exactly 5,000, they are not necessarily that common.
- The purpose of looking at round numbers is to isolate estimates. It also has a use in identifying split purchases, e.g. combinations of amounts which are artificially low in order to avoid some sort of control. For example here may be a requirement that all purchase orders exceeding $25,000 be reviewed at a certain level. We had a situation a few years ago involving some kickbacks to a vendor and there was a good bit of manipulation in keeping the invoice amounts small in order to avoid the required review process. This kickback situation was prosecuted and some five or six individuals received jail time.
- So the approach used is very similar to that of the others. What I do is classify each amount as either round or not round. Then if the number is round I classify the type of round number based upon the number of zeros immediately to the left of the decimal point. For example, the amount 5,000 would be classified as a round number of type 3 because it contains 3 zeros. The amount 10,200 would be type 2 because it contains two zeros to the left of the decimal point. Once you have the population as a whole classified, you can then prepare a histogram based upon not round, round type 1, round type 2 etc. Then prepare this same process for each group obtaining a histogram for each. Then compare the histograms and develop an assessment of the difference, either chi square or K-S statistic or both.
- Now here is an example of what your results may look like. In this case a round number test was run against a file of journal entries and the testing was made based upon general ledger account. You can see that two accounts have extreme values as shown by the Chi square and the d-statistic. These two accounts also have very high d-statistic values. What this says is the chance that the round numbers contained in the se account postings due solely to chance is very remote. These are the accounts that the auditor may want to look at further.
- The next topic is the detection of numbers which are just made up. As I’m sure many of you are aware there are specific tests that can be performed in collections of numbers to determine if their distribution is as would be expected.
- The issue of made up numbers is one which has been addressed in many areas such as tax returns, insurance claims, schedules of amounts, and curb stoning. Curb stoning refers to the practice that some surveyors use instead of making a telephone call or visiting the person to be interviewed, the surveyor simply makes up the numbers. The term curb stoning is a whimsical reference to the surveyor sitting on the curb stone and writing down the amounts. There is a mathematical technique which can be used to test for made up numbers and the name of the law is Benford’s law.
- Benford’s law is based upon the work of a General Electric employee named Charles Benford who expanded upon a theory previously developed by an astronomer. Basically what he noticed was the leading digit of a collection of numbers did not occur in a uniform, or equal basis. Generally the first digit was most often a one and least often a nine. This chart shows the expected frequency of digits as the dark colored bars with the light colored bars being the actual observed frequencies. The two yellow bars are for the digit that is most out of whack, with the light yellow being the actual observations and the yellow bar being the expected number of observations. The test can be applied to a variety of situations, generally when the amounts are derived based upon some sort of computation or tally. The law does not apply to all situations, e.g. Any situation where there is an arbitrary limit on the range. For example, expense reports which have a ceiling of $25 would not necessarily follow benford’s Law.
- The formula for Benford’s Law can be plugged into an Excel worksheet to show the various values for each combination of digits. If you want to save yourself a little work, there is a workbook which has all the formula already entered, and this workbook can be downloaded from my site. In Excel you would obtain the leftmost digit using the Excel formula “left”. The formula shown here is for the first digit only. You could also use a substring formula to obtain the second digit, etc.
- The test can be applied to various combinations of digits, such as the first 1 2 or three digits, the second digit only, the last one or two digits etc. Theoretically more than just the first three digits can be tested, but this test tends to become unwieldy very quickly. This test, just like all the other tests we have talked about can also be checked using chi square or the K-S d-statistic. This enable you to quantify just how close or how much out of whack a particular situation is. There have been numerous articles written on Benford’s Law and I have included links to some of these articles on the last pages of this presentation.
- So to perform this test, the first step is to deiced upon the type of test to be performed. Do you want to test the first digit only, first two digits, etc. Often it may be desirable to first perform the test using the first digit only in order to get a general view of how the population looks. If the result is that the population seems reasonable, then it may be that no further testing is needed. You would then perform this test for each group selected and prepare a histogram which shows the count of the number observed as well as the number that are expected. Then based on these counts you would be able to obtain a Chi Square value or a d-statistic which could be used to quantify just how aligned or out of whack the counts are compared with that which would be expected using benford’s law. You can then sort these results in descending order and identify those which have the largest differences as indicated by the chi square or d-statistic results.
- This chart shows the results of applying the Benford’s law test against transactions at various stores. Here you see that two stores, number 324 and 563 had significantly different results than the rest of the stores. Not only are the Chi squared values sky high, but so too are the d-statistics. As an auditor, you might want to look at what is going on at these two store locations.
- The next topic I’d like to talk about is market basket. This is based on the concept that when you go to a grocery store and buy a bottle of wine, there is a good chance that you will also want to buy some cheese. So the grocer may want to ensure that the cheese is displayed close to the wine in order that you will buy both. But the market basket concept has wide applicability in the detection of fraud patterns as well.
- Basically the market basket concept is a means to quantify the likelihood that if one item exists in a group of items, then another item will also exist in the group. To use the example we had before, you might be able to make a statement that if you buy a bottle of wine there is a 30% chance that you would also buy some bread, or cheese or whatever. Let me give you an example of the application of the market basket concept tot eh area of medical insurance claims. There is an unethical practice in medicine known as ping ponging. The term comes from persons who are shuffled between two or medical medical providers to the extent that they feel like a ping pong ball. When they go to Dr. A he looks at them and refers them to Dr. B. Dr. B examines them and sends them to Dr. C, etc. Now there are many instances where this is completely appropriate. But there can also be instances where the whole scheme is simply one designed to drive up insurance revenues. Market basket relationships can be detected by a program known as apriori. There are some references at the end of this presentation to the apriori program and its application should you be interested. I’m not an expert in Latin, but I’ve been told that the word apriori is derived from the Latin a – from and prior (former) meaning that the conclusion is obtained from a former. In other words the conclusion that you would buy cheese is from the fact that you formerly purchased wine. In other words, making a deduction from the known.
- Some of the examples of patterns that can be detected using apriori include various types of unexpected patterns and associations. For example, you might find unusual combinations of diagnosis codes on a medical insurance claim.
- There was a United States bank that found a somewhat unusual pattern in the use of their credit cards. They found that when their card was used in Japan to pay for a tax fare, there was a fairly high likelihood that it would then be used to make an ATM withdrawal. What was actually happening was that fraudsters were using the card in a taxi to determine if it was a valid card. Once they knew it was valid, they would then use it at an ATM to obtain cash. Pattern associations can also be used in testing journal entries. For example you might find that if there has been a posting to a particular account there is a very high likelihood that there is also a posting to another specific account. This type of test can also be used to tie journal entry approvals to a particular person or group. In other words, if there has been a posting to account ABC, there is a 90% likelihood that it was approved by Mr. X.
- How is the testing done? The first step is to identify all the elements of the population and assign to each element a unique integer number. Then you need to create a text file which has all these values. Once this text file has been created it is then possible to run an apriori analysis using a program.
- The results of the test will show, for each unique value, the probability that there are other values. For example, the test may show that if you visit Dr. Jones then there is an 80% probability that you will also visit Dr. Smith. Similarly, if there is a journal entry posting to account ABC, then there is a 30% chance that this entry will also have a posting to account XYZ.
- In accounting there are many transactions which should consist of consecutive numbers. Examples include, check numbers, cash receipts, purchase orders, invoices, etc. Often it is of interest to determine if any of these sequences has missing numbers, i.e. gaps between consecutive numbers.
- Too many times what you see is interesting but what you don’t see can often be critical.
- This is the sort of test that can identify missing documents, missing checks, missing receiving reports, etc. This is a very commonly performed audit test.
- Any sequence of numbers which should be complete can be tested using this technique. This can include items such as serial numbers, tag numbers, etc.
- If you’re doing this test in excel you would sort the data by numerical sequence and then plug in a formula to check if the difference between any two consecutive amounts was greater than one. This same concept can also be applied to dates.
- Here is an example of a test done on check numbers. Shown here are three check number ranges along with the number of checks which are missing. For example , the first line shows the start of the range at 10789 and then end of the range at 10791, which indicates that one check number,, 10790 is missing. It is also possible for a larger group of checks or other sequentially numbered documents to be missing.
- Next I’d like to talk a little about duplicates, which as a wide use within auditing.
- In my view there are really two general types of duplicates which I term as same, same same and same same different.
- In the case of same same same you have a series of criteria where you are looking for a match in each. An example of a possible duplicate invoice might be defined as the same vendor, the same invoice number same invoice date and the same amount. An example of same same different is when you are looking for a series of criteria where each is the same except for the last. A simple example might be the same employee name, same city but a different social security number. An example in the medical insurance area would be same provider, same patient, same date, same service, but a different claim number.
- Duplicate payments can arise for a variety of reasons – data resides on different platforms/systems, edits for the accounts payable system are inadequate or overridden, mis-keyed invoice information, etc. In order to better identify duplicate payments, “fuzzy” logic may be needed.
- Fuzzy logic can be used to identify two items which are not an exact match, but are close. In some case this may result from a transposition, omission of a digit, etc. The Russian Physicist Levenshetin developed an algorithm which can do this. The algorithm takes two text values and then makes a series of adds, changes and deletions in order to get the text values to match. The number of these transactions is referred to as the Levenshtein distance. If the text values are already an exact match, then the distance is zero. If only one addition, change ort deletion is required, then the distance is one etc. It is also possible to place various weighting criteria, such that additions count for more than deletions, etc. Another widely used technique is soundex. This algorithm has been around for some time and is based upon the concept that some words sound similar to others. For those auditors who have data stored in databases and wish to test it, some of the database implementations have features that can provide means to identify similar text. Examples include the “LIKE” clause as well as regular expressions. Regular expressions are a whole other topic which gets quite complex, but suffice it to say that this is a very powerful tool that is available.
- How can duplicates be identified? The first step is to sort the file in the desired sequence before the testing can be performed. Once the file has been sorted, it’s a matter of comparing items in consecutive rows. Any of these consecutive rows can then be extracted for follow-up. This basic concept applies to both same same same as well as same same different.
- Here is an example of the output you might obtain as a result of a search for possible duplicate invoices. This example is very basic, but it shows that for vendor 10245, using the specified invoiced date and invoice amount, six possible dup0licate invoices may exists. The test also identified potential duplicates for vendor 17546.
- The final metric I’d like to talk about concerns date testing.
- A sometimes asked question by investigators is if the business is closed, why is there a receiving report or why is there an adjusting journal entry?
- Generally, a transaction occurring on a holiday date is a red flag. Certain financial transactions, for example journal entries, may not be expected to occur on holidays, especially if the business is closed on those dates. Thus, under some circumstances, their occurrence can be a “red flag” that further review may be warranted.
- Certain holidays might be fairly easily identified visually, especially if the number of dates is relatively small. However, when the number of dates begins to be numbered in the thousands or tens of thousands or more, the process can become tedious and error prone. (Quick – which day was Independence day celebrated on in 2004 - July 3rd, 4th or 5th? Hint – it was a Monday.) (Pause) In 2004 July 4 th fell on a Sunday so it was celebrated on Monday July 5 th . This is one reason it is difficult to identify holiday dates visually.
- Unless a business is operating during a holiday, it will be unusual for transactions to be initiated during that period. There are numerous areas in which holiday dates might not be expected, including:
- Any transactions which occur when a business is closed are candidates for review. One of the possible types of dates to be checked are holiday dates, which in the government world are established by the federal Office of Budget management. In some cases, transactions on holidays are an excellent indicator of potential fraud.
- Although it is possible to visually inspect dates in an attempt to identify holidays, that process can be tedious and time consuming. The procedures described here are an efficient and effective way to identify and quantify transactions which fall on a federal holiday (U.S. only).
- Federal law (5 U.S.C. 6103) establishes the public holidays for Federal employees. There are ten days defined, and the holiday either falls on a specific day (unless that day is a weekend), or else is a floating day. The table on the next page lists the holidays and how they are determined:
- Federal holidays are established by the Office of Personnel Management. Each holiday is based either on a specific date – e.g. 4 th of July is Independence Day, or else a floating date which is a specific day of the week for a specific week within the month. For example Martin Luther King’s Birthday is celebrated on the third Monday in January, regardless of what that date is.
- So if you’re testing for holiday dates, how is it done? The first step is to obtain a distribution or count of all the federal holidays in the entire population. Then divide the population into groups and for each of these groups determine a count of dates which fall on federal holidays and classify them as to which h of the ten holiday dates they represent. So for example, the results for one sub-group might be 2 dates for Martin Luther Kings Birthday, 1 date for Presidents Day and 2 dates which fell on Thanksgiving. Once this has been done it is then possible to compare the histogram for each group with the population overall. This will tell you which of the sub-groups were the furthest from the overall population average. It is also good to sometimes look at the overall population just to understand what the distribution is. For example, in very few businesses would you have transactions on Christmas day, although it is certainly possible for some businesses. Once all the computations have been done it is then possible to sort the results in descending order by the Chi square value or the d-statistics to see which groups are the most out of line and also to see just how large a gap there is between these transactions and the population overall.
- Here is a hypothetical of tests of employee expense reports. You can see that two employees stand out from the crowd in that they tend to have expenses recorded on Federal holiday dates well outside the norm. Of course there could be a very valid reason that this is the case, but this is something that the auditor/investigator may want to look at, depending upon the circumstances.
- So to summarize our third objective which is to identify ten general tools and techniques which can be used for fraud pattern detection we have gone over ten specific techniques or attributes to be tested for. In each case, the metric is applied to the population as a whole and then the population is divided into groups and the tests reperformed on the group and then the group is compared with the population overall. This comparison results in a Chi square value or d-statistic which can then be sorted in order to identify those sub-groups, if any, which stand out significantly from the population as a whole. Now our next topic is going to go over some examples using Excel which is a very commonly used tool among auditors.
- There is a very useful menu function within Excel called data analysis. This should be a menu item within the Tools menu. On my system it shows up as the fifth menu item on the Tools menu, and is called data analysis. If you don’t see it, it’s possible that it was not installed when Excel was setup. This should be a fairly easy shortcoming to overcome, it just involves another add-in which is part of Excel.
- Another aspect of Excel which is very useful is the ability to work with ranges. There are a variety of functions which can be used, all of which work with ranges within a worksheet. In addition to the standard sum, count, average and standard deviation, there are also functions to identify the largest and smallest values using the largest and smallest functions. The functions here are just a few of those available, you may want to look at the help screens to become more familiar with all of the functions if you are not already aware of them.
- The data analysis function also has the ability to create histograms based upon a specified bin range and data range. The bin range will be a series of values used to establish the bins into which the data counts are to be stored.
- Excel can also bee used to perform gaps testing for numeric sequences once the data has been sorted in sequence. One of the possible formula to be used is the if statement which can be used to place a value of true or false if a condition has been met. Here we are showing a formula which just compares the value in one row with the value in the prior row and stores a 1 in the cell containing the formula if the values are not equal, otherwise it places a value of zero. Generally you will want to copy and paste special the values obtained into another range before sorting them, so you don’t loose the values being tested.
- You can also perform testing for duplicates using the IF function. If you have multiple columns to be tested this can get a bit hairy due to the syntax requirements, but it is doable. This provides a reasonable powerful capability within excel to test for duplicates.
- Also in some circumstances databases can be access directly with Excel assuming that the proper ODBC drivers are present. If this is the case, then all the capabilities which are built into the database software can be used from within Excel. This include control totals, exception identification and the ability to drill down to specific areas of interest through the use of where clauses in Excel. I have an article which should be coming out in EDPACS within the next few months that goes into this in more detail.
- So our fourth objective was to go over some examples of the use of Excel as part of fraud pattern detection. I believe we have covered at least some of the basic functions and add-ins that may be of value. Also, if you have the appropriate drivers and some of the technical knowledge, it is possible to also perform fairly complex data base queries from within Excel to identify potential fraud patterns. Next I want to talk about how I think fraud pattern detection fits into the overall scheme of things.
- I think that fraud pattern detection sits right in the middle of things. On the one hand your organization may already be tracking a variety of business analytics. Generally these business analytics should be considered as part of the overall design of a fraud pattern detection system. Your audit organization may also be using continuous monitoring to one degree or another. Fraud pattern detection can both feed information to a continuous monitoring system as well as receive information from it.
- So fraud analytics can be used to develop business analytics as an offshoot of the data analysis process as well as received business analytics information from the process. This should be a two way street.
- Fraud pattern analytics can and should also feed the continuous monitoring system, if one has been established. Fraud pattern detection can evolve from a once off process to an ongoing program, in other words continuous fraud pattern detection. Output from continuous monitoring can also be used to tune fraud pattern detection.
- So our first objective was to describe to you the how and why of digital analysis in order to provide an overview of the process sufficient that you might be able to decide whether digital analysis might make sense in your organization and some of the alternatives that are available. In the next section I’m going to talk about how to use a methodical, objective approach using various statistical techniques.
- There is an excellent discussion of the Kolmogorov-Smirnov test at the first link which is the physics department of a large university. This is not a topic to shy away from, the article they present is quite informative. The second link is to a site which is that of an internationally recognized expert in the area, Dr. Mark Nigrini. The third bullet is also a link to an excellent article on Chi square which is also at the same university physics department as the Kolmogorov – Smirnov discussion. I have provided a link to an article on continuous monitoring which is a good starting point if you’re interested in the subject.
- This page shows links to an article on ping ponging written at Temple University. I am also providing a link to the page of the open source program apriori which is written by a professor in Spain, his name is Christian Borgelt. This is a DOS based command, but quite useful. I have been informed that a software company picked up this program and embellished it substantially into a commercial program. The last link I have on this page is to an article that I wrote on the subject of ping ponging.
- This last page of links is first to an excellent article by Rich lanza on the use of Excel as an audit software. I t goes into a lot of the details. The second link is to a variety of excel macros which can be used for a variety of audit purposes. The Workbook and article are located on my web site. Finally there is an article which goes into the overall use of spreadsheets for audit purposes. I have covered just some of the links, there are many others on my site.
- This is my contact information, should you wish to get in touch with me. Now I’d like to open up the floor to a general question and answer session.