Prior continuous auditing research focused on detecting exceptions efficiently We need to develop a classification hierarchy or ranking of detected exceptions to limit the need for human intervention.

1. EXCEPTIONAL EXCEPTIONS
Hussein Issa
(Spring 2015)

2. BACKGROUND

3. Background
• Audit by Exception (Vasarhelyi and Halper, 1991)
• Continuous Auditing literature is rich with studies that propose
statistical and machine learning techniques to identify
exceptions (Dull et al., 2006; Groomer & Murthy, 1989a; Alexander Kogan et al.,
1999; Vasarhelyi & Halper, 1991)
• Problem?
• Result?
35/31/201534th WCARS
Large numbers of Exceptions!
-Information overload!
-Ambiguity
-Information relevance

4. Motivation & Contribution
Motivation:
• Prior continuous auditing research focused on detecting
exceptions efficiently
• We need to develop a classification hierarchy or ranking of
detected exceptions to limit the need for human intervention
Contribution:
• Fill the gap in the continuous auditing literature with regards to
providing the human users with assistance in dealing with
large number of exceptions
• Propose exceptions prioritization methodologies
45/31/201534th WCARS

5. Relevant Literature-Exceptional Exceptions
• Analysis usually yields large amounts of exceptions, causing
information overloads due to sub-optimal business processes
or over-conservative CA/CM system (Alles et al 2006, 2008 ;
Debreceny et al. 2003)
• Human users perform complex aggregation and processing
tasks poorly (Iselin, 1988; Kleinmunitz, 1990)
• Murthy 2012 emphasized the lack of and need for studies
addressing the problem of large numbers of exceptions
55/31/201534th WCARS

6. IDENTIFYING AND PRIORITIZING
IRREGULARITIES USING A RULE-
BASED MODEL WITH A WEIGHTING
SYSTEM DERIVED FROM EXPERTS’
KNOWLEDGE

7. Motivation, Research Questions, & Findings
Motivation:
• Plenty of studies for exception identification, few address processing
• Majority of expert systems assign the same weight to rules
• HOWEVER: business rules, and accordingly their violations, do not have
the same importance
Research Questions:
1. How can we integrate the judgment of the domain experts (in this case the
auditors) in a rule-based expert system?
2. How can we develop a weighting system for the various rules in that expert
system?
3. How can we use this weighting system to prioritize exceptions?
75/31/201534th WCARS

8. Suspicion Score
85/31/201534th WCARS
Rank
4
2
3
1
6
5

9. Framework
95/31/201534th WCARS

10. Framework
105/31/201534th WCARS
6

11. Rule-Based System
• Set of IF-THEN rules
• Popularity stems from simplicity, interpretability, flexibility
• Data: Simulated Order to Cash data
• Originally 33 analytics, narrowed down to 12 by experienced
auditors
• 12 analytics can be categorized as tests for:
– Segregation of duties
– Unauthorized transactions
– Missing documents
– Non-matching documents.
115/31/201534th WCARS

12. Rules Weights Inference
• Business rules and accordingly their violations, do not have
the same significance
• 28 participants with 3 and more years of experience:
– Conduct 17 pairwise comparisons
– Select the transaction they believe to present higher control risk
– Provide justification if their assessment
125/31/201534th WCARS
Customer
Transaction
ID
Custom
er ID
Invoice
Number
Invoice
Date
Created
by
Inventory
Item ID
Invoice
Selling
Price
Invoiced
Amount
Shipment
ID
Shipment
document
Created By
Shipment
document
Approved By
10034 1146
10001203 10/15/1997 1546 63
800
9600 10000877 1002 1546
4110 1000
10000262 3/1/1998 1117 628
600
1800
SOD Missing Values
(Orphaned invoices)

13. Rules Weight Inference-LP
• Special Case Linear Program
– 16 pairs
– Each transaction can violate one and only one
• General Case Linear Program
– 17 pairs
– One transaction violated two rules
135/31/201534th WCARS

14. Exceptions Identification & Prioritization
Identification:
• Apply expert system to the whole population to find all the records
that violate one or more rules
• Remaining records are assumed to be normal, thus presenting
negligible risk
Prioritization:
• Calculate the Suspicion Score for each exception such that:
145/31/201534th WCARS

15. Exceptions Prioritization-Example
155/31/201534th WCARS
Rank
4
2
3
1
6
5

16. Investigation & Feedback
Investigation:
• Auditors are provided with Prioritized exceptions
• Scope of investigation depends on their time/budget
constraints
Feedback:
• Helps adjusting the rules that make up the expert system.
• Enables us to modify the weights of the rules according to the
audit teams’ findings
– incorporated as a new set of constraints in the general case Model
• Effect of the original experiment will decrease over time with
more feedback from auditors
165/31/201534th WCARS

17. Weights –Special vs. General Models
175/31/201534th WCARS
• Excessive Write offs ranked highest
• SOD in general ranked low
• Rules with direct impact on financial numbers ranked high
• Operational controls ranked low
Analytic Special Case Model General Case Model
Weight Rank Weight Rank
Analytic_1_SOD_Customers 1.00 11 1.00 11
Analytic_2_Unauthorized_Sales_Order 2.86 2 2.71 3
Analytic_3_Unathorized_Price 1.38 9 1.35 9
Analytic_4_SOD_Credit_Adjustment 1.21 10 1.20 10
Analytic_5_Match_Shipping_to_SO 1.79 8 1.72 8
Analytic_6_Match_Invoice_to_Ship 2.33 4 2.22 5
Analytic_7_Missing_Sales_Orders 2.31 5 3.18 1
Analytic_8_Unauthorized_Shipments 2.27 6 2.17 6
Analytic_9_SOD_Ship_Invoice 1.88 7 1.81 7
Analytic_10_Orphaned_Invoices 2.85 3 2.70 4
Analytic_11_SOD_Invoice_Receipt 1.00 12 1.00 12
Analytic_12_Excessive_Write_Offs 3.11 1 2.94 2
Analytic
Special Case Model General Case Model
Weight Rank Weight Rank
Analytic_12_Excessive_Write_Offs 3.11 1 2.94 2
Analytic_2_Unauthorized_Sales_Order 2.86 2 2.71 3
Analytic_10_Orphaned_Invoices 2.85 3 2.7 4
Analytic_6_Match_Invoice_to_Ship 2.33 4 2.22 5
Analytic_7_Missing_Sales_Orders 2.31 5 3.18 1
Analytic_8_Unauthorized_Shipments 2.27 6 2.17 6
Analytic_9_SOD_Ship_Invoice 1.88 7 1.81 7
Analytic_5_Match_Shipping_to_SO 1.79 8 1.72 8
Analytic_3_Unathorized_Price 1.38 9 1.35 9
Analytic_4_SOD_Credit_Adjustment 1.21 10 1.2 10
Analytic_1_SOD_Customers 1 11 1 11
Analytic_11_SOD_Invoice_Receipt 1 12 1 12

18. Weights – Internal vs. External Auditors
185/31/201534th WCARS
All Responses Internal Auditors External Auditors
Analytic Weight Rank Weight Rank Weight Rank
Analytic_1_SOD_Customers 1.00 11 1.00 10 1.00 11
Analytic_2_Unauthorized_Sales_Order 2.71 3 2.65 4 2.84 3
Analytic_3_Unathorized_Price 1.35 9 1.51 8 1.24 9
Analytic_4_SOD_Credit_Adjustment 1.20 10 1.45 9 1.06 10
Analytic_5_Match_Shipping_to_SO 1.72 8 1.98 7 1.59 8
Analytic_6_Match_Invoice_to_Ship 2.22 5 2.29 6 2.22 5
Analytic_7_Missing_Sales_Orders 3.18 1 3.10 1 3.37 1
Analytic_8_Unauthorized_Shipments 2.17 6 2.36 5 2.08 6
Analytic_9_SOD_Ship_Invoice 1.81 7 1.00 11 1.79 7
Analytic_10_Orphaned_Invoices 2.70 4 2.75 3 2.76 4
Analytic_11_SOD_Invoice_Receipt 1.00 12 1.00 12 1.00 12
Analytic_12_Excessive_Write_Offs 2.94 2 2.93 2 3.05 2

19. Takeaway from today
I think the main take away from this presentation is:
A. Continuous auditing sucks, dude!
B. It is better to stick to traditional sample based auditing
C. Number of exceptions is always very low
D. Hussein is awesome!!! (Hint: correct answer)
195/31/201534th WCARS
Too many exceptions!
Information Overload
Research Opp. in EE