This presentation presents techniques and experimental results for considering prediction reliability and risk during predictive business process monitoring. Considering reliability and risk provides additional decision support for proactive process adaptation.
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
Predictive Business Process Monitoring considering Reliability and Risk
1. Predictive Business Process
Monitoring unter Berücksichtigung
von Prognoseverlässlichkeit und
Risiko
Andreas Metzger, Philipp Bohn, Felix Föcker
CAiSE 2017 - https://doi.org/10.1007/978-3-319-59536-8_28
ICSOC 2017 - https://doi.org/10.1007/978-3-319-69035-3_25
2. Motivation
Predictive Monitoring and Proactive Adaptation
2SE 2018, Ulm
monitor
predict
real-time
decision
proactive
adaptation
time
t t +
planned /
acceptable situations
= Violation
= Non-
Violation
e.g., delay in
freight delivery
time
e.g., schedule
faster means of
transport
4. Considering Reliability
Prediction Accuracy
SE 2018, Ulm 4
• Prediction accuracy is key for proactive process adaptation
• Prediction accuracy = ability of prediction technique
• to forecast as many true violations as possible,
• while generating as few false alarms as possible
• True violation triggering of required adaptations
• Missed required adaptation = less opportunity for proactively preventing
or mitigating a problem
• False alarm triggering of unnecessary adaptation
• Unnecessary adaptation = additional costs for executing the adaptations,
while not addressing actual problems
5. Considering Reliability
Prediction Accuracy
• Research focused on aggregate accuracy
• E.g., precision, recall, mean average prediction error, …
• But: aggregate accuracy gives no direct information about error of an
individual prediction
• Prediction reliability estimates provide such information
SE 2018, Ulm 5
Aggregate Accuracy
75%
75%
75%
75%
Distinguish between more or less reliable predictions on case by case basis
Prediction #
1
2
3
…
Reliability Estimate
60%
90%
70%
…
6. Considering Reliability
Predictive Monitoring with Reliability Estimates
SE 2018, Ulm 6
monitor
predict
real-time
decision
time
t t +
planned /
acceptable situations
= Violation
= Non-
Violation
≤ threshold no adaptation
> threshold adaptation
+ Reliability estimate
Reliability estimates offer more information for decision making
proactive
adaptation
7. Considering Reliability
Computing Predictions and Reliability Estimates
Foundation: Ensemble prediction using Machine Learning
SE 2018, Ulm 7
Prediction T
Reliability estimate
Process
Monitoring
Data
Classification Model 1
Classification Model m{
{{ Each model of ensemble
trained differently
(bagging)
T1
Tm
9. Considering Reliability
Experimental Design
Process Model and Data Set
Airfreight process
• 5 months of operational data
• 3 942 process instances
• 56 082 service invocations
9SE 2018, Ulm
Point of
Prediction
11. Considering Reliability
Experimental Results
• Observations for full range of , , (= 5000 cases)
• Striving balance between avoiding unnecessary proactive
actions and rejecting required proactive actions
• Cost savings due to proactive process adaptation
• No, in 47.5% of the cases
• Yes, in 52.5% of the cases
• Cost savings due to considering
reliability estimates
• No, in 17,1% of the cases
• Yes, in 82.9% of the cases
12SE 2018, Ulm
Cost savings
Frequency
Savings from 2%
to 54%,
14% on average
13. Accuracy of individual prediction
Reliability estimate to quantify probability of violation
Severity of violation
• E.g., contractual penalties (such as stipulated in SLAs)
Estimated penalty to quantify severity (in terms of costs)
Risk = Probability of occurrence × Severity [ISO 31000:2009]
Risk = Reliability estimate × Estimated penalty
SE 2018, Ulm 14
Considering Risk
Factors impacting the success of the adaptation decision
14. Considering Risk
Risk-based Proactive Process Adaptation Decision
SE 2018, Ulm 15
monitor
predict
real-time
decision
proactive
adaptation
time
t t +
planned /
acceptable situations
= Violation
= Non-
Violation
R ≤ threshold no adaptation
R > threshold adaptation
+ Risk R
Risk estimate as basis for decision making
15. Ensemble
Considering Risk
Computing Penalty
16SE 2018, Ulm
Penalty =
Process
Monitoring
Data
{ =
1
𝑛
×
𝑖=1,…,𝑛
𝑎𝑖 − 𝐴
Regression Model 1
Regression Model n
a1
an
{
Deviation
δ
Linear with cap
clin
c
0 1
δ
Constantc
0
cconst
1
c
Step-wise (s steps)
δ
1/s 2/s
cstep
1
2/s·cstep
1/s·cstep
(s-1)/s
…
0
𝑐()
16. Considering Risk
Experimental Results
Penalty R = 0.1 R = 0.3 R = 0.5 R = 0.7 R = 0.9
constant -19.0 -20.0 -17.0 -3.0 3.1
step-wise -14.0 12.0 20.0 20.0 8.6
linear 0.6 21.0 27.0 26.0 11.0
SE 2018, Ulm 17
Cost savings (averaged over α ={0.1, 0.2, 0.3, … ,1})
Constant penalty Step-wise penalty Linear penalty
Risk threshold R
Costsavings
17. Conclusions and Perspectives
Predictive business process monitoring
• Prediction of potential problems before they occur
• Proactive adaptation of processes
• Cost Savings
• Reliability: 14% on average
• Risk: + 23% on average
Deep Learning for Process Prediction
• Recurrent Neural Networks (RNNs) with LSTM
• Initial results: 27% higher accuracy than Multi-Layer Perceptron (MLP)
SE 2018, Ulm 18
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0 10 20 30 40 50 60 70 80 90
Diagrammtitel
num s2e noplannednopath mlp
Checkpoint [relative prefix]
Accuracy[MCC]
RNN
MLP
18. Thank You / Danke
SE 2018, Ulm 19
…the EFRE co-financed operational
program NRW.Ziel2
http://www.lofip.de
…the EU’s Horizon 2020 research and
innovation programme under Objective
ICT-15 ‘Big Data PPP: Large Scale Pilot
Actions ‘
http://www.transformingtransport.eu
Research leading to these results has received
funding from…