This document presents research on predictive business process monitoring that considers reliability estimates of predictions. The researchers conducted an experiment to evaluate the impact of using reliability estimates on process performance and costs. They found that considering reliability estimates can positively impact costs in some situations by helping to balance avoiding unnecessary actions while still addressing required ones. However, reliability estimates did not always provide benefits. Open questions remain around how to determine the situations where reliability estimates are most helpful and how to provide additional information to decision makers.

1. Predictive Business Process
Monitoring Considering
Reliability Estimates
Andreas Metzger, Felix Föcker
Full Paper Presentation at the 29th International Conference on Advanced Information
Systems Engineering - CAiSE 2017, Essen, Germany, June 12-16, 2017, Lecture Notes in
Computer Science, E. Dubois and K. Pohl, Eds., vol. 10253. Springer, 2017.
https://doi.org/10.1007/978-3-319-59536-8_28
(Open Access)

2. Agenda
1. Motivation
2. Experimental Design
3. Main Results
4. Conclusions
CAiSE 2017, Essen

3. Motivation
Predictive Monitoring and Proactive Adaptation
3CAiSE 2017, Essen
monitor
predict
real-time
decision
proactive
adaptation
time
t t + 
planned /
acceptable situations
= Violation
= Non-
Violation

• A. Metzger, P. Leitner, D. Ivanovic, E. Schmieders, R. Franklin, M. Carro, S. Dustdar, and K. Pohl, “Comparing and combining predictive business
process monitoring techniques,” IEEE Trans. on Systems Man Cybernetics: Systems, vol. 45, no. 2, pp. 276–290, 2015.
• Z. Feldmann, F. Fournier, R. Franklin, and A. Metzger, “Industry article: Proactive event processing in action: A case study on the proactive
management of transport processes,” in DEBS 2013, Arlington, Texas, USA, ACM, 2013, pp. 97–106.
e.g., delay in
freight delivery
time
e.g., schedule
faster means of
transport

4. Motivation
Prediction Accuracy
CAiSE 2017, Essen 4
• Prediction accuracy is key for proactive process adaptation
• Prediction accuracy = ability of prediction technique
• to forecast as many true violations as possible,
• while generating as few false alarms as possible
• True violation  triggering of required adaptations
• Missed required adaptation = less opportunity for proactively preventing
or mitigating a problem
• False alarm  triggering of unnecessary adaptation
• Unnecessary adaptation = additional costs for executing the adaptations,
while not addressing actual problems

5. Motivation
Prediction Accuracy
• Research focused on aggregate accuracy
• E.g., precision, recall, mean average prediction error, …
• But: aggregate accuracy gives no direct information about error of an
individual prediction
• Prediction reliability estimates provide such information
CAiSE 2017, Essen 5
Aggregate Accuracy
75%
75%
75%
75%
 Distinguish between more or less reliable predictions on case by case basis
Prediction #
1
2
3
…
Reliability Estimate
60%
90%
70%
…

6. Motivation
Predictive Monitoring with Reliability Estimates
CAiSE 2017, Essen 6
monitor
predict
real-time
decision
proactive
adaptation
time
t t + 
planned /
acceptable situations
= Violation
= Non-
Violation

 ≤ threshold  no adaptation
 > threshold  adaptation
+ Reliability estimate 
Reliability estimates offer more information for decision making

7. Agenda
1. Motivation
2. Experimental Design
3. Main Results
4. Conclusions
CAiSE 2017, Essen

8. Experimental Design
Computing Predictions and Reliability Estimates
Foundation: Ensemble prediction using Machine Learning
CAiSE 2017, Essen 8
Prediction T
Reliability 
Process
Monitoring
Data
Classification Model 1
Classification Model m{
{{Each model of ensemble
trained differently
(bagging)
 T1
 Tm

9. Experimental Design
Cost Model and Experimental Variables
CAiSE 2017, Essen 9
Costs
Adaptation Cost
Adaptation Cost
+ Penalty
 > 
 ≤  No
Adaptation
Adaptation
Reliability 
Violation
Non-Violationeffective
not
effective
0
PenaltyViolation
Non-Violation
• Reliability threshold 
• Adaptation effectiveness 
• Relative adaptation costs : adaptation cost =  · penalty

10. Experimental Design
Process Model and Data Set
Domain: Freight Transport and Logistics
• One of the most-used industries in the world and in EU
• 15% of GDP (source: KLU), 4,824 megatonnes CO2 (source: DG MOVE),
increase by 40 % in 2030 and by 80% in 2050 (source: ALICE ETP)
• Airfreight process
• 5 months of operational data
• 3 942 process instances
• 56 082 service invocations
10CAiSE 2017, Essen
Point of
Prediction

11. Agenda
1. Motivation
2. Experimental Design
3. Main Results
4. Conclusions
CAiSE 2017, Essen

12. Experimental Results
Effect on Process Performance
12CAiSE 2017, Essen
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0.5 alpha-0.6 alpha-0.70000005
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00 1,05
Reliability Threshold
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
0,88
NonViolationRate
 (reliability threshold)
Non-violationRate
 = 1
 = .9
 = .8
 = .1
 = .2
 = .7
 = .6
 = .5
 = .4
 = .3
No proactive
process adaptation
Proactive process adaptation
without reliability estimates
.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.0





.88
.86
.84
.82
.80
.78
.76
.74
.72

Key:
Negative effect
Positive effect
 Optimum
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
0,88
NonViolationRate
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
0,88
NonViolationRate
Missed required adaptations
due to high loss of predictions,
not compensated by less
unnecessary adaptations

13. Experimental Results
Effect on Costs
13CAiSE 2017, Essen
EnsSize=100, BootsSize=80, Voting=simple, AdaptSuccRate=0.80
Tot - Lambda [%] 0 Tot - Lambda [%] 10 Tot - Lambda [%] 20 Tot - Lambda [%] 30
Tot - Lambda [%] 40 Tot - Lambda [%] 50 Tot - Lambda [%] 60 Tot - Lambda [%] 70
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00 1,05
Reliability Threshold
60.000
70.000
80.000
90.000
100.000
110.000
120.000
130.000
Cost
 (reliability threshold)
costs
 = .9
 = .8
 = 1
 = .7
 = .6
 = .5
 = .4
 = .3
 = .2
 = 0
 = .1
.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.0






130,000
120,000
110,000
100,000
90,000
80,000
70,000
60,000
No proactive
process adaptation
Proactive process adaptation
without reliability estimates
Key:
Negative effect
Positive effect
 Optimum
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
0,88
NonViolationRate
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
NonViolationRate
 = .8
Very high adaptation
costs, not compensated
by avoided penalties
Missed required adaptations
(even though cheap) due to
loss of predictions

14. Experimental Results
Effect on Costs
• Observations for full range of , ,  (= 5000 cases)
• Striving balance between avoiding unnecessary proactive
actions and rejecting required proactive actions
• Cost savings due to proactive process adaptation
• No, in 47.5% of the cases
• Yes, in 52.5% of the cases
• Cost savings due to considering
reliability estimates
• No, in 17,1% of the cases
• Yes, in 82.9% of the cases
14CAiSE 2017, Essen
Cost savings
Frequency
Savings from 2%
to 54%,
14% on average

15. Agenda
1. Motivation
2. Experimental Design
3. Main Results
4. Conclusions
CAiSE 2017, Essen

16. Conclusions
Observation
• Considering reliability estimates can have a positive effect on
costs – but not in all situations!
Open Questions
• How to upfront determine these situations?
• How to provide further information for decision making (e.g.,
risk = probability x severity)?
• How to consider different shapes of costs (penalties and
adaptation costs)?
16CAiSE 2017, Essen

17. Thanks
CAiSE 2017, Essen 17
…the EFRE co-financed operational
program NRW.Ziel2
http://www.lofip.de
…the EU’s Horizon 2020 research and
innovation programme under Objective
ICT-15 ‘Big Data PPP: Large Scale Pilot
Actions ‘
http://www.transformingtransport.eu
Research leading to these results has received
funding from…