Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore um University of Tartu and Apromore Pty Ltd
Can I Trust My Simulation
Model? Measuring the
Quality of Business Process
Simulation Models
David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2,
Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3
21st International Conference on Business Process
Management (BPM 2023)
1 University of Tartu, Estonia
2 University of Toronto, Canada
3 York University, Canada
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2
Introduction
Business Process Simulation (BPS)
3
BPS allows users to address “what-if” analysis questions.
What would be the cycle time of the process if the rate of arrival of new cases
doubles?
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS)
4
BPS models can be manually created by modeling experts.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS)
5
BPS models can be manually created by modeling experts.
Use of process mining techniques to automatically discover BPS
models from business process event logs.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS)
6
How to assess the quality of a BPS model?
Automatic assessment.
Useful to detect the sources of deviations.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7
Proposed Framework
Quality of a BPS model
8
How to assess the quality of a BPS model?
Comparing an event log with a BPS model.
Variety of different BPS models formats.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log
Quality of a BPS model
9
How to assess the quality of a BPS model?
Generate K simulated event logs.
Compare individually and report the average and confidence interval.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
9
Process event log
K simulated event logs
Quality of a BPS model
10
A BPS model can be very accurate in one aspect (e.g., control-flow), yet
very different in another (e.g., processing times).
Three main dimensions: control-flow, temporal, congestion.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11
Proposed Framework
Control-flow measures
Control-flow: Control-Flow Log
Distance
12
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs:
data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021)
Process event log Simulated event log
Control-flow: Control-Flow Log
Distance
13
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
Control-flow: Control-Flow Log
Distance
14
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
0
Control-flow: Control-Flow Log
Distance
15
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
0
0
0.2
Control-flow: Control-Flow Log
Distance
16
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
CFLD =
0+0+0.75+0+0.2
5
= 0.19
Control-flow: N-Gram Distance
17
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance
checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019)
Process event log Simulated event log
Control-flow: N-Gram Distance
18
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
19
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
20
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
21
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
22
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
0
1
2
3
4
5
6
_ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _
Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23
Proposed Framework
Temporal measures
Process event log Simulated event log
Temporal: Absolute Event
Distribution
24
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
Temporal: Absolute Event
Distribution
25
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Temporal: Absolute Event
Distribution
26
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
06-10-2022 10am – 11am
07-10-2022 11am – 12pm
Temporal: Absolute Event
Distribution
27
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Earth
mover's
distance
Temporal: Circadian Event
Distribution
28
Circadian Event Distribution (CED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, for each
day of the week.
How different they are distributed through each day of the week.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Monday
Tuesday
Wednesday
Thursday 10am – 11am
Friday 11am – 12pm
Temporal: Circadian Event
Distribution
29
Circadian Event Distribution (CED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, for each
day of the week.
How different they are distributed through each day of the week.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Monday Monday
Temporal: Relative Event
Distribution
30
Relative Event Distribution (RED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, with
respect to the start of their case.
How different they are distributed within each process case.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
00:00:00 01:01:47
Temporal: Relative Event
Distribution
Relative Event Distribution (RED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, with
respect to the start of their case.
How different they are distributed within each process case.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32
Proposed Framework
Congestion measures
Congestion: Case Arrival Rate
33
Case Arrival Rate (CAR): given two event logs L1 and L2, distance
between how the case arrivals are distributed in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
06-10-2022 10am – 11am
Congestion: Case Arrival Rate
34
Case Arrival Rate (CAR): given two event logs L1 and L2, distance
between how the case arrivals are distributed in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Congestion: Cycle Time Distribution
35
Cycle Time Distribution (CTD): given two event logs L1 and L2, distance
between the distribution of cycle times in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
01:07:02
Congestion: Cycle Time Distribution
36
Cycle Time Distribution (CTD): given two event logs L1 and L2, distance
between the distribution of cycle times in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37
Evaluation
Evaluation
38
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ1: Are the proposed measures able to discern the impact of different
known modifications to a BPS model?
EQ2: Is the N-Gram Distance’s performance significantly different from
the CFLD’s performance?
No modifications
Control-flow
Gateway probabilities
Case arrival rate
Activity durations
Resource contention
Working calendars
Extraneous delays
Evaluation
39
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Evaluation
40
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ1: Are the proposed measures able to discern the impact of different
known modifications to a BPS model?
Evaluation
41
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ2: Is the N-Gram Distance’s performance significantly different from
the CFLD’s performance?
Kendall
rank
correlation
coefficient
1.0
Evaluation
42
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
43
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
4 real-life processes: each split into disjoint training and test.
Evaluation
44
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Automatically discover BPS model with SIMOD and Service Miner.
Evaluation
45
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
46
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
47
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
48
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
49
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ4: Does the 1-WD report the same insights in real-life scenarios as
the EMD?
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50
Conclusion
Conclusion
51
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Proposed a framework to measure the quality of a BPS model:
decomposing into three perspectives (control-flow, temporal, and
congestion), and defined measures for each of these perspectives.
The measures proved their ability to detect the alterations in their
corresponding perspectives.
Beyond capturing the quality of BPS model and identifying the sources of
discrepancies, the measures can also assist in eliciting areas for
improvement in these techniques.
The presented computationally efficient alternatives led to similar
conclusions.
Future Work
52
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Explore the applicability of the proposed measures to other process
mining problems, e.g., concept drift detection and variant analysis.
Studying how to assess the quality of BPS models in the context of
object-centric event logs.
Study other quality measures for BPS models adapted from the field of
generative machine learning, for example, by using a discriminative
model that attempts to distinguish between data generated by the
BPS model and real data.
1 von 52

Recomendados

Kerncompetenties voor de architect, informatiemanager en IT-governance-adviseur von
Kerncompetenties voor de architect, informatiemanager en IT-governance-adviseurKerncompetenties voor de architect, informatiemanager en IT-governance-adviseur
Kerncompetenties voor de architect, informatiemanager en IT-governance-adviseurDanny Greefhorst
3.7K views25 Folien
Introduction to Business Process Monitoring and Process Mining von
Introduction to Business Process Monitoring and Process MiningIntroduction to Business Process Monitoring and Process Mining
Introduction to Business Process Monitoring and Process MiningMarlon Dumas
1.7K views207 Folien
BPMN on One Page von
BPMN on One PageBPMN on One Page
BPMN on One PageMohamed Zakarya Abdelgawad
1.6K views217 Folien
Enterprise Architecture Framework: Chase Global Bank von
Enterprise Architecture Framework: Chase Global BankEnterprise Architecture Framework: Chase Global Bank
Enterprise Architecture Framework: Chase Global BankHampus Ahlqvist
12.7K views30 Folien
Process architecture - Part II von
Process architecture - Part IIProcess architecture - Part II
Process architecture - Part IIMarcello La Rosa
12.7K views30 Folien
MES systems von
MES systemsMES systems
MES systemsBarry Curry
2.6K views17 Folien

Más contenido relacionado

Was ist angesagt?

IT Performance Measurement using IT Governance Metric von
IT Performance Measurement using IT Governance MetricIT Performance Measurement using IT Governance Metric
IT Performance Measurement using IT Governance MetricPECB
7.5K views31 Folien
CobiT, Val IT & Balanced Scorecards von
CobiT, Val IT & Balanced ScorecardsCobiT, Val IT & Balanced Scorecards
CobiT, Val IT & Balanced ScorecardsMichael Sim
9.3K views34 Folien
Business Process Modeling with BPMN 2.0 - Second edition von
Business Process Modeling with BPMN 2.0 - Second editionBusiness Process Modeling with BPMN 2.0 - Second edition
Business Process Modeling with BPMN 2.0 - Second editionGregor Polančič
73K views102 Folien
Demystifying Oracle Cloud ERP Financials von
Demystifying Oracle Cloud ERP FinancialsDemystifying Oracle Cloud ERP Financials
Demystifying Oracle Cloud ERP FinancialsPerficient, Inc.
2.2K views72 Folien
BPM (Business Process Management) Introduction von
BPM (Business Process Management) IntroductionBPM (Business Process Management) Introduction
BPM (Business Process Management) IntroductionIntegrify
27.7K views24 Folien
Business Process Management Training | By ex-Deloitte & McKinsey Consultants von
Business Process Management Training | By ex-Deloitte & McKinsey ConsultantsBusiness Process Management Training | By ex-Deloitte & McKinsey Consultants
Business Process Management Training | By ex-Deloitte & McKinsey ConsultantsAurelien Domont, MBA
4.9K views44 Folien

Was ist angesagt?(20)

IT Performance Measurement using IT Governance Metric von PECB
IT Performance Measurement using IT Governance MetricIT Performance Measurement using IT Governance Metric
IT Performance Measurement using IT Governance Metric
PECB 7.5K views
CobiT, Val IT & Balanced Scorecards von Michael Sim
CobiT, Val IT & Balanced ScorecardsCobiT, Val IT & Balanced Scorecards
CobiT, Val IT & Balanced Scorecards
Michael Sim9.3K views
Business Process Modeling with BPMN 2.0 - Second edition von Gregor Polančič
Business Process Modeling with BPMN 2.0 - Second editionBusiness Process Modeling with BPMN 2.0 - Second edition
Business Process Modeling with BPMN 2.0 - Second edition
Gregor Polančič73K views
Demystifying Oracle Cloud ERP Financials von Perficient, Inc.
Demystifying Oracle Cloud ERP FinancialsDemystifying Oracle Cloud ERP Financials
Demystifying Oracle Cloud ERP Financials
Perficient, Inc.2.2K views
BPM (Business Process Management) Introduction von Integrify
BPM (Business Process Management) IntroductionBPM (Business Process Management) Introduction
BPM (Business Process Management) Introduction
Integrify27.7K views
Business Process Management Training | By ex-Deloitte & McKinsey Consultants von Aurelien Domont, MBA
Business Process Management Training | By ex-Deloitte & McKinsey ConsultantsBusiness Process Management Training | By ex-Deloitte & McKinsey Consultants
Business Process Management Training | By ex-Deloitte & McKinsey Consultants
Oracle Fusion HCM Presentation von Feras Ahmad
Oracle Fusion HCM PresentationOracle Fusion HCM Presentation
Oracle Fusion HCM Presentation
Feras Ahmad650 views
What is Enterprise Service Management and How You Can Get Started von Cherwell Software
What is Enterprise Service Management and How You Can Get StartedWhat is Enterprise Service Management and How You Can Get Started
What is Enterprise Service Management and How You Can Get Started
Cherwell Software831 views
Operational Performance Management PowerPoint Presentation Slides von SlideTeam
Operational Performance Management PowerPoint Presentation Slides Operational Performance Management PowerPoint Presentation Slides
Operational Performance Management PowerPoint Presentation Slides
SlideTeam257 views
From Conceptual to Executable BPMN Process Models A Step-by-Step Method von Marlon Dumas
From Conceptual to Executable BPMN Process Models A Step-by-Step MethodFrom Conceptual to Executable BPMN Process Models A Step-by-Step Method
From Conceptual to Executable BPMN Process Models A Step-by-Step Method
Marlon Dumas25.2K views
Business Architecture the Key to Enterprise Transformation von Mike Walker
Business Architecture the Key to Enterprise TransformationBusiness Architecture the Key to Enterprise Transformation
Business Architecture the Key to Enterprise Transformation
Mike Walker46.8K views
How to use BPMN* for modelling business processes von Alexander SAMARIN
How to use BPMN* for modelling business processesHow to use BPMN* for modelling business processes
How to use BPMN* for modelling business processes
Alexander SAMARIN15.7K views
Odoo Presentasi.pptx von Yudi512144
Odoo Presentasi.pptxOdoo Presentasi.pptx
Odoo Presentasi.pptx
Yudi512144798 views
Oracle Project Financial Management Cloud in 9 Weeks von Perficient, Inc.
Oracle Project Financial Management Cloud in 9 WeeksOracle Project Financial Management Cloud in 9 Weeks
Oracle Project Financial Management Cloud in 9 Weeks
Perficient, Inc.2.8K views
Comparision of ERP Vendors von Gaurav Rane
Comparision of ERP VendorsComparision of ERP Vendors
Comparision of ERP Vendors
Gaurav Rane1.7K views
Build a Financial Data Hub von Workday, Inc.
Build a Financial Data HubBuild a Financial Data Hub
Build a Financial Data Hub
Workday, Inc.301 views
ITIL implementation and Service Management Best Practices – useful informatio... von SriramITISConsultant
ITIL implementation and Service Management Best Practices – useful informatio...ITIL implementation and Service Management Best Practices – useful informatio...
ITIL implementation and Service Management Best Practices – useful informatio...
Frepple-Odoo integration - Odoo experience days 2016 von Johan De Taeye
Frepple-Odoo integration - Odoo experience days 2016Frepple-Odoo integration - Odoo experience days 2016
Frepple-Odoo integration - Odoo experience days 2016
Johan De Taeye14.8K views

Similar a Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Process Capability von
Process CapabilityProcess Capability
Process CapabilityThomas Abraham
7K views41 Folien
Process Mining and Predictive Process Monitoring von
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process MonitoringMarlon Dumas
3.3K views40 Folien
Statistical Process Control - SPC von
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPCPrasenjit Puri
153 views25 Folien
Six Sigma Dfss Application In Data Accarucy von
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucyxyhfun
529 views28 Folien
Process Mining and Predictive Process Monitoring in Apromore von
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in ApromoreMarlon Dumas
713 views34 Folien
A process to improve the accuracy of mk ii fp to cosmic charles symons von
A process to improve the accuracy of mk ii fp to cosmic    charles symonsA process to improve the accuracy of mk ii fp to cosmic    charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symonsIWSM Mensura
349 views30 Folien

Similar a Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models(20)

Process Mining and Predictive Process Monitoring von Marlon Dumas
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process Monitoring
Marlon Dumas3.3K views
Statistical Process Control - SPC von Prasenjit Puri
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPC
Prasenjit Puri153 views
Six Sigma Dfss Application In Data Accarucy von xyhfun
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucy
xyhfun529 views
Process Mining and Predictive Process Monitoring in Apromore von Marlon Dumas
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in Apromore
Marlon Dumas713 views
A process to improve the accuracy of mk ii fp to cosmic charles symons von IWSM Mensura
A process to improve the accuracy of mk ii fp to cosmic    charles symonsA process to improve the accuracy of mk ii fp to cosmic    charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symons
IWSM Mensura349 views
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs von Marlon Dumas
Multi-Perspective Comparison of Business Processes Variants Based on Event LogsMulti-Perspective Comparison of Business Processes Variants Based on Event Logs
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs
Marlon Dumas557 views
Advantages Of Linear Programming Models von Brenda Torres
Advantages Of Linear Programming ModelsAdvantages Of Linear Programming Models
Advantages Of Linear Programming Models
Brenda Torres3 views
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op... von IRJET Journal
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
IRJET Journal41 views
Comprehensive acquisition management methodology rev6 von clifpt
Comprehensive acquisition  management methodology   rev6Comprehensive acquisition  management methodology   rev6
Comprehensive acquisition management methodology rev6
clifpt44 views
Gestión de la calidad sem 2 von youffre
Gestión de la calidad sem 2Gestión de la calidad sem 2
Gestión de la calidad sem 2
youffre118 views
Practical Guides on Programming with Big Number Library in Scientific Researches von theijes
Practical Guides on Programming with Big Number Library in Scientific ResearchesPractical Guides on Programming with Big Number Library in Scientific Researches
Practical Guides on Programming with Big Number Library in Scientific Researches
theijes31 views
Unbiased, Fine-Grained Description of Processes Performance from Event Data von Vadim Denisov
Unbiased, Fine-Grained Description of Processes Performance from Event DataUnbiased, Fine-Grained Description of Processes Performance from Event Data
Unbiased, Fine-Grained Description of Processes Performance from Event Data
Vadim Denisov34 views
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING von IJCI JOURNAL
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKINGBUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
IJCI JOURNAL2 views
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN... von IAEME Publication
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
IAEME Publication267 views
Decisions in a supply chain modeling for comparative evaluation strategies in... von IAEME Publication
Decisions in a supply chain modeling for comparative evaluation strategies in...Decisions in a supply chain modeling for comparative evaluation strategies in...
Decisions in a supply chain modeling for comparative evaluation strategies in...
IAEME Publication294 views
Software Architecture in the age of Cloud Computing von Jaroslav Gergic
Software Architecture in the age of Cloud ComputingSoftware Architecture in the age of Cloud Computing
Software Architecture in the age of Cloud Computing
Jaroslav Gergic524 views

Más de Marlon Dumas

Walking the Way from Process Mining to AI-Driven Process Optimization von
Walking the Way from Process Mining to AI-Driven Process OptimizationWalking the Way from Process Mining to AI-Driven Process Optimization
Walking the Way from Process Mining to AI-Driven Process OptimizationMarlon Dumas
670 views27 Folien
Discovery and Simulation of Business Processes with Probabilistic Resource Av... von
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Marlon Dumas
63 views15 Folien
Business Process Optimization: Status and Perspectives von
Business Process Optimization: Status and PerspectivesBusiness Process Optimization: Status and Perspectives
Business Process Optimization: Status and PerspectivesMarlon Dumas
544 views18 Folien
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... von
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Marlon Dumas
35 views19 Folien
Augmented Business Process Management von
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process ManagementMarlon Dumas
1.2K views23 Folien
Process Mining and Data-Driven Process Simulation von
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process SimulationMarlon Dumas
299 views29 Folien

Más de Marlon Dumas(20)

Walking the Way from Process Mining to AI-Driven Process Optimization von Marlon Dumas
Walking the Way from Process Mining to AI-Driven Process OptimizationWalking the Way from Process Mining to AI-Driven Process Optimization
Walking the Way from Process Mining to AI-Driven Process Optimization
Marlon Dumas670 views
Discovery and Simulation of Business Processes with Probabilistic Resource Av... von Marlon Dumas
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Marlon Dumas63 views
Business Process Optimization: Status and Perspectives von Marlon Dumas
Business Process Optimization: Status and PerspectivesBusiness Process Optimization: Status and Perspectives
Business Process Optimization: Status and Perspectives
Marlon Dumas544 views
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... von Marlon Dumas
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Marlon Dumas35 views
Augmented Business Process Management von Marlon Dumas
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process Management
Marlon Dumas1.2K views
Process Mining and Data-Driven Process Simulation von Marlon Dumas
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process Simulation
Marlon Dumas299 views
Modeling Extraneous Activity Delays in Business Process Simulation von Marlon Dumas
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process Simulation
Marlon Dumas148 views
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints von Marlon Dumas
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints
Marlon Dumas92 views
Robotic Process Mining von Marlon Dumas
Robotic Process MiningRobotic Process Mining
Robotic Process Mining
Marlon Dumas333 views
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable? von Marlon Dumas
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Marlon Dumas49 views
Learning Accurate Business Process Simulation Models from Event Logs via Auto... von Marlon Dumas
Learning Accurate Business Process Simulation Models from Event Logs via Auto...Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Marlon Dumas419 views
Process Mining: A Guide for Practitioners von Marlon Dumas
Process Mining: A Guide for PractitionersProcess Mining: A Guide for Practitioners
Process Mining: A Guide for Practitioners
Marlon Dumas141 views
Optimización de procesos basada en datos von Marlon Dumas
Optimización de procesos basada en datosOptimización de procesos basada en datos
Optimización de procesos basada en datos
Marlon Dumas386 views
Process Mining and AI for Continuous Process Improvement von Marlon Dumas
Process Mining and AI for Continuous Process ImprovementProcess Mining and AI for Continuous Process Improvement
Process Mining and AI for Continuous Process Improvement
Marlon Dumas207 views
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction von Marlon Dumas
Prescriptive Process Monitoring for Cost-Aware Cycle Time ReductionPrescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Marlon Dumas362 views
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio... von Marlon Dumas
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Marlon Dumas652 views
On the Road to AI-Infused Process Execution von Marlon Dumas
On the Road to AI-Infused Process ExecutionOn the Road to AI-Infused Process Execution
On the Road to AI-Infused Process Execution
Marlon Dumas270 views
Process Mining 2.0: From Insights to Actions von Marlon Dumas
Process Mining 2.0: From Insights to ActionsProcess Mining 2.0: From Insights to Actions
Process Mining 2.0: From Insights to Actions
Marlon Dumas742 views
Process Mining in Action: Self-service data science for business teams von Marlon Dumas
Process Mining in Action: Self-service data science for business teamsProcess Mining in Action: Self-service data science for business teams
Process Mining in Action: Self-service data science for business teams
Marlon Dumas407 views

Último

OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an... von
OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...
OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...StatsCommunications
7 views26 Folien
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx von
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptxDataScienceConferenc1
10 views16 Folien
apple.pptx von
apple.pptxapple.pptx
apple.pptxhoneybeeqwe
6 views15 Folien
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf von
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf10urkyr34
7 views259 Folien
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init... von
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...DataScienceConferenc1
5 views18 Folien
[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx von
[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx
[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptxDataScienceConferenc1
5 views15 Folien

Último(20)

OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an... von StatsCommunications
OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...
OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx von DataScienceConferenc1
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf von 10urkyr34
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf
10urkyr347 views
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init... von DataScienceConferenc1
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...
[DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh ... von DataScienceConferenc1
[DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh ...[DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh ...
[DSC Europe 23] Predrag Ilic & Simeon Rilling - From Data Lakes to Data Mesh ...
[DSC Europe 23] Matteo Molteni - Implementing a Robust CI Workflow with dbt f... von DataScienceConferenc1
[DSC Europe 23] Matteo Molteni - Implementing a Robust CI Workflow with dbt f...[DSC Europe 23] Matteo Molteni - Implementing a Robust CI Workflow with dbt f...
[DSC Europe 23] Matteo Molteni - Implementing a Robust CI Workflow with dbt f...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo... von DataScienceConferenc1
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
OPPOTUS - Malaysians on Malaysia 3Q2023.pdf von Oppotus
OPPOTUS - Malaysians on Malaysia 3Q2023.pdfOPPOTUS - Malaysians on Malaysia 3Q2023.pdf
OPPOTUS - Malaysians on Malaysia 3Q2023.pdf
Oppotus27 views
Product Research sample.pdf von AllenSingson
Product Research sample.pdfProduct Research sample.pdf
Product Research sample.pdf
AllenSingson33 views
Chapter 3b- Process Communication (1) (1)(1) (1).pptx von ayeshabaig2004
Chapter 3b- Process Communication (1) (1)(1) (1).pptxChapter 3b- Process Communication (1) (1)(1) (1).pptx
Chapter 3b- Process Communication (1) (1)(1) (1).pptx
ayeshabaig20048 views
[DSC Europe 23] Ales Gros - Quantum and Today s security with Quantum.pdf von DataScienceConferenc1
[DSC Europe 23] Ales Gros - Quantum and Today s security with Quantum.pdf[DSC Europe 23] Ales Gros - Quantum and Today s security with Quantum.pdf
[DSC Europe 23] Ales Gros - Quantum and Today s security with Quantum.pdf
Dr. Ousmane Badiane-2023 ReSAKSS Conference von AKADEMIYA2063
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceDr. Ousmane Badiane-2023 ReSAKSS Conference
Dr. Ousmane Badiane-2023 ReSAKSS Conference
AKADEMIYA20635 views
Ukraine Infographic_22NOV2023_v2.pdf von AnastosiyaGurin
Ukraine Infographic_22NOV2023_v2.pdfUkraine Infographic_22NOV2023_v2.pdf
Ukraine Infographic_22NOV2023_v2.pdf
AnastosiyaGurin1.4K views
Best Home Security Systems.pptx von mogalang
Best Home Security Systems.pptxBest Home Security Systems.pptx
Best Home Security Systems.pptx
mogalang9 views
CRIJ4385_Death Penalty_F23.pptx von yvettemm100
CRIJ4385_Death Penalty_F23.pptxCRIJ4385_Death Penalty_F23.pptx
CRIJ4385_Death Penalty_F23.pptx
yvettemm1007 views
Listed Instruments Survey 2022.pptx von secretariat4
Listed Instruments Survey  2022.pptxListed Instruments Survey  2022.pptx
Listed Instruments Survey 2022.pptx
secretariat452 views
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx von DataScienceConferenc1
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

  • 1. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
  • 2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
  • 3. Business Process Simulation (BPS) 3 BPS allows users to address “what-if” analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 4. Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 5. Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 6. Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 7. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
  • 8. Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
  • 9. Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
  • 10. Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
  • 11. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
  • 12. Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
  • 13. Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
  • 14. Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
  • 15. Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
  • 16. Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
  • 17. Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019) Process event log Simulated event log
  • 18. Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 19. Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 20. Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 21. Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 22. Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
  • 23. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
  • 24. Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 25. Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 26. Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am 07-10-2022 11am – 12pm
  • 27. Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
  • 28. Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am – 11am Friday 11am – 12pm
  • 29. Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
  • 30. Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
  • 31. Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 32. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
  • 33. Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am
  • 34. Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 35. Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
  • 36. Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 37. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
  • 38. Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
  • 39. Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 40. Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
  • 41. Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? Kendall rank correlation coefficient 1.0
  • 42. Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 43. Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
  • 44. Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
  • 45. Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 46. Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 47. Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 48. Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 49. Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
  • 50. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
  • 51. Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
  • 52. Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.

Hinweis der Redaktion

  1. So, the first thing we need to know is, what is business process simulation?
  2. BPS aims to replicate the execution of a process, to mimic the behavior of the process, in a certain scenario (set of resources, etc.) analyzing its performance (KPIs) This allows users… The starting point is a BPS model… A process model annotated with a set of simulation parameters that define the scenario (resources, calendars, activity durations…). [NEXT]
  3. BPS models may be manually created based on information collected via interviews or empirical observations.. Or [NEXT…]
  4. they may be automatically discovered from execution data recorded in process-aware information systems (event logs) Regardless of the origin, a key question when using a BPS model is… [NEXT]
  5. how to assess its quality? Several approaches have been proposed to address this problem. However, these approaches are either manual and qualitative or they produce a single number that does not allow one to identify the source(s) of deviations between the BPS model and the observed reality
  6. First we need to decide what to compare when assessing the quality of a BPS model. What we are comparing is a BPS model, with a PROCESS What we usually have is…
  7. …an event log! Now, the first thing we asked ourselves was: should we compare a BPS model against a event log? But it is true that BPS models do not follow a standard structure… They can be formed by queue systems, but less of more models (resources, more complex waiting times), and they will change during time with new research. Thus, what we can do is simulate an event log out of the BPS model
  8. and compare log to log. K runs and compute the avg and conf int
  9. Abstract event logs into time-series or histograms and compare them
  10. We have two event logs, we are focusing on the control-flow, so the first step is to… [NEXT]
  11. obtain the activity sequences of each event log. Then, we compute the Damerau-Levenshtein (string edit distance) distance between each pair of cases… [NEXT]
  12. For exampe, [comment examples], we repeat this for each case Once we have all the pairings computed, we compute the matching between cases of one log to another (such as each case in one log is matched to one case in the other event log, with no repetitions… [NEXT]
  13. While minimizing the sum of distances using the Hungarian algorithm for optimal alignment. Finally, the CFLD measure is the average of these distances… [NEXT]
  14. The computational complexity of computing the DL-distance for all possible pairings is O(N2 ×MTL3) where N is the number of traces in the logs (assuming both logs have an equal number of cases, which holds in our setting) and MTL is the maximum trace length. Since all pairings are put into a matrix to compute the optimal alignment of cases (the one that minimizes the total sum of distances), CFLD’s memory complexity is quadratic on the number of cases. The optimal alignment of traces using the Hungarian algorithm has a cubic complexity on the number of cases.
  15. In the same way than for the CFLD, we are focusing on the control-flow, so the first step is to obtain the activity sequences of each event log. Leemans et al. measure the quality of a stochastic process model by mapping the model and a log to their Directly-Follows Graph (DFG), viewing each DFG as a histogram, and measuring the distance between these histograms. We note that the histogram of 2-grams of a log is equal to the histogram of its DFG. Given this observation, we generalize the approach of to n-grams, noting that the histogram of n-grams of a log is equal to the (n-1)th-Markovian abstraction of the log. Then… [NEXT]
  16. Let’s assume a size of N=3, so the N-grams are 3-grams (sequences of three activites). We compute all 3-grams observed in both logs, considering two dummy activities in the start and end of each trace. Then… [NEXT]
  17. We measure the frequency of each N-gram in each log…
  18. We measure the frequency of each N-gram in each log…
  19. And compute the sum of absolute differences between them, normalized by the sum of frequencies of all n-grams (value between 0-1). NGD is considerably more efficient than CFLD, as the construction of the histogram of n-grams is linear on the number of events in the log, and the same goes for computing the differences between the n-gram histograms.
  20. For the temporal measures, we first do the opposite of the control-flow, we abstract from the control-flow information… [NEXT]
  21. retaining only the events (in this case start and end)… [NEXT]
  22. Then we discretize these events into bins of 1h in the following way. Obtaining a time-series with the number of events happening in each hour of the process timeline.
  23. Once we have the temporal distribution (not a probabilistic distribution, but just the events occurring in the timeline), we compare both time-series with the EMD to measure the distance. We measure the trend.
  24. The same process is followed for the next measure, but in this case discretized to weekdays. In this way, we measure the seasonality of the events happening in the process.
  25. The temporal distribution of events of each day of the week is compared and then we compute the average distance of the 7 days.
  26. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  27. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  28. For the case arrival rate we want to measure how different the arrival of cases is. So, we retain only the events denoting the arrival of each case (start of first activity instance). Then we build the distribution in the same way than the previous metrics… [NEXT]
  29. and compare with EMD.