Benchmarking Workflow Engines: Open Research Challenges - Presentation of BTW 2015
•Als PPTX, PDF herunterladen•
3 gefällt mir•1,381 views
Workflow Management Systems (WfMS) provide platforms for delivering complex service-oriented applications that need to satisfy enterprise-grade quality of service requirements such as dependability and scalability. The performance of these applications largely depends on the performance of the WfMS supporting them. Com- paring the performance of different WfMSs and optimizing their configuration requires that appropriate benchmarks are made available. In this position paper we make the case for benchmarking the performance of WfMSs that are compliant with the Busi- ness Process Model and Notation 2.0 standard and explore most of the challenges that one must tackle when constructing such benchmarks by Pautasso C., Ferme V., Roller H.D., Leymann F., Skouradaki M.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Benchmarking Workflow Engines: Open Research Challenges - Presentation of BTW 2015
Ähnlich wie Benchmarking Workflow Engines: Open Research Challenges - Presentation of BTW 2015 (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Benchmarking Workflow Engines: Open Research Challenges - Presentation of BTW 2015
- 1. Research Dieter H. Roller, Frank Leymann, Marigianna Skouradaki, Institute of Architecture of Application Systems {skouradaki, dieter.h.roller,leymann}@iaas.uni-stuttgart.de Cesare Pautasso, Vincenzo Ferme Faculty of Informatics University of Lugano Switzerland {firstname.lastname@usi.ch} Towards Workflow Engine Benchmarking: Open Research Challenges
- 2. 22 Research © Marigianna Skouradaki Motivation *the appearance and company selection are random
- 3. 33 Research © Marigianna Skouradaki Agenda Performance Factors Logistic Challenges on Benchmarking Technical Challenges on Benchmarking Project Status Conclusions & Future Work
- 4. 44 Research © Marigianna Skouradaki What is a Workflow Engine? 4.Navigator 3.Web Services 6.Application Server Workflow Engine 5.Instance Database 2.Users & IT tools … 1.Process Models
- 6. 66 Research © Marigianna Skouradaki Challenge: Process Model Collection More than 8.000 Process Models
- 7. 77 Research © Marigianna Skouradaki Challenge: Process Model Collection - Data and Timing Information Load Distribution?
- 8. 88 Research © Marigianna Skouradaki Challenge: Process Model Collection - Data Flow and Timing Information 80% 20% Process Logs Time for Completion
- 9. 99 Research © Marigianna Skouradaki Challenge: Synthesizing Benchmark Flows BPMN 2.0 Process Models Collection Reflect Diversity Usage Patterns of BPMN 2.0 Compliant with Workflow Engines Benchmark Models General vs. Domain Specific
- 10. 1010 Research © Marigianna Skouradaki Challenge: What to Measure? • Throughput • Response Time • Required Memory • Power Consumption Workflow Engine request Workflow Engine response • Network Traffic • SLA violations • Failure Rate • Recovery Time • … client
- 12. 1212 Research © Marigianna Skouradaki Challenge: Eliminate Impact of External Factors Workflow Engine requestclient response monitoring
- 13. 1313 Research © Marigianna Skouradaki Challenge: Prevent System Overloading Workflow Engine request client request request Adjust delay rate between requests! request request
- 14. 1414 Research © Marigianna Skouradaki Challenge: Reliability, Recovery, Robustness Workflow Engine request client request request request request Control and measure failure?
- 15. 1515 Research © Marigianna Skouradaki Challenge: System Internal Load Optimization request client response Workflow Engine Throughput?
- 16. 1616 Research © Marigianna Skouradaki Challenge: Long Running Processes request client response Workflow Engine 3 months Benchmark in reasonable time?
- 18. 1818 Research © Marigianna Skouradaki Collected Process Models 1960 3168 2177 608 21 427 Models (Non-Executable Files) .pnml .bpmn2 .bpmn .yawl .bpel .EPC Total: 8363 IBM Industry Models BPM Academic Initiative Research Projects
- 19. © Marigianna Skouradaki 19 Research Generation of Realistic Workload BPMN 2.0 Process Models Collection Fragments Repository Workload Mix Graph Matching Selection Criteria Composition Criteria 80% 20%
- 20. 20 Research Workflow Engine Enabling the Benchmark Execution harness Web ServiceDBMS 1. Flexible Deployment 2. Flexible HW Resources 3. Frozen Initial Condition Faban Drivers Docker Containers Servers A B C D Load Functions Vincenzo Ferme
- 21. 21 Research 1. Automatically deploy and start the benchmark environment; Enabling the Benchmark Execution harness Faban Drivers 3. Determine when the benchmark ends;Faban + Workflow Engine DBMS A B C D MONITOR Instance Database COLLECTORS 2. Automatically deploy the workload mix; Vincenzo Ferme 4. Collect the execution and process logs.
- 22. 2222 Research © Marigianna Skouradaki Conclusions & Future Work BenchFlow Project: http://www.iaas.uni-stuttgart.de/forschung/projects/benchflow.php Benchmark long running processes? Throughput in asynchronous processes execution? Optimal delay rate in requests? Release the first prototype for Workload Mix Synthesizer First experiments with KPIs definition and computation Collect more processes Collect process execution logs
Hinweis der Redaktion
- Hello, my name is Marigianna Skouradaki. I am working for IAAS institute, university of Stuttgart, and today I am going to present to you a work done in collaboration with University of Lugano. Our work will discuss the challenges of Workflow Engine Benchmarking
- There are various Workflow Engines available, and our goal is to compare and rank them. As ranking is always related to a specific context, i.e. criteria, metrics, application scenario and/or domain, a process is needed to be followed to enable comparison This is what benchmarking is for and therefore in the BenchFlow Project our meta goal is to develop the first standard benchmark for Workflow Engines
- This brings me to today’s agenda. First I will discuss the environment of the Workflow Engine, and the factors that affect its performance. Then I will analyze the logistic and technical challenges that we have recognized. I will present the current project status. And finally I will conclude and discuss the plans for future work
- The WfMS takes as input process models that are basically a visual representation of a process. Process models can be expressed in various definition languages, however for this work, and for reasons that we will explain later we focus on process models expressed in bpmn 2 language. The wfms interacts with external entities. It interacts with external web services as specified from the process models, and with users and IT tools. Services and IT users play the role of a task requestor, or wait for the response of a task. The heart of the workflow engine is the navigator. It typically translates the process models into an intermediate form that can be executed efficiently. To do so the WfE uses a database, called instance database that stores valuable information for run and build time. For example it stores the process models, and their instances. For the efficient execution, the wfe may also create pool worker, that will run concurrently to execute independent process instances. For this work the wfe relies on an underlying middleware the application server that efficiently handles the synchronous and asynchronous interactions. The elements described in this slide are basically the factors that affect the wfe’s performance The structure and complexity of the business processes that are being carried out, such as the level of parallelism or the handling of data The interactions that the business process carries out such as responding to a request from a client or invoking a Web Service The architecture of the workflow engine, the exploitation of the underlying middleware, such as transaction handling or message processing the management of data in a persistent store for the reliable execution of long-running processes, and the load that the workflow management system must sustain Then say about the logistic and technical challenges…
- There are logistic challenges bla bla and then we seen the more logistic technical challenges
- The first recognized challenge was to collect as many real-world process models as possible. Only then it is possible to come up with a benchmark that correctly affects the real wold. We have therefore contacted tens of companies and research institutes and requested process models. The usual case is that the different stakeholders are not willing to share the process models because it will reveal a lot of information about their company and product. However, we have managed to gather thousands of process models. I will later discuss the unity and resources of our process model collection.
- Although we managed to collect a big variety of process models, important information is still missing. For example we do not have any information about a range of data that are given as input to the process models. Or on how many instances a process model is expected to have. Timinig information is also missing. We do not have any statistics on how long does it take for a web service to complete, or if there are specific periods where we expect more load for the workflow engine
- We miss information on the data flow of the process model. For example we would like to know what is a realistic percentage to follow a specific path of a process model. This information as well as timing information can be available by applying process mining to real-life Process Logs which we are also currently missing.
- Since we have currently a collection of process models that reflect a diversity of applications. This reflection of diversity needs to be expressed by the synthetic process models that will go into the benchmark. After analyzing the appropriate data we need to make a series of decisions to decide which these models will be For example we need to answer: Which are the most frequent usage patterns in the real life, if the synthetic process models are compliant with all the workflow engines, and finally if we will choose general or domain specific process models
- The last logistical challenge is to determine what metrics to measure. The benchmark needs to be a single number that allows the comparison of different engines. There is however a large number of metrics that can be observed for performance and we need to select some of them and aggregate them into a meaningful number.
- Moving on to the technical challenges..
- When someone thinks of a database benchmark this is particularly simple The client sends a requests against the database and the test is completed when all of the requests have been completed. However in the case of workflow engines it is not that simple because the workflow engine interacts with external partners such as web services, users and other IT tools. The challenge here is to isolate the non-Workflow engine resources from the benchmark results and make sure that only the Workflow Engine is benchmarked. Finally we need to consider that many workflow engines include monitoring that allows clients to track the real-time of the process model execution. We need ot take into account the impact of the monitoring-related features to the execution of benchmarks, and if possible exclude it.
- In the real world scenario the requests can be either synchronous or asynchronous. In the case of asynchronous requests the client will fire the request but we do not know when it will receive the response. Therefore there might be the case were the client has sent more requests that the workflow engine can process. Here the challenge is to automatically adjust the delay rate between the requests that will stress the engine but will not overload it. Capacity testing vs. Performance Testing
- Although we want to avoid the workflow engine failure in some benchmark cases, there are also the cases where we need to cause it, in order ot measure reliability, recovery and robustness. In this case we need to define methods for controlled failure injection experiments in order to measure the failure rate of the system as it is observed by the client
- As discussed before a request to a workflow engine can also be asynchronous. There are many cases then were the workflow engine will shift the request for a time of the day where the load is lower. The challenge here is to accommodate this type of optimized behavior to measure the throughput
- Finally some process models can be long running. This means that their execution may take weeks or even years. The challenge here is to determine a way to simulate long-running processes and benchmark them without having to wait for years.
- We are working on defining a representative workload mix. For this we are defining a methodology to: extract reoccurring structures out of real-world processes by applying graph isomorphism AND select interesting reoccurring structures out of the obtained collection and glue them according to pre-defined composition criteria.
- Moving forward to the design of the benchmark environment, the load driver has to deal with engine-specific APIs and BPMN 2 customizations made by the engines. Each engine defines its own API at least to interact with Users and Web Services, to receive messages and to expose its functionalities to a client system. Given the Workload Mix, we have to define the Load Functions, not only to decide when to start a new process, but also to define the behavior of Users, Web Services and Events. We are using Faban, an open source framework for performance workload creation and execution, to define the load drivers and handle the benchmark execution. benchmark-harness is designed to make it easy to create simple suites of standalone benchmarks while avoiding some common pitfalls in benchmarking. AND We are using Docker to manage the deployment of the benchmark environment. Docker is a lightweight containerization solution to deploy applications. Among other features, Docker offers a flexible deployment mechanism and hardware resources configuration. Moreover it allows to freeze the initial condition of the benchmark environment enabling the reproducibility of the benchmark execution.
- We have also to deal with the asynchronous execution of processes. Given the presence of Users, Web Services and Events interacting with the Workflow Engine, the driver can only be aware of the starting time of a process, because it fires the start request. This means we have to look at the process execution logs and the instance database to gather information about the end time of started processes. We are integrating Faban and Docker, in order to: Automatically deploy and start the benchmark environment; Automatically deploy the workload mix on the Workflow Engine; And because the workload runs asynchronously, we also need to: Determine when the benchmark ends; AND Collect the execution and process logs at the end of the benchmark execution.
- As next steps of the BenchFlow project we plan to: Release the first simplified prototype of the Benchmark environment and of the Workload Mix synthesizer; Perform the first experiments with KPIs definition and computation; Gather more process models and process execution logs from industry and practitioners. You can learn more about the project on the link provided on the slide. Thank you for you attention.