A python based grid computing project. With process work-flow built in, deploy and manage simple through to complex business processes across a distributed network of dedicated or on demand commodity computers. Run command line apps, native Python, Java and .Net code.
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
ScaleFast Grid And Flow
1. Grid and Flow By Robert Betts robert.betts@scalefast.com
2. The Offering A distributed, stable and well synchronised platform for grid computing. A methodology, toolset and library for deploying an efficient, co-ordinated and parallel platform.
3. Current Operational Challenges Processing of large datasets open to high rates of failure Processing takes a long time to complete Processing is often executed sequentially Technology bottlenecks e.g. 32 bit software often only supports between 2 and 3 GB RAM 64 bit processes can hog available resources Most tools can’t exploit multi core configurations Dedicated hardware allocated to accommodate the maximum processing load Difficult to audit and trace processing problems
4. With ScaleFast you can ... Centralise all business processes Define hierarchical processes with step inter-dependencies Parallelise the running of processes and steps Re-run failed processes from any point. Automatically split large process steps Make use of multi core/processor computers Distribute jobs across multiple computers Make use of user workstations and other idle computing resources
5. ScaleFast Grid Distributed Computing Grid A distributor and worker nodes Implements map/reduce Workers can run on user workstations or dedicated infrastructure Can be easily deployed to a cloud platform Supports the native running of Python, Java and .Net
6. ScaleFast Flow Process Workflow Engine Processes are made up of individual jobs which have inter-dependencies The output of one job can be the input of the next job Processes have notifications based on success or failure Flow has a built in scheduler which can be triggered by: Time with multiple time zone support User Interface API Processes can be restarted from any point of failure Processes can be made up of sub processes
7. Common Use Cases Reporting and data processing Stabilising processes that fail due to resource constraints Speeding up processes that take a long time to run Improve and/or balance resource utilisation Process orchestration and scheduling Co-ordinating processes with event based synchronisation Parameter and data flow between process steps Centralisation and versioning of processes Reducing support administration with full process auditing General processing and application development Any application/process that would benefit from parallelism Risk Management and PL Processing Distributed Computations
33. SCALEFAST Architecture Flow stores, versions and schedules workflows which are predefined and synchronised grid jobs. Grid Clients are any processes able to submit grid jobs. Grid Distributor receives job requests and maps the reduced jobs as tasks across workers. Grid Workers request and process job tasks Flow Grid Client 0 Grid Client ... Grid Client Q Grid Distributor Worker 0 Worker ... Worker X Server 0 Runner Runner Runner Server ... Server N Local disk Shared Storage
34. Flow GUI A simple process example with 3 steps Processes can have multiple branch dependencies e.g. 1 to many and many to 1 Processes can be build up from sub processes Flow highlights the status of the individual steps By clicking on a step, you are redirected to the Grid for further details. Processes can be paused and restarted. On a processes failure, it can be restarted at any step in the process.
35. Grid Job Details GUI Parameters, status and details of a grid job Individual tasks can be drilled down into Stderr and Stdout out can be accessed and queried across all tasks Input parameters, context and output visible at job or task level
36. Grid Summary GUI High level view on the Grid status and activity View active worker nodes View job activity and history
Hinweis der Redaktion
Scheduling Tools: Active Batch
Issues in the top grouping can be addressed by tools like Active Batch
Support staff can now see in very granular process details where process failed.It is no easier to determine the cause of a failure, was it: Resource issues Bad static data Bugs in code
There is also an instance of flow that can be easily integrated into an Enterprise Workflow/Automation Application