SlideShare ist ein Scribd-Unternehmen logo

Ciel universal distributed execution engine

•
•
Save Manos
Save Manos

Presentation of the CIEL framework as described in the paper: "CIEL: a universal execution engine for distributed data-flow computing"

Technologie
1 von 40
Downloaden Sie, um offline zu lesen
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . CIEL universal distributed execution engine . .. . . Presenter: Emmanouil Dimogerontakis @{AdvDS} EMDC KTH November 6, 2012 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Transparent Fault Tolerance Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Task graphs used up to now: Static Acyclic . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Limitations : Task graphs used up to now: Limited Expressive Power Static Poor Performance Acyclic Insufficient Fault Tolerance . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Overview Figure: Distributed Execution Engines comparison . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM HOW ? using Dynamic Task Graphs . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL primitives objects references tasks Figure: A Task Grapha a Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 8/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Dynamic Task Graphs Figure: A Dynamic Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 9/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master Figure: CIEL Worker . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Architecture Figure: CIEL Architecture . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 11/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 12/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task Figure: Dereferencing futures . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 14/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Optimizations Globally unique identifiers enable memoization Streaming partially written objects between tasks . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 15/23
Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Fault Tolerance Client (no driver program) Worker (periodic heartbeat) Master (persistent logging, secondary masters, object table reconstruction) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 16/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 17/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Performance Comparison with production system Figure: DistrubutedGrep on Hadoop and Ciel . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 18/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Perfomance of Iterative Algorithm Figure: K-means on Hadoop and Ciel with 20 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 19/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Overheads Figure: Speedup of Binomial Options Pricing Model on 47 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 20/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Future Work Integrate CIEL with existing programming languages Partition master state Explore use of multiple cores (see [5]) Explore use of non-deterministic parallelism (see [3]) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 21/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: sharing large amounts of data fine-grain parallelization fully automatic parallelism relation algebra environment distributed operating system . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines Questions ? . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . [1] D.G. Murray. A distributed execution engine supporting data-dependent control flow. PhD thesis, PhD thesis, Univ. of Cambridge, 2011. [2] D.G. Murray and S. Hand. Scripting the cloud with skywriting. In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, pages 12–12. USENIX Association, 2010. [3] D.G. Murray and S. Hand. Non-deterministic parallelism considered useful. In HotOS XIII, 13th Workshop on Hot Topics in Operating Systems, 2011. [4] D.G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand. Ciel: a universal execution engine for distributed data-flow computing. In Proceedings of the 8th USENIX conference on Networked systems design and implementation, page 9. USENIX Association, 2011. [5] M. Schwarzkopf, D.G. Murray, and S. Hand. Condensing the cloud: running ciel on many-core. . Proceedings of EuroSys SFMA, 2011. . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 23/23
CIEL Skywriting Experiments Part I . . Appendix . .. . . . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 24/23
CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/7
CIEL Skywriting Experiments . Hidden slide 1 Figure: Task and Object table maintained in Master node . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/7
CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/7
CIEL Skywriting Experiments . Hidden slide 2 Figure: Spawning Tasks1 1 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/7
CIEL Skywriting Experiments . Hidden slide 3 Figure: Blocking on futures2 2 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/7
CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/7
CIEL Skywriting Experiments . Hidden slide 4 Figure: Primary Master Failure . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/7

Empfohlen

Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...IAEME Publication
 
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...INFOGAIN PUBLICATION
 
Virtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudVirtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudiaemedu
 
Mixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressMixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressIBM Decision Optimization
 
An overview and survey of simulation models
An overview and survey of simulation modelsAn overview and survey of simulation models
An overview and survey of simulation modelsAnjandeep Rai
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalabilitymabuhr
 
HCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionHCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionGennady Baranov
 
How to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaHow to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaNikita Koksharov
 

Empfohlen

Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...Performance boosting of discrete cosine transform using parallel programming ...
Performance boosting of discrete cosine transform using parallel programming ...IAEME Publication
 
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...
4 ijaems dec-2015-10-realization of astrosat model with fused deposition mode...INFOGAIN PUBLICATION
 
Virtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudVirtual machine placement in a virtualized cloud
Virtual machine placement in a virtualized cloudiaemedu
 
Mixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressMixed Integer Programming: Analyzing 12 Years of Progress
Mixed Integer Programming: Analyzing 12 Years of ProgressIBM Decision Optimization
 
An overview and survey of simulation models
An overview and survey of simulation modelsAn overview and survey of simulation models
An overview and survey of simulation modelsAnjandeep Rai
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalabilitymabuhr
 
HCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionHCE project Distributed Remote Command Execution
HCE project Distributed Remote Command ExecutionGennady Baranov
 
How to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaHow to reduce hosting costs for Redis based applications on Java
How to reduce hosting costs for Redis based applications on JavaNikita Koksharov
 
Constraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingConstraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingEray Cakici
 
Edge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsEdge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsDebmalya Biswas
 
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Mickael Istria
 
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...Edge AI and Vision Alliance
 
Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 GovCloud Network
 
Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab
 
Creating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersCreating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersIRJET Journal
 
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyThyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyAltair ProductDesign
 
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Alessandra Bagnato
 
Photo Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesPhoto Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesIRJET Journal
 
Unit i computer aided design
Unit i computer aided designUnit i computer aided design
Unit i computer aided designmanimaran m
 
FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2GovCloud Network
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfAPSanyal1
 
Model Execution and Meta Layout
Model Execution and Meta LayoutModel Execution and Meta Layout
Model Execution and Meta LayoutChristian-Albrechts-Universität zu Kiel
 
Transforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingTransforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingAMD
 
Cray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge SimulationsCray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge Simulationsinside-BigData.com
 
Agile archiecture iltam 2014
Agile archiecture iltam 2014Agile archiecture iltam 2014
Agile archiecture iltam 2014Dani Mannes
 
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Benoit Combemale
 
Ch1
Ch1Ch1
Ch1Rahul prasad
 
Applitools - Cross Browser
Applitools - Cross BrowserApplitools - Cross Browser
Applitools - Cross BrowserApplitools
 
Software Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSoftware Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSave Manos
 
Lock Service with Paxos in Erlang
Lock Service with Paxos in ErlangLock Service with Paxos in Erlang
Lock Service with Paxos in ErlangSave Manos
 

Weitere ähnliche Inhalte

Ähnlich wie Ciel universal distributed execution engine

Constraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingConstraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingEray Cakici
 
Edge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsEdge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsDebmalya Biswas
 
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Mickael Istria
 
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...Edge AI and Vision Alliance
 
Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 GovCloud Network
 
Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab
 
Creating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersCreating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersIRJET Journal
 
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyThyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyAltair ProductDesign
 
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Alessandra Bagnato
 
Photo Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesPhoto Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesIRJET Journal
 
Unit i computer aided design
Unit i computer aided designUnit i computer aided design
Unit i computer aided designmanimaran m
 
FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2GovCloud Network
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfAPSanyal1
 
Transforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingTransforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingAMD
 
Cray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge SimulationsCray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge Simulationsinside-BigData.com
 
Agile archiecture iltam 2014
Agile archiecture iltam 2014Agile archiecture iltam 2014
Agile archiecture iltam 2014Dani Mannes
 
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Benoit Combemale
 
Applitools - Cross Browser
Applitools - Cross BrowserApplitools - Cross Browser
Applitools - Cross BrowserApplitools
 

Ähnlich wie Ciel universal distributed execution engine (20)

Constraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in SchedulingConstraint Programming - An Alternative Approach to Heuristics in Scheduling
Constraint Programming - An Alternative Approach to Heuristics in Scheduling
 
Edge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare ApplicationsEdge AI Framework for Healthcare Applications
Edge AI Framework for Healthcare Applications
 
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
Whats new in Eclipse Indigo ? (@DemoCamp Grenoble 2011)
 
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
“Explainability in Computer Vision: A Machine Learning Engineer’s Overview,” ...
 
Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020 Why Should the Financial Community Care About Cloud - Cobis2020
Why Should the Financial Community Care About Cloud - Cobis2020
 
Scilab Modelica conference 20150921
Scilab Modelica conference 20150921Scilab Modelica conference 20150921
Scilab Modelica conference 20150921
 
Creating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and AutoencodersCreating Objects for Metaverse using GANs and Autoencoders
Creating Objects for Metaverse using GANs and Autoencoders
 
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis TechnologyThyssenkrupp Tallent - Evolutionary Design in Chassis Technology
Thyssenkrupp Tallent - Evolutionary Design in Chassis Technology
 
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
Flexible and Scalable Modelling in the MONDO Project: 3 Industrial Case Studi...
 
Photo Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted FeaturesPhoto Editing And Sharing Web Application With AI- Assisted Features
Photo Editing And Sharing Web Application With AI- Assisted Features
 
Unit i computer aided design
Unit i computer aided designUnit i computer aided design
Unit i computer aided design
 
FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2FBI Symposium on Cloud Computing and Security v2
FBI Symposium on Cloud Computing and Security v2
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdf
 
Model Execution and Meta Layout
Model Execution and Meta LayoutModel Execution and Meta Layout
Model Execution and Meta Layout
 
Transforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud ComputingTransforming Your Business Through Cloud Computing
Transforming Your Business Through Cloud Computing
 
Cray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge SimulationsCray HPC Environments for Leading Edge Simulations
Cray HPC Environments for Leading Edge Simulations
 
Agile archiecture iltam 2014
Agile archiecture iltam 2014Agile archiecture iltam 2014
Agile archiecture iltam 2014
 
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
Efficient and Advanced Omniscient Debugging for xDSMLs (SLE 2015)
 
Ch1
Ch1Ch1
Ch1
 
Applitools - Cross Browser
Applitools - Cross BrowserApplitools - Cross Browser
Applitools - Cross Browser
 

Mehr von Save Manos

Software Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSoftware Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSave Manos
 
Lock Service with Paxos in Erlang
Lock Service with Paxos in ErlangLock Service with Paxos in Erlang
Lock Service with Paxos in ErlangSave Manos
 
FOSS Licenses: A first attempt
FOSS Licenses: A first attemptFOSS Licenses: A first attempt
FOSS Licenses: A first attemptSave Manos
 
A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...Save Manos
 
Man In The Browser
Man In The BrowserMan In The Browser
Man In The BrowserSave Manos
 
P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform Save Manos
 
A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs . A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs . Save Manos
 
Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services Save Manos
 
Network as a Service
Network as a ServiceNetwork as a Service
Network as a ServiceSave Manos
 
RESTful Web Services
RESTful Web ServicesRESTful Web Services
RESTful Web ServicesSave Manos
 
Distributed systems
Distributed systemsDistributed systems
Distributed systemsSave Manos
 

Mehr von Save Manos (14)

Software Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network TestbedsSoftware Defined Networking for Community Network Testbeds
Software Defined Networking for Community Network Testbeds
 
Lock Service with Paxos in Erlang
Lock Service with Paxos in ErlangLock Service with Paxos in Erlang
Lock Service with Paxos in Erlang
 
NaaS
NaaSNaaS
NaaS
 
FOSS Licenses: A first attempt
FOSS Licenses: A first attemptFOSS Licenses: A first attempt
FOSS Licenses: A first attempt
 
A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...A boring presentation about social mobile communication patterns and opportun...
A boring presentation about social mobile communication patterns and opportun...
 
Man In The Browser
Man In The BrowserMan In The Browser
Man In The Browser
 
Apache Mahout
Apache MahoutApache Mahout
Apache Mahout
 
P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform P2P-Tuple: Towards a Robust Volunteer Computing Platform
P2P-Tuple: Towards a Robust Volunteer Computing Platform
 
A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs . A survey on modifications for unstructured P2P in WMNs .
A survey on modifications for unstructured P2P in WMNs .
 
Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services Intelligent Placement of Datacenter for Internet Services
Intelligent Placement of Datacenter for Internet Services
 
Network as a Service
Network as a ServiceNetwork as a Service
Network as a Service
 
Openflow
OpenflowOpenflow
Openflow
 
RESTful Web Services
RESTful Web ServicesRESTful Web Services
RESTful Web Services
 
Distributed systems
Distributed systemsDistributed systems
Distributed systems
 

Kürzlich hochgeladen

[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 

Kürzlich hochgeladen (20)

[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 

Ciel universal distributed execution engine

  • 1. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . CIEL universal distributed execution engine . .. . . Presenter: Emmanouil Dimogerontakis @{AdvDS} EMDC KTH November 6, 2012 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/23
  • 2. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/23
  • 3. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 4. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 5. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 6. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Purpose Execute a Task Graph providing: Task Scheduling Data Distribution Load Balancing Transparent Fault Tolerance Figure: Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/23
  • 7. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Task graphs used up to now: Static Acyclic . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
  • 8. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Limitations Limitations : Task graphs used up to now: Limited Expressive Power Static Poor Performance Acyclic Insufficient Fault Tolerance . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/23
  • 9. Motivation CIEL Skywriting Distributed Execution Engines Optimizations & Fault Tolerance Evaluation & Future work . Overview Figure: Distributed Execution Engines comparison . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/23
  • 10. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/23
  • 11. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
  • 12. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL WHY Universal? Support same cluster of algorithms as a TM HOW ? using Dynamic Task Graphs . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/23
  • 13. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . CIEL primitives objects references tasks Figure: A Task Grapha a Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 8/23
  • 14. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Dynamic Task Graphs Figure: A Dynamic Task Graph . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 9/23
  • 15. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
  • 16. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Master & Worker Figure: CIEL Master Figure: CIEL Worker . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 10/23
  • 17. Motivation CIEL Dynamic Task Graphs Skywriting Architecture Optimizations & Fault Tolerance Evaluation & Future work . Architecture Figure: CIEL Architecture . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 11/23
  • 18. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 12/23
  • 19. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
  • 20. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Creating Tasks with Skywriting Figure: Spawning a new task Figure: Dereferencing futures . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 13/23
  • 21. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 14/23
  • 22. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Optimizations Globally unique identifiers enable memoization Streaming partially written objects between tasks . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 15/23
  • 23. Motivation CIEL Skywriting Optimizations & Fault Tolerance Evaluation & Future work . Fault Tolerance Client (no driver program) Worker (periodic heartbeat) Master (persistent logging, secondary masters, object table reconstruction) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 16/23
  • 24. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . . . Motivation 1 Distributed Execution Engines . . . CIEL 2 Dynamic Task Graphs Architecture . . . Skywriting 3 . . . Optimizations & Fault Tolerance 4 . . . Evaluation & Future work 5 Evaluation Future Work Conclusions . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 17/23
  • 25. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Performance Comparison with production system Figure: DistrubutedGrep on Hadoop and Ciel . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 18/23
  • 26. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Perfomance of Iterative Algorithm Figure: K-means on Hadoop and Ciel with 20 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 19/23
  • 27. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Overheads Figure: Speedup of Binomial Options Pricing Model on 47 workers . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 20/23
  • 28. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Future Work Integrate CIEL with existing programming languages Partition master state Explore use of multiple cores (see [5]) Explore use of non-deterministic parallelism (see [3]) . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 21/23
  • 29. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: sharing large amounts of data fine-grain parallelization fully automatic parallelism relation algebra environment distributed operating system . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
  • 30. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
  • 31. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . Conclusions CIEL[4, 1] and Skywriting[2] are not good for: are really good for : sharing large amounts of writing iterative algorithms data data-dependent control flow fine-grain parallelization using dynamic task graphs fully automatic parallelism transparent fault tolerance relation algebra environment and automatic distribution distributed operating system scaling across hundreds of machines Questions ? . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 22/23
  • 32. Motivation CIEL Evaluation Skywriting Future Work Optimizations & Fault Tolerance Conclusions Evaluation & Future work . [1] D.G. Murray. A distributed execution engine supporting data-dependent control flow. PhD thesis, PhD thesis, Univ. of Cambridge, 2011. [2] D.G. Murray and S. Hand. Scripting the cloud with skywriting. In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, pages 12–12. USENIX Association, 2010. [3] D.G. Murray and S. Hand. Non-deterministic parallelism considered useful. In HotOS XIII, 13th Workshop on Hot Topics in Operating Systems, 2011. [4] D.G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand. Ciel: a universal execution engine for distributed data-flow computing. In Proceedings of the 8th USENIX conference on Networked systems design and implementation, page 9. USENIX Association, 2011. [5] M. Schwarzkopf, D.G. Murray, and S. Hand. Condensing the cloud: running ciel on many-core. . Proceedings of EuroSys SFMA, 2011. . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 23/23
  • 33. CIEL Skywriting Experiments Part I . . Appendix . .. . . . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 24/23
  • 34. CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 1/7
  • 35. CIEL Skywriting Experiments . Hidden slide 1 Figure: Task and Object table maintained in Master node . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 2/7
  • 36. CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 3/7
  • 37. CIEL Skywriting Experiments . Hidden slide 2 Figure: Spawning Tasks1 1 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 4/7
  • 38. CIEL Skywriting Experiments . Hidden slide 3 Figure: Blocking on futures2 2 Source: http://www.cl.cam.ac.uk/~dgm36/CIEL-NSDI-slides.pdf . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 5/7
  • 39. CIEL Skywriting Experiments . . . CIEL 6 . . . Skywriting 7 . . . Experiments 8 . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 6/7
  • 40. CIEL Skywriting Experiments . Hidden slide 4 Figure: Primary Master Failure . . . . . . Presenter: Emmanouil Dimogerontakis @{AdvDS} CIEL universal distributed execution engine 7/7