LarKC Tutorial at ISWC 2009 - Parallelisation

•Als PPT, PDF herunterladen•

0 gefällt mir•346 views

The document discusses parallelization and distribution strategies in the LarKC semantic web platform. It describes how LarKC supports coarse-grained parallelization across plug-ins and fine-grained parallelization within plug-ins. LarKC also supports distributed execution of plug-ins on remote systems. This allows it to make use of multiple computing resources and handle large amounts of data. The document provides an example of how parallelization could be applied across different plug-ins in a pipeline.

Technologie Bildung

Parallelization and Distribution - Motivation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Scalability

“ within a plug-in ” parallelization MPI OpenMP hybrid … “ across plug-ins ” or “ across instances of the same plug-in ” parallelization IBIS/JavaGAT … Grid middleware solutions Parallelization and Distribution strategies in LarKC Scalability at plug-in level Scalability at pipeline level Plug-in scope Platform scope

Parallelization and Distribution in the LarKC Platform – Local execution Current Prototype ,[object Object],[object Object],[object Object],[object Object],Local Plug-in Manager Query Transformer Plug-in API Local Plug-in Manager Identifier Plug-in API Local Plug-in Manager Info. Set Transformer Plug-in API Local Plug-in Manager Selecter Plug-in API Local Plug-in Manager Reasoner Plug-in API Decider Plug-in Registry Pipeline Support System

Parallelization and Distribution in the LarKC Platform – Remote execution Implementation in progress Remote Plug-in Manager Query Transformer Plug-in API Remote Plug-in Manager Identifier Plug-in API Remote Plug-in Manager Info. Set Transformer Plug-in API Remote Plug-in Manager Selecter Plug-in API Remote Plug-in Manager Reasoner Plug-in API Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Decider Plug-in Registry Pipeline Support System + Support for distributed remote execution

Application of Parallelization and Distribution - Example Parallelization across plug-ins Identifier Selecter 1 Reasoner Decider Selecter 2 Query Transformer Reasoner Parallelization within plug-in Distribution ,[object Object],[object Object],[object Object],[object Object]

High Performance and Distributed Computing support in LarKC (1/2) LarKC supports large-scale HPC and distributed computing environments for executing plug-ins/pipelines Plug-in layer Platform layer Decider Identifier LarKC platform Reasoner LarKC Data Layer Resource layer … Developer extensions LarKC middleware adapters/extensions User environment High-performance and Grid (Cloud) environment Data Storage RDF Store RDF Doc RDF Doc RDF Store High-performance and cluster systems Public Desktop Grid Volunteer resources User desktop machine Cloud resources Native middleware solutions

High-performance computing systems (clusters of SMP nodes) Computing environments potentially supported by LarKC Public Desktop Grid (BOINC based) Public Desktop Grid (XtremWeb based) Volunteer resources Public Desktop Grid Local Desktop Grid High Performance and Distributed Computing support in LarKC (2/2) High Performance Computing Grid infrastructure (e.g. EGEE, DEISA, etc.) Cloud computing environments Service Grid (e.g. EDGeS) Implementation in progress

Weitere ähnliche Inhalte

Was ist angesagt?

The History and Use of RAnalyticsWeek

Hadoop Map ReduceVNIT-ACM Student Chapter

Apache flinkpranay kumar

Adam_Mcconnell_SPR11_v3Adam McConnell

Data analysis with R and JuliaMark Tabladillo

Big Data LDN 2018: PROJECT HYDROGEN: UNIFYING AI WITH APACHE SPARKMatt Stubbs

Map ReduceManuel Correa

Logical TimeYoshi Shih-Chieh Huang

Indic threads pune12-apache-crunchIndicThreads

Personal Research Overview presented at the KU-NAIST Research MeetingChawanat Nakasan

Unit 2 part-2vishal choudhary

Data provenance in HopsworksAlexandru Adrian Ormenisan

OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...Oscar Corcho

Resumemuddanas

flat_presentation_time_evolving_OD_matrix_estimationLuís Moreira-Matias

Introduction into scalable graph analysis with Apache Giraph and Spark GraphXrhatr

Map Reduceschapht

Was ist angesagt? (17)

The History and Use of R

Hadoop Map Reduce

Apache flink

Adam_Mcconnell_SPR11_v3

Data analysis with R and Julia

Big Data LDN 2018: PROJECT HYDROGEN: UNIFYING AI WITH APACHE SPARK

Map Reduce

Logical Time

Indic threads pune12-apache-crunch

Personal Research Overview presented at the KU-NAIST Research Meeting

Unit 2 part-2

Data provenance in Hopsworks

OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...

Resume

flat_presentation_time_evolving_OD_matrix_estimation

Introduction into scalable graph analysis with Apache Giraph and Spark GraphX

Map Reduce

Andere mochten auch

LarKC Tutorial at ISWC 2009 - IntroductionLarKC

LarKC Tutorial at ISWC 2009 - ArchitectureLarKC

Integrando C com Pythongsroma

CLIPS xiha tareen

CLIPS Basic Student GuideUniv of Umm Al Qura , Makkah

Inference engineAbhishek Pachisia

Introduction to CLIPS Expert SystemMotaz Saad

Andere mochten auch (7)

LarKC Tutorial at ISWC 2009 - Introduction

LarKC Tutorial at ISWC 2009 - Architecture

Integrando C com Python

CLIPS

CLIPS Basic Student Guide

Inference engine

Introduction to CLIPS Expert System

Ähnlich wie LarKC Tutorial at ISWC 2009 - Parallelisation

Pacemaker+DRBDDan Frincu

Petapath HP Cast 12 - Programming for High Performance Accelerated Systemsdairsie

Context-aware Fast Food Recommendation with Ray on Apache Spark at Burger KingDatabricks

Software defined network and Virtualizationidrajeev

Big Data Streams Architectures. Why? What? How?Anton Nazaruk

CS8091_BDA_Unit_IV_Stream_ComputingPalani Kumar

Data Engineering for Data Scientists jlacefie

Systems Support for Many Task ComputingEric Van Hensbergen

TransPAC3/ACE Measurement & PerfSONAR UpdateInternational Networking at Indiana University

Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Guido Schmutz

Accelerating Apache Hadoop through High-Performance Networking and I/O Techno...DataWorks Summit/Hadoop Summit

Opentracing 101HungWei Chiu

Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks

10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...Mullaiselvan Mohan

Apache Spark 101 - Demi Ben-AriDemi Ben-Ari

Sector Sphere 2009lilyco

sector-spherexlight

Apache spark - History and market overviewMartin Zapletal

LAS16-405:OpenDataPlane: Software Defined Dataplane leaderLinaro

Big Data Meets HPC - Exploiting HPC Technologies for Accelerating Big Data Pr...inside-BigData.com

Ähnlich wie LarKC Tutorial at ISWC 2009 - Parallelisation (20)

Pacemaker+DRBD

Petapath HP Cast 12 - Programming for High Performance Accelerated Systems

Context-aware Fast Food Recommendation with Ray on Apache Spark at Burger King

Software defined network and Virtualization

Big Data Streams Architectures. Why? What? How?

CS8091_BDA_Unit_IV_Stream_Computing

Data Engineering for Data Scientists

Systems Support for Many Task Computing

TransPAC3/ACE Measurement & PerfSONAR Update

Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...

Accelerating Apache Hadoop through High-Performance Networking and I/O Techno...

Opentracing 101

Running Emerging AI Applications on Big Data Platforms with Ray On Apache Spark

10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...

Apache Spark 101 - Demi Ben-Ari

Sector Sphere 2009

sector-sphere

Apache spark - History and market overview

LAS16-405:OpenDataPlane: Software Defined Dataplane leader

Big Data Meets HPC - Exploiting HPC Technologies for Accelerating Big Data Pr...

Kürzlich hochgeladen

Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2

Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

Histor y of HAM Radio presentation slidevu2urc

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

Artificial Intelligence: Facts and MythsJoaquim Jorge

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous

Apidays New York 2024 - The value of a flexible API Management solution for O...apidays

Real Time Object Detection Using Open CVKhem

Kürzlich hochgeladen (20)

Advantages of Hiring UIUX Design Service Providers for Your Business

Scaling API-first – The story of a global engineering organization

Exploring the Future Potential of AI-Enabled Smartphone Processors

Powerful Google developer tools for immediate impact! (2023-24 C)

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Automating Google Workspace (GWS) & more with Apps Script

Histor y of HAM Radio presentation slide

Data Cloud, More than a CDP by Matt Robison

Handwritten Text Recognition for manuscripts and early printed texts

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Driving Behavioral Change for Information Management through Data-Driven Gree...

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Artificial Intelligence: Facts and Myths

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

presentation ICT roal in 21st century education

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Apidays New York 2024 - The value of a flexible API Management solution for O...

Real Time Object Detection Using Open CV

LarKC Tutorial at ISWC 2009 - Parallelisation

1. Parallelisation in LarKC

3. “ within a plug-in ” parallelization MPI OpenMP hybrid … “ across plug-ins ” or “ across instances of the same plug-in ” parallelization IBIS/JavaGAT … Grid middleware solutions Parallelization and Distribution strategies in LarKC Scalability at plug-in level Scalability at pipeline level Plug-in scope Platform scope

5. Parallelization and Distribution in the LarKC Platform – Remote execution Implementation in progress Remote Plug-in Manager Query Transformer Plug-in API Remote Plug-in Manager Identifier Plug-in API Remote Plug-in Manager Info. Set Transformer Plug-in API Remote Plug-in Manager Selecter Plug-in API Remote Plug-in Manager Reasoner Plug-in API Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Decider Plug-in Registry Pipeline Support System + Support for distributed remote execution

7. High Performance and Distributed Computing support in LarKC (1/2) LarKC supports large-scale HPC and distributed computing environments for executing plug-ins/pipelines Plug-in layer Platform layer Decider Identifier LarKC platform Reasoner LarKC Data Layer Resource layer … Developer extensions LarKC middleware adapters/extensions User environment High-performance and Grid (Cloud) environment Data Storage RDF Store RDF Doc RDF Doc RDF Store High-performance and cluster systems Public Desktop Grid Volunteer resources User desktop machine Cloud resources Native middleware solutions

8. High-performance computing systems (clusters of SMP nodes) Computing environments potentially supported by LarKC Public Desktop Grid (BOINC based) Public Desktop Grid (XtremWeb based) Volunteer resources Public Desktop Grid Local Desktop Grid High Performance and Distributed Computing support in LarKC (2/2) High Performance Computing Grid infrastructure (e.g. EGEE, DEISA, etc.) Cloud computing environments Service Grid (e.g. EDGeS) Implementation in progress

9. footer 04/11/09 end

LarKC Tutorial at ISWC 2009 - Parallelisation

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (17)

Andere mochten auch

Andere mochten auch (7)

Ähnlich wie LarKC Tutorial at ISWC 2009 - Parallelisation

Ähnlich wie LarKC Tutorial at ISWC 2009 - Parallelisation (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

LarKC Tutorial at ISWC 2009 - Parallelisation