Siyapath - A P2P, Gossip based Volunteer Computing Framework
•
1 gefällt mir•425 views
Slide deck presented at APICTA 2012 - Brunei, Darussalam participating in the tertiary category
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Siyapath - A P2P, Gossip based Volunteer Computing Framework
Ähnlich wie Siyapath - A P2P, Gossip based Volunteer Computing Framework (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Siyapath - A P2P, Gossip based Volunteer Computing Framework
- 1. SIYAPATH P2P, Gossip Based Volunteer Computing Framework
- 2. RISING DEMAND FOR HIGH COMPUTATIONAL POWER 2 APICTA 2012 - Brunei Darussalam
- 3. HOW TO ADDRESS? Supercomputing Distributed Computing Cluster Computing Grid/Desktop-Grid Computing Volunteer Computing Computational power volunteered by general public Large resource pool 3 APICTA 2012 - Brunei Darussalam
- 4. INTRODUCING SIYAPATH P2P, GOSSIP BASED VOLUNTEER COMPUTING FRAMEWORK P2P Architecture Absence of server bottleneck Absence of single point of failure Gossip Protocols Scalability Robustness Reliability Simplicity in Configuring 4 APICTA 2012 - Brunei Darussalam
- 5. FEATURES Job Submission Job Scheduling User Job Processor Task Distribution Task Processing Result Validation Result Delivery Task Task Task Fault Tolerance Processor Processor Processor Security 5 APICTA 2012 - Brunei Darussalam
- 6. DEMONSTRATION 6 APICTA 2012 - Brunei Darussalam
- 7. PERFORMANCE TESTING Test 1 2 3 4 5 Users 5 10 15 20 25 Volunteers 30 30 30 30 30 Avg. Response Time (s) 52.16 89.63 136.29 211.02 249.25 Avg. Throughput (Tasks per 126.76 136.45 130.03 109.49 125.96 minute) Average Response Time Average Throughput 300 160 250 140 120 Tasks per minute Time (seconds) 200 100 150 80 100 60 40 50 20 0 0 5 10 15 20 25 5 10 15 20 25 7 Users APICTA 2012 - Brunei Darussalam Users
- 8. BEST PRACTICES Version Control System – Subversion Build Automation – Apache Maven Unit / Integrated Testing – JUnit Logging – Apache Log4J Continuous Integration – Apache Jenkins User Friendly Website – siyapath.org 8 APICTA 2012 - Brunei Darussalam
- 9. TECHNOLOGIES Services Framework - Apache Thrift Communication via a service interface Addresses implementation details of the network P2P nature using thrift client/server Gossip Protocols System Monitoring – SIGAR Java Standard Edition 9 APICTA 2012 - Brunei Darussalam
- 10. ACHIEVEMENTS Best Product Developed Using Gold Award Research Paper Open Source Technology Tertiary Category ERU Symposium NBQSA 2012 NBQSA 2012 10 APICTA 2012 - Brunei Darussalam
- 11. REACHING THE COMMUNITY Siyapath distribution available at Siyapath website http://siyapath.org/download.html Source code available at Google Code http://code.google.com/p/siyapath/ User manuals for the framework User API docs 11 APICTA 2012 - Brunei Darussalam
- 12. COMMERCIALIZATON Support researchers in conducting high performance computing Use laboratories of universities Provide support & consulting Evaluation Deployment Production 12 APICTA 2012 - Brunei Darussalam
- 13. TEAM Members: Amila Manoj Silva Nadeeshani Hewage Mohamed Nufail Malith Dhanushka Supervisors: Dr. Srinath Perera Mrs. Vishaka Nanayakkara 13 APICTA 2012 - Brunei Darussalam
Hinweis der Redaktion
- Siyapath - in my mother tongue means a hundred of petals. A lotus.L&G, we present to you - Siyapath, the P2P VCF based on gossip protocols.This, is a team work of the 4 of us; myself - Nadeeshani, I’ll be introducing you into Siyapath. Next, my colleagues Amila & Malith will take you through the demonstration. Finally, Nufail will add the ending notes.
- Applications demanding high computational power can be lavishly found today. And the demand has been rising over the time. Simulations,Climate models,Gene sequencers, Biologically inspired algorithms such as ant colonies, genetic algorithmsare some of the instances applied both in research & industry.
- So how do we address the problem of HPC - High performance computing?Supercomputing, is one approach; this involves a considerable expense.Distributed computing is the next approach, where a set/group of computers collectively get together to solve a single computational problem.Variations exist - clusters, grids, desktop grids & volunteer computing.Let me highlight volunteer computing, the approach Siyapath is based on.It’s the general public who volunteer/donor computational power.Thus the resource pool is large. It is self financing and self maintaining, eliminating the cost burden from the project owner.
- Siyapath - the unique, innovative solution to address HPC.1. Siyapath possesses a P2P architecture whereas the existing VCFs are client server based. Client server architecture exhibits server bottleneck, when numerous clients request the service of a single server. In case the server crashes, the entire system may become dysfunctional, displaying single point of failure. But in Siyapath’s P2P architecture, a single siyapath node runs a server thread as well as a client thread. Roles played by each node are dynamic. Hence, due to P2P, there is no server bottleneck, no single point of failure.2. Siyapath uses gossip protocol as the group communication protocol. i.e. to communicate between siyapath nodes, discover peers and formulate the network.None of the existing VCFs use gossip for communication. Yet the protocol is proven to be appealing in large scaled decentralized systems. It exhibits well known advantages such as improved scalability, robustness, reliability.3. Siyapath is also very simple to configure, whereas existing VCFs consume lots of time & work including creating projects, databases, and configuring hundreds of parameters. But in Siyapath it’s a matter of downloading the binary distribution and configuring a few parameters.
- Now lets focus on Siyapath’s features.A job in Siyapath can be defined as a problem demanding high computational power. This problem can be divided into subproblems, each known as a task in Siyapath terminology.Now a user has to chunk his job into several tasks, and submit it to the volunteer computing network. The immediate volunteer receiving the job plays the role of ‘job processor. Once again, roles are dynamic.Received jobs get scheduled at the job processor in a queue, FCFS.Next the tasks are distributed to other volunteer nodes in the network. Each individual task is processed, and results are sent back to the job processor.The user, at his discretion can configure how many nodes run the same task. This is redundancy. It is checked whether all nodes running the same task, return the same result, in order to validate; and discard irregularities.In a volunteer computing system, volunteers can join or leave the network without notice. Siyapath exhibits fault tolerance where a Task-Tracker keeps track of whether a task processor dies, and a backup handler keeps track if a job-processor dies, and resumes from that point.Security is also addressed in Siyapath. We use java sandboxing to implement security.
- Now L&G, we’ll take you through the demonstration.
- We have done a load test to test the performance of the Siyapath system.We had a constant number of volunteer nodes and increased the no. of users submitting jobs to the system. So the load on the system was gradually increased while the resources in the system was at a constant.The average response time and the average throughput was measured. The response time shows a gradual increase which is in accordance with the increase in the load. The average throughput is more or less constant and shows no spikes in the graph. Therefore it can be concluded that Siyapath performs well under increasing load and displays scalable properties.
- To ensure good quality, Siyapath has been developed under industry standards and best practices. Being an open source project, Siyapath achieves source code management and version control using subversion. Apache Maven has been used for build automation. Unit and integration testing was done using junit. Apache log4j framework has been used for all logging purposes. We have a continuous integration system hosted at cloudbees which uses apache jenkins. And a user friendly website is maintained at siyapath.org.
- Then we'll look at some of the technologies deployed in Siyapath. Apache Thrift is used as the services framework. All communication between nodes are done using service interface provided by thrift. Thrift handles all the implementation details of the network. We have achieved p2p nature by using both thrift client and thrift server in each node.Gossip protocol is used as the group communication protocol in Siyapath. We have our very own implementation of gossip protocol.We have used SIGAR in our system monitoring module. This is used to monitor system resources of volunteer nodes such as the processor and memory available.Implementation of Siyapath has been done using Java Standard Edition.
- Siyapath won the Gold award in the Tertiary category at national best quality ICT awards 2012. Siyapath was also selected as the best product developed using open source technology at the same award ceremony. A research paper has been accepted for the Engineering Research Unit symposium of university of moratuwa.
- The binary distribution of Siyapath is accessible through the Siyapath website. Anyone could access the source code of Siyapath through the Google code project page of Siyapath. We have provided user manuals and user API docs for users to get familiar with Siyapath.
- Siyapath would be helpful to researchers with high performance computing needs. Incountries such as Sri Lanka universities do not have supercomputers. Thereforeresearchers find it difficult to research on subjects that involve high performancecomputing. But they could use siypath and make use of the large number of computersin their university laboratories, which are idle most of the time, to satisfy theircomputing needs.As the product is freely available, siyapath could provide commercial support to gain economical benefits. We could build an economic model based on evalution of the requirements and feasibility, deployment of the system and providing continued production support which provides latest feature upgrades and fixes.