The document proposes a new paradigm called the Social Cloud for collaborative scientific computation. It leverages existing social networks by allowing scientists to contribute resources like compute, storage, and services to groups, forming virtual research environments. This provides a flexible and extensible platform for global collaboration where access to shared resources is based on the implicit trust within social networks.
Computer 10: Lesson 10 - Online Crimes and Hazards
Collaborative eResearch in a Social Cloud
1. The Social Cloud for
Collaborative Scientific Computation
Ashfag Thaufeeg, Kris Bubendorfer and Kyle Chard
eScience 2011
2. Background
Increasingly research requires access to computation and storage of
scientific data.
This is often beyond the resources that individual scientists or
individual organisations can provide.
Access to national fixed infrastructure is often limited to selected
larger projects and well established researchers.
Research conducted on commercial clouds is potentially expensive.
Large usability barriers for non expert users, i.e. command line etc.
Coordination within existing projects and establishing new
collaborations is difficult, tools and systems vary across members
and organisations.
22
eScience 2011
3. Researchers & Social Networks
Multi-institute collaboration is desirable, but
communication and coordination is hard in practice, and
face to face meetings tend to occur only sporadically, i.e., at
conferences and workshops.
Social networks can potentially provide a natural basis
for collaboration, they:
decrease the effort needed to initiate a new collaborative
effort, by using tools and infrastructure already familiar to
and understood by most people,
Can facilitate discovery of other scientists working on
similar projects, through relationships, feeds and
invitations.
33
eScience 2011
4. Social Networks
Are based around pre-existing relationships,
i.e. your friends and colleagues, friends of friends…
Have a pre-existent fabric of trust inherently
interwoven into the network
How many of your friends do you not trust?
Friends can be grouped based on interest and level of
trust.
Many applications now use social networks as a
platform for:
Authentication e.g. Facebook Connect
Application Portals e.g. ASPEN and PolarGrid
projects
Established application APIs
4
5. Researchers & Social Networks
But, the social network itself is not a complete
collaboration solution.
collaboratorsalso often have resources they wish to
share dynamically for the duration of a project, and
resources may include documents, media, data,
services, compute, storage, and so on.
Currently this is a difficult process that
Requires access to unfamiliar tools and systems
outside the social network,
manual (reciprocal) user account creation etc.
sharing involves access, rights, accounting and
auditing – that increases the overhead of project
management.
55
eScience 2011
6. The Social Cloud
The Social Cloud is a new way of thinking about providing cloud
like resources for eResearch.
A Social Cloud is a resource and service sharing framework utilizing
relationships established between members of a social network.
Social networks (such as Facebook, Linkedin etc):
provide considerable infrastructure, including authorisation
and APIs for external apps.
contain many tools for managing relationships, forming groups
and defining security policies.
Include basic incentive mechanisms to moderate behaviour.
This is ‘free’ user infrastructure we can take advantage of.
66
eScience 2011
7. The Social Cloud
What is different about the social cloud?
The social network comes first:
It is not a cloud or collaboration environment extended with a social
network, it is a social network extended with cloud functionality.
The people and their networks form the basic infrastructure,
the resources they access and shared are formed around their
unique social graph.
In some ways this turns the provision of research infrastructure
on its head. Rather than fitting the researcher to the
infrastructure, we fit the infrastructure to the researcher.
Users fulfill all roles, from provision of resources, management
of collaborations through to use of services and computation -
all of which are performed with familiar Social Networking
tools.
The researchers choose how to delegate resources between the
77
eScience 2011
groups within which they collaborate.
8. Groups in a Social Cloud
We exploit the analogy between social networking
groups and dynamic virtual organizations.
Groups like virtual organizations have an intent,
membership and policies that define sharing (in
social networks this relates to photos, media, etc).
We extend the concept of sharing to include
resources such as, compute, storage and other
services.
This forms a Virtual Research Environment (VRE)
within the SoCC.
We explicitly bind a VRE/VO to a Facebook group
within the SoCC.
88
eScience 2011
9. Social Cloud and Groups
Co-workers
Family B
Friends
A
C
99
eScience 2011
10. Advantages
We believe that a cloud architecture integrated with a
social network can benefit the scientific community in the
following ways:
Lowers usability barriers, a minimal move away from a
non CS expert’s comfort zone.
It allows researchers to share resources for the duration
of collaboration.
Such collaborations are light-weight and dynamic,
resources can be delegated, removed and accessed using
the SN group structure.
Authorization and access control take place
transparently for the owner and users.
no visible certificate management,
single sign on and
seamless integration with SN functionality.
10
10
eScience 2011
11. Advantages
A Social Cloud could also be considered a:
resource fabric overlay over a Social Network, or,
generalized and extended form of crowd sourcing.
It is not intended to replace specialist HPC
infrastructure for big science.
The social cloud is intended to be smaller, more
general, adhoc and dynamic.
Created and destroyed on need, ideal for early
work, initial studies, smaller projects etc.
11
11
eScience 2011
12. The Social Cloud Project
Collaboration between researchers at:
Victoria, Cardiff, Chicago and KiT
There are a number of implementations exploring
different aspects of the Social Cloud domain:
The Social Storage cloud (the first implementation),
The Social Cloud for public eScience, and
The Social Cloud for Collaborative Scientific Computation.
In addition projects have been looking at:
Business models for the social cloud,
Incentives, economies, gamification to underpin sharing.
The Social Cloud project began in 2009.
12
12
eScience 2011
13. Social Storage Cloud
SaaS: Simple Storage Service
Implemented as a Facebook application.
first experiments with a socially oriented market.
Agreement
13
14. Posted Price
Enables interactions based upon active trading and or
collaborative decisions
Intuitively facilitates reciprocal resource exchange
Current “norm” in industry solutions
Social Cloud
MDS
User UR Capaci Price
ID L ty
User1 100MB 5
Storag
User2 500MB 10
e
Storag
Storag
User3 5GB 7 e
e
14
15. Dynamic Auctions
Auction:
Enables dynamic participant pairing
Sealed bid second price reverse auction
Could be extended to any other auction mechanism
15
16. SoCC Architecture
A IaaS platform constructed and accessed via a facebook
application.
Schedules VMs across a VRE (group/VO)
Built on top of Nimbus
Handles the VM lifecycle.
Facebook interface to:
Group/VRE management.
Create, join, leave a VRE
Delegate resources to a VRE
Monitor your VMs
Monitor your delegated resources.
Facebook interface, used for groups, tweets, feeds, and
other project personnel coordination.
16
16
eScience 2011
18. SoCC Architecture
Application Server (logical):
The application server hosts a web application built
using the Facebook public API that renders directly
inside the Facebook UI, thereby giving the
impression of seamless integration to users.
The application server is responsible for collecting
registered user data from Facebook through the
graph API and providing an interface to the SoCC.
Within the logical Application Server there is also an
Image Store, Scheduler and Context Broker.
18
18
eResearch 2011
19. SoCC Architecture
Image store: contains the base set of OS images
for the VMs. These images are pre-packaged to
provide quick instantiation for users who do not
want to prepare their own images
Scheduler: schedules submitted VMs to one of
the available clusters** depending on resources
owed and available.
Context Broker: contextualizes the VMs,
including the setting up ssh keys and user
accounts so that members belonging to the same
VO (social network group) can connect to the VM.
OCCI
19
19
eResearch 2011
21. VM startup times
1000
created->running
900
received->created
800
Total Time (seconds)
700
600
500
400
300
200
100
0
ylinux-op on-1 ubuntu-op on-1 ylinux-op on-2 ubuntu-op on-2
21
21
eScience 2011
22. Sharing
Different contexts require different models, we
are exploring a range of allocation tecniques in
the social cloud context:
Storage– credits
Compute – shares or slices
Volunteer – gamification, incentives (talk tomorrow)
22
24. Summary
A new cloud paradigm, the social cloud.
The Social Cloud for Collaborative Computation
Uses existing Social Network infrastructure.
Scientists contribute to groups, both with resources
and postings (social media) forming VREs.
Open flexible, extensible. Global.
Access to resources based on implicit trust.
24
24
eScience 2011
Hinweis der Redaktion
Good to have the general public participate in science and research Boinc until recently had more compute available than the top supercomputer Tianchi-III. Only recently surpassed by the step change at he top of the SC table. The resources available through facebook are massive.
Tools that are designed for NON EXPERT users!!!
Social Networks model relationships – can also model collaborations. Authentication e.g. Facebook Connect, rather than Open ID Automated Service Provisioning ENvironment (ASPEN) [6] exposes applications hosted by Cloud providers to user communities in Facebook. The focus of ASPEN is exposing applications and sharing data within an enterprise through an intuitive and integrated environment, however this could also be applied to a scientific domain does not proviion the infrastructure via FB. OpenSocial & OpenId, used by most social networking sites, and Facebook ’ s bespoke application framework
One role for a social cloud is in the early stages of research when the costs of dedicated research infrastructure would be prohibitive. It is light weight, The value of social networking has been observed in multi- ple scientific domains as a means of facilitating collaboration. Increasingly social networks are being used to coordinate re- search communities, two such examples are MyExperiment [3] for biologists and nanoHub [4] for the nanoscience community. MyExperiment provides a virtual research environment where collaborators can share and execute scientific work- flows. nanoHub allows users to share data and transparently execute applications on distributed resource providers such as TeraGrid. While similar to a Social Cloud, MyExperiment and nanoHub each have specific sharing focuses and build their own proprietary social network stack.
User-specific groups, defined by relationship types, are shown in the context of a Social network. In this example group A is composed of only co-worker members, whereas group B is formed by family members and group C includes only friends. Clearly the level of trust and mechanisms for social correction (identifying incentives and disincentives for users to participate) differ between groups. This figure also highlights that Social Clouds are not mutually exclusive, that is, users may be simultaneously members of multiple Social Clouds. Whereas a VO is often associated with a particular application or activity, and is often disbanded once this activity completes, a group is longer lasting and may be used in the context of multiple applications or activities.
Lower/remove barriers to usability. To a larger, non expert base of users. A minimal move away from comfort zone, make the facilities fit the users rtaher than the users fit the facilities. There are many hurdles struck by putting users pushed into/on raw infrastructure, certificates particularly problematic. Old unfamilar techniques such as command lines etc. intimidating for non expert users. But the point of the social cloud is uniform over resources due to being within Social Network.
The Social Storage cloud (SaaS the first implementation), The Social Cloud for public eScience, and The Social Cloud for Collaborative Scientific Computation (IaaS or PaaS).
FB renders the interface, provides the social connectivity data – computation is done outside FB on our own servers.
Take it or leave it fixed price Used MDS for resource discovery.
Attributes of a Second Price Sealed Bid auction Encourages truth telling Lowers communication overhead We weren ’ t really happy with the economic models underpinning resource trading between friends. But this was a first attempt at implementing a social cloud, and in this it was successful, we built an application in FB that enabled FB users to trade storage with their friends. We didn ’ t implement obvious facilities such as replication as that would have been a trivial extension had we gone beyond the protoctype.
Talk about why facebook. Maturity. Challenges, API lacks stability. Baby steps! Obviously the end goal would be SaaS – members can provide IaaS and SaaS freely within the VRE.
OCCI = Open Cloud Computing Interface, draft standard. Used for IOC and between user resources and application server.
** most of our recent work has been on the scheduler, now 2 layer meta scheduler that schedules from multiple virtual clusters. Open Cloud Computing Interface (OCCI) adopted the OCCI draft specification in the SoCC for communication between user resources and the application server, as well as for communication between user resources.
Also an Android app.
With spot pricing, both for consumption and provision of Compute Shares. Credits here = shares. Experiment naturally resource limited – potentially an artifical situation. Each VO consumes credits. Members earn credits through providing resources that can then be used in turn for computation. Global price. After preparation, the AuverGrid trace had the following characteristics: • Number of users: 401 • Number of VOs: 8 (VO1: 158 users, VO2: 107 users, VO3: 16 users, VO4: 47 users, VO5: 46 users, VO6: 11 users, VO7: 10 users, VO8: 8 users) • Number of computes: 339314
Asym, researchers vs public. Not always clear what public gets out of it, hence incentives. Sym, peers, collaborations, participants on equal footing. Group/VRE ower can kick out misbehaving members.