The presentation at OpenStack design summit (colony session) on 18th of April, 2012. http://www.openstack.org/conference/san-francisco-2012/sessions/

1. Intercloud object storage: colony
- Trial to use swift 2012.4.18
@jxta
in wide area network - Shigetoshi Yokoyama
National Institute of Informatics

2. Cloud projects @ NII
1

3. Goal: academic community cloud
NII Cloud
Academic
Community Cloud
Education Cloud Univ.-X Cloud
Univ. -A Cloud Univ.-B Cloud
Research Cloud
・・・
Intercloud services
Science Information Network
2

4. Colony and dodai in a scenario
VMI: Virtual Machine Image
VM: Virtual Machines VMI colony VMI
Launch machine images
Virtual
Machines
VM VM VM VM VM VM VM VM
IaaS IaaS IaaS IaaS
Cluster Cluster Cluster Cluster
dodai
Physical
Machines
SINET (L2VPN)
3

5. Colony
4

6. Intercloud object storage service
Colony is a project which
Nova
federates cloud object
storage services, like swift
Glance
Swift for
intercloud
Swift
to realize intercloud object
storage service.
use
Swift for
intercloud
use
Swift for
Nova
Swift for Swift local use
intercloud
Glance use
Glance
Swift
Nova
Swift for
intercloud use
5

7. Users’ points of view
Cloud Services
Cloud-B
Object B1-1
Cloud-A Container B1
Container B2
Container B3
Object B1-2
Object B1-3
Object A1-1
Swift-B Object I1-1
Container A1 Object A1-2 Inter-cloud Container I1 Object I1-2
Container A2 Object A1-3 Inter-cloud Container I8 Object I1-3
Swift-A Container A3 Inter-cloud Container I10
Inter-cloud Container I1 Object I4-1
Inter-cloud Container I4 Object I4-2
Inter-cloud Container I13 Object I4-3
Geographically Inter-cloud Container
Object I1-1
Object I1-2
Distributed
I1
Inter-cloud Container I2 Object I1-3
Swift-I Inter-cloud Container I3
Inter-cloud Container I4
Object I4-1
Object I4-2
Object I4-3
Inter-cloud object storage service : colony
6

8. Colony realizes the federation
Shibboleth IdP
Authenticate with
Shibboleth IdP
Cloud-A User
Colony
Apache
mod_wsgi mod_shib
Colony-horizon
Colony-keystone
Colony-dispatcher
Squid
Provide seamless access to
Slapd
Ubuntu
multiple swifts
Swift Swift
Swift-I Colony-Keystone Colony-Keystone Swift-A
Slapd Slapd
7

9. Developed software components in colony
• Colony-Horizon – based on diablo/stable Horizon with some enhancements
• Multi-region support – Users can choose which swift is used to store/retrieve objects
• Swift Container’s ACL ,metadata support
• Swift Object’s metadata support
• >5G segment upload support …
• Colony-Keystone – based on diablo/stable Keystone with some enhancements
• Authenticate with Shibboleth
• %{tanant_name} can be used for endpointTemplates in addition to %{tenant_id} for
federating cloud services
• Colony-Dispatcher - new
• Relay requests to multiple object services (and merge response for clients)
• Relay requests to a specific object service indicated by URI
• Choose the “nearest” swift-proxy server to relay requests
• Copy objects among different swifts
• Utilities - new
• Tools to make admin task easier to provide the federation of object storage services
8

10. Colony-horizon
Users can choose swift Swift -I
Swift -A
9

11. Colony – keystone
Shibboleth IdP
Modifications to keystone
• Add ePPN field to keystone schema
• ADD rest api services to create token by ePPN
('/token_by/eppn') and email address('/token_by/email')
('/ /email')
• Add a rest api service to register/update ePPN
('/users/{user_id}/eppn')
('/users/{
1. ID/passwd 2. Attribute: ePPN, mail_addr
0-1. User registration by mail_addr
0-2 . Associate ePPN to mail_addr
by initial access
Shibboleth SP
Colony-
Colony-Horizon
3. Attribute: ePPN
Colony-
4. auth_token
Keystone
10

12. Colony-dispatcher
1. Swift client can send requests to Swift-A and Swift-I through Swift Dispatcher
2. Swift Dispatcher merges the response from each Swift and send it to Swift Client
Swift Client
Requests modified for merging responses.
A:container1 •Account Info
A:container2 •Container List
•X-Copy-from/to
I:container1
I:container2
Colony Dispatcher
Response merged by Swift Proxy Swift Proxy Swift Proxy
Colony Dispatcher has
a prefix to indicate
which Swift is used to
store.
Swift-A (local) Swift-I (intercloud )
11

13. Caching
Swift Dispatcher can use cache proxy (like squid) per
swift proxy to retrieve objects from remote swifts.
A:container1
A:container2
I:container1 Colony Dispatcher
I:container2
Cache(Proxy)
Swift Client
Swift Proxy Swift Proxy Swift Proxy
Swift-A (local) Swift-I (intercloud )
12

14. Download performance with cache
Download performance (5times, average)
300,000,000
250,000,000
200,000,000 Cache is
bytes/sec
150,000,000 effective,
100,000,000
especially for
small size
50,000,000
0
objects.
1K(bytes) 1M(bytes) 10M(bytes) 100M(bytes) 1G(bytes) 5G(bytes)
Normal 13,201 11843208 77113446 152263382 163115324 154252275
Cache 382,004 135938185 226800796 256492362 261189612 260956927
・AMD Opteron(tm) Processor 6128 2000Mhz 16core Ubuntu Server 11.04 x86_64 Client
・32GB Memory Colony-Dispatcher
・10000baseT 10000Mb/s Full Duplex Squid
Swift-Proxy
・AMD Opteron(tm) Processor 6128 2000Mhz 16core Ubuntu Server 11.04 x86_64 Swift-Storage(account, container, object)x 3
・32GB Memory
・10000baseT 10000Mb/s Full Duplex
13

15. Performance : original swift
Swift cannot perform well when
confgirure it in inter-region.
Swift
@Sapporo
•Upload – Always effected by worst case
1 2 3 4 5
1K 1,755 2,165 395 3,178 1.375
1M 431,840 446,824 537,768 412,542 43,307
10M 710,179 741,350 737,565 723,446 731,824
100M 780,548 775,131 788,620 782,503 786,224
Swift •Download – worse case is ramdomly choosed
@Tokyo Swift@ (about 1/3)
Chiba
Client 1 2 3 4 5
6.15Mbits/sec
1K 11,608 13,293 12,916 14,691 12,126
1G 6.29Mbits/sec
7.80Mbits/sec 1M 3,427,827 3,427,402 3,455,788 673,614 3,386,009
7.73Mbits/sec netperf 10M 958,018 959,098 7,188,901 974,072 949,389
941.23Mbits/sec
10G 100M 1,109,440 98,400,555 99,641,184 15,411,221 92,400,774
929.15Mbits/sec
14

16. Performance : with cache
Swift Cache works
@Sapporo
•Upload – Always effected by worst case
1 2 3 4 5
1K 2,388 2,027 1,629 2,718 2,423
1M 521,456 479,606 488,142 455,359 541,622
10M 727,240 678,511 724,562 730,127 699,518
100M 786,893 777,227 787,707 782,546 784,544
Swift
@Tokyo Swift@
Chiba •Download (through Cache)
1 2 3 4 5
Client
6.15Mbits/sec 1K 398,133 415,753 397,979 446,576 406,510
1G 6.29Mbits/sec
1M 9,238,473 202,193,598 177,694,628 161,992,275 134,450,057
7.80Mbits/sec
7.73Mbits/sec netperf 10M 55,186,177 288,776,403 319,892,614 331,691,392 307,500,293
941.23Mbits/sec 100M 473,135,008 471,484,134 467,654,981 412,153,419 451,464,516
10G 929.15Mbits/sec
15

17. Performance : with zone-awareness
Swift should be zone-aware
for geographically distributed use.
Swift Modifications for zone-awareness (just a hack)
@Sapporo
•Uploading
• Calculate the number of unfinished tasks in
the send queue for each area and when one
area are much worth than others, quit the
uploading job for its area.
•Downloading
• Check connection performance of object
servers and try to retrieve an object from the
fastest one.
Swift •Upload
@Tokyo Swift@
Chiba 1 2 3 4 5
1K 11,356 13,157 13,074 12,758 12,680
Client
6.15Mbits/sec
1M 9,824,750 11,205,249 7,599,312 10,931,206 11,199,982
1G 6.29Mbits/sec
7.80Mbits/sec 10M 52,294,403 51,437,092 51,050,686 52,641,471 52,300,141
7.73Mbits/sec netperf 100M 97,937,987 101,847,002 102,385,002 102,413,801 101,462,855
941.23Mbits/sec
10G 929.15Mbits/sec
16

18. Project status of colony
• Prototyping is done in NII research cloud
https://github.com/nii-cloud/colony/
• Future plan
- Performance enhancements
- Real deployment and evaluation
17

19. Dodai
18

20. CaaS : Cluster as a Service
Clusters for each private cloud
Elastic
Private cloud - A Private cloud - B Private cloud - C
Private
Cloud Release servers
to server resource pool
Allocate servers
from server resource pool
Server resource pool
19

21. Dodai = dodai-deploy + dodai-compute
Dynamically and securely separated clusters
PaaS Hadoop MPI Hadoop MPI Hadoop MPI
…
SunGrid
Hadoop
Engine
IaaS OpenStack-2
OpenStack-1 Eucalyptus
Upper layer: Deploying software [dodai-deploy]
CaaS
Lower layer: Preparation of nodes [dodai-compute]
20

22. Dodai-deploy
• It is a framework for deployment tool
– New software can be added by anyone
– Now supports
• Openstack Essex
• Openstack Diablo
• Hadoop
• SGE
• It is fast
– Do installation parallel on different machines.
• It is easy to use
– Not just Web UI
– CLI for maintainer
– Rest API for integration
21

23. How to use dodai-deploy
• Install
– Just one command to install all-in-one
• curl https://raw.github.com/nii-cloud/dodai-deploy/master/setup-env/setup-
all-in-one.sh | sudo sh
• Use
– Web UI
• Learn step-by-step guidance at
http://$dodai_deploy_server:3000/
– REST APIs
• Try at http://$dodai_deploy_server:3000/rest_apis/index.html
– CLI
• script/cli.rb list SERVER
22

24. What’s inside in dodai-deploy
MCollective
dodai-deploy server puppet
Ruby on Rails
dodai-deploy nodes
23

25. dodai-compute
• In a word
–Bare metal version nova
• Futures
–Provide the same interface as nova to
operate on physical machines
–Separate network segments with Open
Flow Controller
24

26. What’s inside in dodai-compute
OpenFlow
modified nova controller
cobbler
OpenFlow
switch
25

27. Project status of dodai
1. Dodai-deploy
- One of the OpenStack compute automated installations tools
http://docs.openstack.org/trunk/openstack-compute/admin/content/openstack-compute-deployment-tool-with-puppet.html
https://github.com/nii-cloud/dodai-deploy/
- Used in NII research cloud development
2. Dodai-compute
https://github.com/nii-cloud/dodai-compute/
- Used in NII research cloud development
3. Future plan
- Enahance based on Feedbacks from NII researchers
- Experiments on intercloud compute service
26

28. NII research cloud
gunnii
27

29. NII academic cloud : gunnii
“Elastic Private Cloud”
on on on
demand demand demand
Cluster as a Service
Based on
Deployment Tool
dodai-deploy
generalized
Compute dodai-compute
bare metal
supported
28

30. gunnii prototype system
NII shibboleth IdP
Researchers
L3 switch
Object storage (Swift)
GUI server
OpenFlow
Switch/controller Storage nodes X 8 (initial)
NFS
Deployed Hadoop, OpenStack, Sun Grid Engine
Compute nodes×40 (initial)
29

31. Demo ?
gunnii is shibboleth ready.
30

32. https://github.com/nii-cloud/
Thank you
31