by James Broberg - Presentation given at the 2nd International Workshop on Web APIs and Mashups (at ICSOC2008) on December 1st, 2008 in Sydney, Australia. http://www.icsoc-mashups.org/

1. MetaCDN: Creating a ‘Cloud
Storage’ Mashup for High
Performance, Low Cost
Content Delivery
Dr. James Broberg (brobergj@csse.unimelb.edu.au)
http://www.csse.unimelb.edu.au/~brobergj
http://www.metacdn.org

2. Content Delivery
Networks (CDNs)
• What is a CDN?
• Content Delivery Networks (CDNs) such as
Akamai, Mirror Image and Limelight place
web server clusters in numerous
geographical locations to improve the
responsiveness and locality of the content it
hosts for end-users.

3. Existing CDN
providers
• Akamai is the clear leader in coverage and
market share (approx. 80%)
• Price is prohibitive for SME, NGO, Gov...
• Anecdotally 2−15 times more expensive
than Cloud Storage, and require 1−2 year
commitments and min. data use (10TB+)
• Academic CDNs include Coral, Codeen,
Globule....
• No SLA / QoS provided, only ‘best effort’

4. Storage Clouds
‘Storage as a Service’
Now!
Late ‘08
??????

5. SaaS Comparison

6. Introducing MetaCDN
• What if we could create a low-cost, high
performance overlay CDN using these
storage clouds?
• Exploit the strengths and coverage
footprints of multiple providers
• Cherry pick providers based on
customers QoS needs / cost budget
• MetaCDN provides this while hiding the
complexity from users

7. How MetaCDN works
AmazonS3Connector NirvanixConnector
createFolder(foldername, location) DefaultConnector createFolder(foldername, location)
deleteFolder(foldername) DEPLOY_USA deleteFolder(foldername)
createFile(file, foldername, DEPLOY_EU createFile(file, foldername,
location, date) DEPLOY_ASIA location, date)
createFile(fileURL, foldername, DEPLOY_AUS createFile(fileURL, foldername,
location, date) createFolder(foldername, location)
location, date)
renameFile(filename, newname, deleteFolder(foldername)
renameFile(filename, newname,
location) throws createFile(file, foldername,
location)
FeatureNotSupportedException location, date)
createTorrent(file) throws
createTorrent(file) createFile(fileURL, foldername,
FeatureNotSupportedException
createTorrent(fileURL) location, date)
createTorrent(fileURL) throws
deleteFile(file, location) renameFile(filename, newname,
FeatureNotSupportedException
listFilesAndFolders() location)
deleteFile(file, location)
deleteFilesAndFolders() createTorrent(file)
listFilesAndFolders()
deleteFilesAndFolders()
createTorrent(fileURL)
<<exception>> deleteFile(file, location)
FeatureNotSupportedException listFilesAndFolders()
FeatureNotSupportedException(msg) deleteFilesAndFolders()

8. How MetaCDN works
Amazon S3 & Mosso Cloud Shared/Private
Nirvanix SDN Coral CDN
CloudFront Files CDN Host
WebDAVConnector
JetS3t toolkit Nirvanix SDK Cloud Files SDK Java stub
Java SDK Java SDK Java SDK MetaCDN.org
Open Source Nirvanix, Inc Mosso, Inc SCPConnector
Java stub
MetaCDN.org
AmazonS3Connector NirvanixConnector CloudFilesConnector CoralConnector FTPConnector
Java stub Java stub Java stub Java stub Java stub
MetaCDN.org MetaCDN.org MetaCDN.org MetaCDN.org MetaCDN.org
MetaCDN
MetaCDN MetaCDN QoS MetaCDN MetaCDN
Manager Monitor Allocator Database
Web Portal Load Redirector Web Service
Java (JSF/EJB) based portal Random redirection SOAP Web Service
Support HTTP POST Geographical redirection RESTful Web Service
New/view/modify deployment Least cost redirection Programmatic access

9. MetaCDN Manager
• The MetaCDN Manager ensures that:
• All current deployments are meeting
QoS targets (where applicable)
• Replicas are removed when no longer
required (minimising cost)
• A users’ budget has not been exceeded,
by tracking usage (i.e. storage/download)

10. MetaCDN QoS
Monitor
• The MetaCDN QoS Monitor:
• Tracks the performance of participating
providers at all times
• Monitors and records throughput and
response time from a variety of locations
• Ensures that upstream providers are
meeting their Service Level Agreements

11. MetaCDN Allocator
• The MetaCDN Allocator allows users to
deploy files either directly or from a public
origin URL, with the following options:
• Maximise coverage and performance
• Deploy content in specific locations
• Cost optimised deployment
• Quality of Service (QoS) optimised

12. MetaCDN Database
MetaCDN 1 0:M CDN 1 1 CDN
has for
User Credentials Provider
1 1 1
hosted
has has
by
M
1 M
1 deployed 0:M MetaCDN Coverage
Content as Replica locations

13. MetaCDN Web Portal
• Developed using Java Enterprise and Java
Server Faces (JSF) technologies
• MySQL back-end to store persistent data
• Web portal acts as the entry point to the
system and application-level load balancer
• Most suited for small or ad-hoc
deployments, and especially useful for less
technically inclined content creators.

18. MetaCDN Web Service
• Makes all functionality available via Web
Services (SOAP & REST/HTTP)
• Web interface is useful for novices and
for ad-hoc deployments, but doesn’t scale
• Larger customers have 1,000’s - 10,000 -
100,000s of files that need deployment
• Let them automate their deployment and
management via Web Services!
• Perfect for Mashup developers!!!

19. MetaCDN Load
Redirector
• We have created a unified namespace to
simply deployment, routing, management
• Currently, file deployment results in
multiple URLs, each mapping to a replica
• http://metacdn-eu-user.s3.amazonaws.com/myfile.mp4
• http://metacdn-us-user.s3.amazonaws.com/myfile.mp4
• http://node3.nirvanix.com/MetaCDN/user/myfile.mp4
• Single namespace is created for fine control
• http://www.metacdn.org/FileMapper?itemid=2

20. MetaCDN Load
Redirector (cont.)
MetaCDN end-user DNS Server MetaCDN gateway
Resolve www.metacdn.org
Return IP of closest MetaCDN gateway,
www-na.metacdn.org
GET http://metacdn.org/MetaCDN/FileMapper?itemid=1
processRequest ()
geoRedirect ()
HTTP 302 Redirect to
http://metacdn-us-username.s3.amazonaws.com/filename.pdf
Resolve metacdn-us-username.s3.amazonaws.com Amazon S3 USA
Return IP of metacdn-us-username.s3.amazonaws.com
GET http://metacdn-us-username.s3.amazonaws.com/filename.pdf
Return replica

21. Redirector Overhead
1.8
from USA
S3 USA
MetaCDN
1.6
1.4
1.2
1
Seconds
0.8
0.6
0.4
0.2
0
0 5 10 15 20 25
Hour

22. Redirector Overhead
from Australia
0.9
Nirvanix SDN #3
MetaCDN
0.85
0.8
0.75
Seconds
0.7
0.65
0.6
0.55
0 5 10 15 20 25
Hour

23. Evaluating MetaCDN
performance
• Ran tests over 24 hour period (mid-week)
• Downloading test replicas (1KB, 10MB) 30
times per hour, take average and conf. inter.
• 10MB - Throughput, 1KB - Response
Time
• Ran test from 6 locations: Melbourne
(AUS), Paris (FRA),Vienna (AUT), San
Diego & New Jersey (USA), Seoul (KOR)
• Replicas located across US, EU, ASIA, AUS

24. Summary of Results -
Throughput (KB/s)
S3 S3 SDN SDN SDN SDN Coral
US EU #1 #2 #3 #4
Melbourne
Australia
264.3 389.1 30 366.8 408.5 405.6 173.7
Paris France 703.1 2116 483.8 2948 416.8 1043 530.2
Vienna
Austria
490.7 1347 288.4 2271 211 538.7 453.4
Seoul
South Korea
312.8 376.1 466.5 411.9 2456 588.2 152
San Diego
USA
1234 323.5 5946 380.1 506.1 820.4 338.5
New Jersey
USA
2381 1949 860.8 967.1 572.8 4230 636.4

25. Summary of results -
Response Time (Sec)
S3 S3 SDN SDN SDN SDN Coral
US EU #1 #2 #3 #4
Melbourne
Australia
1.378 1.458 0.663 0.703 1.195 0.816 5.452
Paris France 0.533 0.2 0.538 0.099 1.078 0.316 3.11
Vienna
Austria
0.723 0.442 0.585 0.099 1.088 0.406 3.171
Seoul
South Korea
1.135 1.21 0.856 0.896 1 0.848 3.318
San Diego
USA
0.232 0.455 0.23 0.361 0.775 0.319 4.655
New Jersey
USA
0.532 0.491 0.621 0.475 1.263 0.516 1.916

26. Summary of results
(cont)
• Clients benefited greatly from local replicas
• Results are consistent in terms of response
time and throughput with previous studies
of dedicated CDNs
• Back-end providers have sufficient
performance and reliability to be used to
host replicas in the MetaCDN system

27. MetaCDN features in
planning / development
• Support as many providers as possible
• Ensures redundancy, reliability, coverage
and choice (reduces single vendor lock-
in)
• Windows Azure Storage Service support
will be added ASAP.
• Other storage & delivery services can be
rapidly added as they are released....

28. MetaCDN features in
planning / development
• Create Apache module (mod_metacdn) to
utilise at content origin sites
• Module would utilise MetaCDN web
services to maintain optimal deployment
• Maximise performance during peak load
via intelligent replica deployment
• Minimise storage cost during periods of
low demand

29. Collaborations
• Always looking for people to collaborate
on the project
• Work to be done on:
• Load balancing / redirection algorithms
• Intelligent Caching / replication
algorithms
• Security / Access Control of content
• Improving MetaCDN Web Services
• Please contact me if you are interested...

30. Acknowledgements
• Thanks to reviewers for useful feedback.
• Australian Research Council (ARC) for
funding the project.
• Cory & Barry (Nirvanix) and Eric
(Rackspace/Mosso) for development
support.

31. Thanks
Latest information will be available at:
www.metacdn.org
International Workshop on Cloud
Computing (Cloud 2009):
http://www.gridbus.org/cloud2009