SlideShare ist ein Scribd-Unternehmen logo
1 von 42
Downloaden Sie, um offline zu lesen
7 STEPS
to Roll-out a Private
OpenStack Swift Cluster
          Joe Arnold
          OpenStack Conference
                       April 19, 2012
OpenStack Swift
 Powers Rackspace Cloud Files
Data is
                                           Sticky

http://www.ïŹ‚ickr.com/photos/josefgrunig/
App       App       App


Storage   Storage   Storage


Compute   Compute   Compute
Rackspace     Openstack
Cloud Files    Launch
                          59 developers
                          contributing
                          to Swift
    ’09          ’10
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
Swi ft is good for
          - Application data
          - Filesharing
          - Backups
          - Unstructured data
Swift is laser focused
   Swift Attributes




          Scalable
Swift Attributes




      Durable
Swift Attributes




Concurrency
Not good at . . .
 Databases - transactional workloads
 No File Locking
Not good at . . .
            Single
            Stream
              I/O
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
Data Durability - Zones
Data Durability - Replication & Consistency
High Availability

 Proxy

   +

Zones
Example: 2 Nodes



  - Whole Files
  - 3 Replicas
  - Withstand multiple
    disk failures
Example: 5 Nodes


- Withstand multiple
  Node Failures
- Consider Separate
  Proxy Tier
Example: 50 Nodes
- Withstand Rack Failures
- Requires Separate
  Proxy Tier
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
Swi ft is economical

                     $0.40/GB Capex

Example: $42,520 for 105TB usable
1 x Dell r815 ($4,500)
2x Supermicro JBODs ($5,400)
144 x 3TB hard drives ($33,120)
Thailand ïŹ‚ooding drive prices
Swi ft is economical
     Size        ConïŹguration            Cost    $ / GB

             2 ToR Switch
             2 Proxy/Auth Servers
    145 TB   5 Object Stores             $95k    $0.65
             ...and cables, rack, etc

             2 Agg Switches
             6 Proxy/Auth Servers
    1.3 PB   5 ToR Switches              $750k   $0.52
             50 Object Stores
             ...and cables, racks, etc
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
High volume web /
              mobile application


 Various     Enterprise Filesharing

Work loads
                  Backup /
                Active Archive
Measure and Test


                                  Frequency
      File Sizes   Concurrency   of upload vs
                                  download
RAM - 12GB
 PROXY NODES     Networking - 2 x 10GbE



            CPU - quad-core
           RAM - 1GB of RAM / 2TB
           Drives - 2-3 TB Drives
           Controller Cards - No RAID
           Using SSDs - User Accounts &
                        Container Listings
STORAGE NODES
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
People are
Expensive
Added StatsD to Watch EVERYTHING

       Swift Process
        Swift Process
         Swift Process
          Swift Process        Lightweight
           Swift Process
            Swift Process      UDP StatsD


                                       StatsD



                Time-Series Backend
                  Ganglia / Graphite
Integrating Components around Swift

                   Load Balancing


         SSL Termination     Authentication


               OpenStack Object Storage


         OS, Node & Drive     Swift Runtime
           Monitoring          Monitoring


             Operating System + Kernel
Managing a distributed system


                                Controller
Capacity adjustments
Capacity




                       time
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
S3
           Cloud Files
                         ’10
’06        ’08
When to use Swift (and when not to)


Achieving High Durability and Availability (cluster design)


         Convincing Your Boss (the economics)


    Selecting Appropriate Hardware (benchmarking)


 Knowing How It’s Working (management & monitoring)


       What Interfaces Exist (compatible clients)


   How to Build Killer Apps (features for developers)
HTTP API




Data              Storage
Media Delivery

Direct to Device

Integrates to CDN

Streaming

Static Documents

Direct from Device
TRY              http://swiftstack.com



                    joe@swiftstack.com
                    @joearnold - @swiftstack




          rkshop     àčInstall
Swi ft wo            àčBuild an Application
 tom   orrow at
  1:3  0 - 4 : 00

Weitere Àhnliche Inhalte

Andere mochten auch

Deploying OpenStack Object Storage (Swift)
Deploying OpenStack Object Storage (Swift)Deploying OpenStack Object Storage (Swift)
Deploying OpenStack Object Storage (Swift)
Juan José Martínez
 

Andere mochten auch (11)

Openstack swift - VietOpenStack 6thmeeetup
Openstack swift - VietOpenStack 6thmeeetupOpenstack swift - VietOpenStack 6thmeeetup
Openstack swift - VietOpenStack 6thmeeetup
 
Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012
 
Deploying OpenStack Object Storage (Swift)
Deploying OpenStack Object Storage (Swift)Deploying OpenStack Object Storage (Swift)
Deploying OpenStack Object Storage (Swift)
 
OpenStack Swift In the Enterprise
OpenStack Swift In the EnterpriseOpenStack Swift In the Enterprise
OpenStack Swift In the Enterprise
 
Fossasia 16 Integrating oVirt, Foreman and Katello to empower your data-center
Fossasia 16 Integrating oVirt, Foreman and Katello to empower your data-centerFossasia 16 Integrating oVirt, Foreman and Katello to empower your data-center
Fossasia 16 Integrating oVirt, Foreman and Katello to empower your data-center
 
Open vStorage Meetup - Santa Clara 04/16
Open vStorage Meetup -  Santa Clara 04/16Open vStorage Meetup -  Santa Clara 04/16
Open vStorage Meetup - Santa Clara 04/16
 
Turning object storage into vm storage
Turning object storage into vm storageTurning object storage into vm storage
Turning object storage into vm storage
 
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC IsilonImproving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
 
Deploying and managing gluster using ovirt - fudcon2015
Deploying and managing gluster using ovirt - fudcon2015Deploying and managing gluster using ovirt - fudcon2015
Deploying and managing gluster using ovirt - fudcon2015
 
EMC ScaleIO
EMC ScaleIOEMC ScaleIO
EMC ScaleIO
 
Issues of OpenStack multi-region mode
Issues of OpenStack multi-region modeIssues of OpenStack multi-region mode
Issues of OpenStack multi-region mode
 

KĂŒrzlich hochgeladen

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

KĂŒrzlich hochgeladen (20)

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 

7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack Conference -2012-04-17

  • 1. 7 STEPS to Roll-out a Private OpenStack Swift Cluster Joe Arnold OpenStack Conference April 19, 2012
  • 2.
  • 3. OpenStack Swift Powers Rackspace Cloud Files
  • 4. Data is Sticky http://www.ïŹ‚ickr.com/photos/josefgrunig/
  • 5. App App App Storage Storage Storage Compute Compute Compute
  • 6. Rackspace Openstack Cloud Files Launch 59 developers contributing to Swift ’09 ’10
  • 7. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 8. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 9. Swi ft is good for - Application data - Filesharing - Backups - Unstructured data
  • 10. Swift is laser focused Swift Attributes Scalable
  • 11. Swift Attributes Durable
  • 13. Not good at . . . Databases - transactional workloads No File Locking
  • 14. Not good at . . . Single Stream I/O
  • 15. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 17. Data Durability - Replication & Consistency
  • 19. Example: 2 Nodes - Whole Files - 3 Replicas - Withstand multiple disk failures
  • 20. Example: 5 Nodes - Withstand multiple Node Failures - Consider Separate Proxy Tier
  • 21. Example: 50 Nodes - Withstand Rack Failures - Requires Separate Proxy Tier
  • 22. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 23. Swi ft is economical $0.40/GB Capex Example: $42,520 for 105TB usable 1 x Dell r815 ($4,500) 2x Supermicro JBODs ($5,400) 144 x 3TB hard drives ($33,120) Thailand ïŹ‚ooding drive prices
  • 24. Swi ft is economical Size ConïŹguration Cost $ / GB 2 ToR Switch 2 Proxy/Auth Servers 145 TB 5 Object Stores $95k $0.65 ...and cables, rack, etc 2 Agg Switches 6 Proxy/Auth Servers 1.3 PB 5 ToR Switches $750k $0.52 50 Object Stores ...and cables, racks, etc
  • 25. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 26. High volume web / mobile application Various Enterprise Filesharing Work loads Backup / Active Archive
  • 27. Measure and Test Frequency File Sizes Concurrency of upload vs download
  • 28. RAM - 12GB PROXY NODES Networking - 2 x 10GbE CPU - quad-core RAM - 1GB of RAM / 2TB Drives - 2-3 TB Drives Controller Cards - No RAID Using SSDs - User Accounts & Container Listings STORAGE NODES
  • 29. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 31. Added StatsD to Watch EVERYTHING Swift Process Swift Process Swift Process Swift Process Lightweight Swift Process Swift Process UDP StatsD StatsD Time-Series Backend Ganglia / Graphite
  • 32. Integrating Components around Swift Load Balancing SSL Termination Authentication OpenStack Object Storage OS, Node & Drive Swift Runtime Monitoring Monitoring Operating System + Kernel
  • 33. Managing a distributed system Controller
  • 35. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 36. S3 Cloud Files ’10 ’06 ’08
  • 37.
  • 38. When to use Swift (and when not to) Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  • 39. HTTP API Data Storage
  • 40. Media Delivery Direct to Device Integrates to CDN Streaming Static Documents Direct from Device
  • 41.
  • 42. TRY http://swiftstack.com joe@swiftstack.com @joearnold - @swiftstack rkshop àčInstall Swi ft wo àčBuild an Application tom orrow at 1:3 0 - 4 : 00