SlideShare ist ein Scribd-Unternehmen logo
1 von 41
Simple, Available Cloud Storage
For Cloudstack
Overview
On March 27, 2012


Basho
announced a new
product called

Riak CS
On September 5, 2012


   BASHO
joined Apache Cloudstack
On March 20, 2013


Riak CS
became open source
Riak CS
        is... storage
enterprise cloud
built                    g
                       in S3-compatibility
   on top          fe r
of           o f
                           multi-tenancy

Riak                     per user reporting
                        large object storage
Enabling you to host your own


      PUBLIC &
      PRIVATE CLOUDS

             or….
Reliable Storage Behind Apps
Basho's Commits

@john_burwell 's contribution:
S3-backed secondary storage feature in 4.1.0
Uses S3 to sync secondary storage across zones


Long term: (shhhhhh!)
Native S3 Support
Federated authentication and authorization
DataPipe
  blog.datapipe.com/datapipe-cloudstack

          “Riak CS provides the high-performance,

                              distributed datastore

          we need to deliver a sound foundation for

                      our cloud storage needs now

                and for many years into the future”

       - Ed Laczynski, VP Cloud Strategy, Datapipe.
Yahoo!
   “Today, Yahoo! leverages Riak CS Enterprise to offer an

               S3-compatible public cloud storage service,

                    as well as dedicated hosting options ...

       Yahoo! is highly supportive of open source software

            and we view Basho’s (OSS) announcement as

                             a positive move that will work

                       to accelerate its ability to innovate

              and ultimately strengthen our cloud platform.”

      - Shingo Saito, cloud product manager, Yahoo!
About Riak
Riak
   Dynamo-inspired key/value store

   Written in Erlang with C/C++

   Open source under Apache 2 license

   Thousands of production deployments
Riak
  High availability

  Low-latency

  Horizontal scalability

  Fault-tolerance

  Ops friendliness
Riak
   Masterless
    • No master/slave or different roles
    • All nodes are equal
    • Write availability and scalability
    • All nodes can accept/route requests
Riak
   No Sharding
     • Consistent hashing
     • Prevents “hot spots”
     • Lowers operational burden of scale
     • Data rebalanced automatically
Riak
  Availability and Fault-Tolerance
    • Automatically replicates data
    • Read and write data during hardware
       failure and network partition
    • Hinted handoff
How It
Works
Riak CS
Stanchion
Riak
1
    Riak CS node
      for every
    node of Riak
Large Object                        1. User uploads an
                                                                                                      object


  S3     Reporting             S3      Reporting                S3      Reporting            S3       Reporting            S3          Reporting
  API      API                 API       API                    API       API                API        API                API           API

   Riak CS                      Riak CS                          Riak CS                       Riak CS                          Riak CS


        1 MB   1 MB   1 MB   1 MB   1 MB   1 MB   1 MB   1 MB    1 MB   1 MB   1 MB   1 MB   1 MB   1 MB   1 MB   1 MB   1 MB   1 MB



                                                                                                 2. Riak CS
   3. Riak CS                                                         Riak                     breaks object
streams chunks
                                                                      Node
                                                                                             into 1 MB chunks
 to Riak nodes                                    Riak                                Riak
                                                  Node                                Node




                                                         Riak                  Riak
                                                                                                    4. Riak replicates
                                                         Node                  Node                 and stores chunks
IC S
     S T
   BA EP
     C
 CON
               USERS
                multi-tenancy:
              Riak CS will track
            individual usage/stats

users identified by     users authenticated by
   access_key                 secret_key
IC S
    S T
  BA EP
    C
CON
      BUCKETS
       users create buckets.
      buckets are like folders.
      store objects in buckets.
     names are globally unique.
IC S
    S T
  BA EP
    C
CON
       OBJECTS
         stored in buckets.
        objects are opaque.
         store any file type.
Features
Riak CS
   Large Object Support
     • Started with 5GB / object
     • Now have multipart upload
     • Content agnostic
Riak CS
   S3-Compatible API
     • Use existing S3 libraries and tools
     • RESTful operations
     • Multipart upload
     • S3-style ACLs for object/bucket
       permissions
     • S3 authentication scheme
Riak CS
   Administration and Users
     • Interface for user creation, deletion,
       and credentials
     • Configure so only admins can create
       users
Riak CS
   New Stuff in Riak 1.3
    • Multipart upload: parts between 5MB
       and 5GB
    • Support for GET range queries
    • Restrict access to buckets based on
       source IP
Riak CS
Riak CS
   Packages
     • Debian
     • Ubuntu
     • FreeBSD
     • Mac
     • Red Hat Enterprise
     • Fedora
     • SmartOS
     • Solaris
     • Source
Operations
built-in
stats &
           track access &
           storage per user

           inspect ops with

DTrace     DTrace probes

           monitor total
support    cluster ops
OPERATIONAL STATS
       exposed via HTTP resource: /riak-cs/stats



HISTOGRAMS & COUNTERS
      block               bucket                object
                      LIST KEYS, CREATE,    GET, PUT, DELETE
  GET, PUT, DELETE
                     DELETE, GET/PUT ACL   HEAD, GET/PUT ACL
THE
      “USAGE”
      BUCKET
TRACK INDIVIDUAL USER’S
ACCESS STORAGE
QUERY USAGE STATS

 Storage and access statistics tracked on
 per-user basis, as rollups for slices of time

 •Operations, Count, BytesIn,
 BytesOut, + system and user
 error
 •Objects, Bytes
Enterprise
Multi-Datacenter Replication

        • For active backups, availability zones,
          disaster recovery, global traffic
        • Real-time or full-sync
        • 24/7 support
        • Per-node or storage-based pricing
SIGN UP FOR AN
ENTERPRISE DEVELOPER
        TRIAL
            basho.com

     http://docs.basho.com/
Riak London
A distributed systems
   meet/drink up

  www.meetup.com/riak-london
github.com/basho
twitter.com/basho
 docs.basho.com
Q&A
 @_stu_

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Scalable Media Processing in the Cloud (MED302) | AWS re:Invent 2013
Scalable Media Processing in the Cloud (MED302) | AWS re:Invent 2013Scalable Media Processing in the Cloud (MED302) | AWS re:Invent 2013
Scalable Media Processing in the Cloud (MED302) | AWS re:Invent 2013
 
Scaling spark on kubernetes at Lyft
Scaling spark on kubernetes at LyftScaling spark on kubernetes at Lyft
Scaling spark on kubernetes at Lyft
 
Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...
Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...
Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...
 
AWS database services
AWS database servicesAWS database services
AWS database services
 
Media Success Stories from the Cloud
Media Success Stories from the CloudMedia Success Stories from the Cloud
Media Success Stories from the Cloud
 
Deep Dive: Amazon RDS
Deep Dive: Amazon RDSDeep Dive: Amazon RDS
Deep Dive: Amazon RDS
 
AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...
AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...
AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...
 
Deep Dive on Amazon RDS
Deep Dive on Amazon RDSDeep Dive on Amazon RDS
Deep Dive on Amazon RDS
 
AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances (...
AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances (...AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances (...
AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances (...
 
Dev ops for big data cluster management tools
Dev ops for big data  cluster management toolsDev ops for big data  cluster management tools
Dev ops for big data cluster management tools
 
Introducing Kubernetes
Introducing Kubernetes Introducing Kubernetes
Introducing Kubernetes
 
Simple, Scalable and Highly Durable NAS in the Cloud - Amazon EFS
Simple, Scalable and Highly Durable NAS in the Cloud - Amazon EFSSimple, Scalable and Highly Durable NAS in the Cloud - Amazon EFS
Simple, Scalable and Highly Durable NAS in the Cloud - Amazon EFS
 
Responding to Digital Transformation With RDS Database Technology
Responding to Digital Transformation With RDS Database TechnologyResponding to Digital Transformation With RDS Database Technology
Responding to Digital Transformation With RDS Database Technology
 
AWS re:Invent 2016: Moving Mountains: Netflix's Migration into VPC (NET304)
AWS re:Invent 2016: Moving Mountains: Netflix's Migration into VPC (NET304)AWS re:Invent 2016: Moving Mountains: Netflix's Migration into VPC (NET304)
AWS re:Invent 2016: Moving Mountains: Netflix's Migration into VPC (NET304)
 
Day 3 - AWS MySQL Relational Database Service Best Practices for Performance ...
Day 3 - AWS MySQL Relational Database Service Best Practices for Performance ...Day 3 - AWS MySQL Relational Database Service Best Practices for Performance ...
Day 3 - AWS MySQL Relational Database Service Best Practices for Performance ...
 
AWS Database Services-Philadelphia AWS User Group-4-17-2018
AWS Database Services-Philadelphia AWS User Group-4-17-2018AWS Database Services-Philadelphia AWS User Group-4-17-2018
AWS Database Services-Philadelphia AWS User Group-4-17-2018
 
Introduction to CloudStack: How to Deploy and Manage Infrastructure-as-a-Serv...
Introduction to CloudStack: How to Deploy and Manage Infrastructure-as-a-Serv...Introduction to CloudStack: How to Deploy and Manage Infrastructure-as-a-Serv...
Introduction to CloudStack: How to Deploy and Manage Infrastructure-as-a-Serv...
 
EC2 and S3 Level 100
EC2 and S3 Level 100EC2 and S3 Level 100
EC2 and S3 Level 100
 
Netflix security monkey overview
Netflix security monkey overviewNetflix security monkey overview
Netflix security monkey overview
 
Netflix in the cloud 2011
Netflix in the cloud 2011Netflix in the cloud 2011
Netflix in the cloud 2011
 

Andere mochten auch

Andere mochten auch (9)

Cloudstack European user group 11 april 2013
Cloudstack European user group   11 april 2013Cloudstack European user group   11 april 2013
Cloudstack European user group 11 april 2013
 
Oliver leech cloudstack
Oliver leech   cloudstackOliver leech   cloudstack
Oliver leech cloudstack
 
European Cloudstack User Group
European Cloudstack User GroupEuropean Cloudstack User Group
European Cloudstack User Group
 
Contributing to Apache CloudStack
Contributing to Apache CloudStackContributing to Apache CloudStack
Contributing to Apache CloudStack
 
TIAD 2016 : Real-Time Data Processing Pipeline & Visualization with Docker, S...
TIAD 2016 : Real-Time Data Processing Pipeline & Visualization with Docker, S...TIAD 2016 : Real-Time Data Processing Pipeline & Visualization with Docker, S...
TIAD 2016 : Real-Time Data Processing Pipeline & Visualization with Docker, S...
 
CloudStack at Schuberg Philis
CloudStack at Schuberg PhilisCloudStack at Schuberg Philis
CloudStack at Schuberg Philis
 
How to add a new hypervisor to CloudStack - Lessons learned from Hyper-V effort
How to add a new hypervisor to CloudStack - Lessons learned from Hyper-V effortHow to add a new hypervisor to CloudStack - Lessons learned from Hyper-V effort
How to add a new hypervisor to CloudStack - Lessons learned from Hyper-V effort
 
Spark Summit EU 2015: SparkUI visualization: a lens into your application
Spark Summit EU 2015: SparkUI visualization: a lens into your applicationSpark Summit EU 2015: SparkUI visualization: a lens into your application
Spark Summit EU 2015: SparkUI visualization: a lens into your application
 
Ansible & CloudStack - Configuration Management
Ansible & CloudStack - Configuration ManagementAnsible & CloudStack - Configuration Management
Ansible & CloudStack - Configuration Management
 

Ähnlich wie Riak CS in Cloudstack

Riak seattle-meetup-august
Riak seattle-meetup-augustRiak seattle-meetup-august
Riak seattle-meetup-august
pharkmillups
 
451 research basho previews distributed database and momentum
451 research  basho previews distributed database and momentum451 research  basho previews distributed database and momentum
451 research basho previews distributed database and momentum
Frank Wu
 

Ähnlich wie Riak CS in Cloudstack (20)

How does Riak compare to Cassandra? [Cassandra London User Group July 2011]
How does Riak compare to Cassandra? [Cassandra London User Group July 2011]How does Riak compare to Cassandra? [Cassandra London User Group July 2011]
How does Riak compare to Cassandra? [Cassandra London User Group July 2011]
 
Hugfr SPARK & RIAK -20160114_hug_france
Hugfr  SPARK & RIAK -20160114_hug_franceHugfr  SPARK & RIAK -20160114_hug_france
Hugfr SPARK & RIAK -20160114_hug_france
 
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxBetter, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
 
Riak at Engine Yard Cloud
Riak at Engine Yard CloudRiak at Engine Yard Cloud
Riak at Engine Yard Cloud
 
Os riak1-pdf
Os riak1-pdfOs riak1-pdf
Os riak1-pdf
 
Building Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak CoreBuilding Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak Core
 
Riak seattle-meetup-august
Riak seattle-meetup-augustRiak seattle-meetup-august
Riak seattle-meetup-august
 
Getting started with Riak in the Cloud
Getting started with Riak in the CloudGetting started with Riak in the Cloud
Getting started with Riak in the Cloud
 
Open stack in sina
Open stack in sinaOpen stack in sina
Open stack in sina
 
Riak - From Small to Large - StrangeLoop
Riak - From Small to Large - StrangeLoopRiak - From Small to Large - StrangeLoop
Riak - From Small to Large - StrangeLoop
 
Riak - From Small to Large
Riak - From Small to LargeRiak - From Small to Large
Riak - From Small to Large
 
AWS re:Invent 2016 - Scality's Open Source AWS S3 Server
AWS re:Invent 2016 - Scality's Open Source AWS S3 ServerAWS re:Invent 2016 - Scality's Open Source AWS S3 Server
AWS re:Invent 2016 - Scality's Open Source AWS S3 Server
 
Getting started with MariaDB with Docker
Getting started with MariaDB with DockerGetting started with MariaDB with Docker
Getting started with MariaDB with Docker
 
Leveraging Scala and Akka to build NSDb
Leveraging Scala and Akka to build NSDbLeveraging Scala and Akka to build NSDb
Leveraging Scala and Akka to build NSDb
 
Migrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global CassandraMigrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global Cassandra
 
What’s new in Apache Spark 2.3
What’s new in Apache Spark 2.3What’s new in Apache Spark 2.3
What’s new in Apache Spark 2.3
 
Modernise your Data Warehouse - AWS Summit Sydney 2018
Modernise your Data Warehouse - AWS Summit Sydney 2018Modernise your Data Warehouse - AWS Summit Sydney 2018
Modernise your Data Warehouse - AWS Summit Sydney 2018
 
Akka Streams And Kafka Streams: Where Microservices Meet Fast Data
Akka Streams And Kafka Streams: Where Microservices Meet Fast DataAkka Streams And Kafka Streams: Where Microservices Meet Fast Data
Akka Streams And Kafka Streams: Where Microservices Meet Fast Data
 
451 research basho previews distributed database and momentum
451 research  basho previews distributed database and momentum451 research  basho previews distributed database and momentum
451 research basho previews distributed database and momentum
 
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN MainzFully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
 

Mehr von ShapeBlue

Mehr von ShapeBlue (20)

CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
 
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHubHow We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
 
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
 
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIOHow We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
 
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
 
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
 
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
 
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
 
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
 
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
 
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
 
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
 
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueElevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
 
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
 
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
 
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlueWhat’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
 

Kürzlich hochgeladen

Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Kürzlich hochgeladen (20)

Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 

Riak CS in Cloudstack

  • 1. Simple, Available Cloud Storage For Cloudstack
  • 3. On March 27, 2012 Basho announced a new product called Riak CS
  • 4. On September 5, 2012 BASHO joined Apache Cloudstack
  • 5. On March 20, 2013 Riak CS became open source
  • 6. Riak CS is... storage enterprise cloud built g in S3-compatibility on top fe r of o f multi-tenancy Riak per user reporting large object storage
  • 7. Enabling you to host your own PUBLIC & PRIVATE CLOUDS or…. Reliable Storage Behind Apps
  • 8. Basho's Commits @john_burwell 's contribution: S3-backed secondary storage feature in 4.1.0 Uses S3 to sync secondary storage across zones Long term: (shhhhhh!) Native S3 Support Federated authentication and authorization
  • 9. DataPipe blog.datapipe.com/datapipe-cloudstack “Riak CS provides the high-performance, distributed datastore we need to deliver a sound foundation for our cloud storage needs now and for many years into the future” - Ed Laczynski, VP Cloud Strategy, Datapipe.
  • 10. Yahoo! “Today, Yahoo! leverages Riak CS Enterprise to offer an S3-compatible public cloud storage service, as well as dedicated hosting options ... Yahoo! is highly supportive of open source software and we view Basho’s (OSS) announcement as a positive move that will work to accelerate its ability to innovate and ultimately strengthen our cloud platform.” - Shingo Saito, cloud product manager, Yahoo!
  • 12. Riak Dynamo-inspired key/value store Written in Erlang with C/C++ Open source under Apache 2 license Thousands of production deployments
  • 13. Riak High availability Low-latency Horizontal scalability Fault-tolerance Ops friendliness
  • 14. Riak Masterless • No master/slave or different roles • All nodes are equal • Write availability and scalability • All nodes can accept/route requests
  • 15. Riak No Sharding • Consistent hashing • Prevents “hot spots” • Lowers operational burden of scale • Data rebalanced automatically
  • 16. Riak Availability and Fault-Tolerance • Automatically replicates data • Read and write data during hardware failure and network partition • Hinted handoff
  • 19. 1 Riak CS node for every node of Riak
  • 20. Large Object 1. User uploads an object S3 Reporting S3 Reporting S3 Reporting S3 Reporting S3 Reporting API API API API API API API API API API Riak CS Riak CS Riak CS Riak CS Riak CS 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 2. Riak CS 3. Riak CS Riak breaks object streams chunks Node into 1 MB chunks to Riak nodes Riak Riak Node Node Riak Riak 4. Riak replicates Node Node and stores chunks
  • 21. IC S S T BA EP C CON USERS multi-tenancy: Riak CS will track individual usage/stats users identified by users authenticated by access_key secret_key
  • 22. IC S S T BA EP C CON BUCKETS users create buckets. buckets are like folders. store objects in buckets. names are globally unique.
  • 23. IC S S T BA EP C CON OBJECTS stored in buckets. objects are opaque. store any file type.
  • 25. Riak CS Large Object Support • Started with 5GB / object • Now have multipart upload • Content agnostic
  • 26. Riak CS S3-Compatible API • Use existing S3 libraries and tools • RESTful operations • Multipart upload • S3-style ACLs for object/bucket permissions • S3 authentication scheme
  • 27. Riak CS Administration and Users • Interface for user creation, deletion, and credentials • Configure so only admins can create users
  • 28. Riak CS New Stuff in Riak 1.3 • Multipart upload: parts between 5MB and 5GB • Support for GET range queries • Restrict access to buckets based on source IP
  • 30. Riak CS Packages • Debian • Ubuntu • FreeBSD • Mac • Red Hat Enterprise • Fedora • SmartOS • Solaris • Source
  • 32. built-in stats & track access & storage per user inspect ops with DTrace DTrace probes monitor total support cluster ops
  • 33. OPERATIONAL STATS exposed via HTTP resource: /riak-cs/stats HISTOGRAMS & COUNTERS block bucket object LIST KEYS, CREATE, GET, PUT, DELETE GET, PUT, DELETE DELETE, GET/PUT ACL HEAD, GET/PUT ACL
  • 34. THE “USAGE” BUCKET TRACK INDIVIDUAL USER’S ACCESS STORAGE
  • 35. QUERY USAGE STATS Storage and access statistics tracked on per-user basis, as rollups for slices of time •Operations, Count, BytesIn, BytesOut, + system and user error •Objects, Bytes
  • 37. Multi-Datacenter Replication • For active backups, availability zones, disaster recovery, global traffic • Real-time or full-sync • 24/7 support • Per-node or storage-based pricing
  • 38. SIGN UP FOR AN ENTERPRISE DEVELOPER TRIAL basho.com http://docs.basho.com/
  • 39. Riak London A distributed systems meet/drink up www.meetup.com/riak-london

Hinweis der Redaktion

  1. Welcome/Intro
  2. Here ’ s the basics
  3. Very high level discussion, segue into brief discussion of Riak
  4. What you get is a platform on which you can host your own public and private clouds.
  5. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  6. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  7. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  8. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  9. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  10. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  11. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  12. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  13. a Riak CS stack is composed of 3 critical components. Riak CS exposes an API to the users and is responsible for logging/tracking stats. All the data is stored in and retrieved from Riak. Run multiple instances of Riak and Riak CS for scale. Theres a third component, a single instance of a piece of software called stanchion that is responsible for tying it all together. Stanchion in essence provides the S3-like behavior at an architectural level, ensures user and bucket uniqueness globally, etc....
  14. 1-to-1 pairing, and why.
  15. 1. user PUTs object into Riak CS. The request will be via an S3 API and signed by their credentials. 2. once authenticated, object is chunked (remind why this is important) 3. as object is chunked, chunks sent to Riak. (you can use haproxy in the middle here) 4. Riak stores the chunks, yay!
  16. Riak CS is multi-tenant. Each user is assigned an access_key and a secret_key. Users are authenticated by the system by signing requests using a combination of both keys. If the keys are valid, the requests will be allowed; else, denied. User details stored in “ user ” bucket, identified by access_key. Furthermore, every user ’ s activity will be tracked by Riak CS and stored for billing/metering purposes(more later)
  17. Objects are stored in buckets. Users ’ s can create and remove buckets as well as list their contents. Buckets are essentially a namespace, and are very much like folders. Bucket names must be globally unique, so if you have two users both try to create a bucket named “ kittens ” , whoever creates that bucket first will own it, etc.
  18. Put objects in buckets. Objects are chunked and replicated, but that all happens behind the scenes and not exposed to the user.
  19. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  20. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  21. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  22. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  23. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  24. You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
  25. Riak CS provides stats on user activity and total cluster operations, as well as ships with DTrace probes you can use to inspect/debug a live system. So at any given time you can monitor a Riak CS cluster for both expected behavior and anomalies. From an administrative perspective, (as mentioned earlier) Riak CS will track each individual user ’ s activity, so that you can define usage limits and billing policies if necessary.
  26. Riak CS, just like Riak, uses Boundary ’ s Folsom stats library for monitoring cluster operations. These start when Riak CS starts, are not persisted to disk. Get stats with an HTTP request to /riak-cs/stats. You ’ ll get back counters and histograms that track the total number of operations performed on blocks, buckets and objects. For instance, see the total number of GET or PUT operations on objects in the Riak CS cluster. These stats are going to be most useful if you ’ re trying to diagnose unexpected behavior. Hopefully that ’ s never the case, but shit happens.
  27. RiakCS has a reserved namespace for tracking user activity. This is the “ usage ” bucket and is the foundation for metering and building custom billing policies in Riak CS. Every time a user performs an operation, RiakCS will store this data in an object in the usage bucket identified by that users ’ s access_key. You can configure the frequency with which these reports are persisted as well as the ability for user ’ s to request their own usage statistics.
  28. Limitations = no periods greater than 31 days
  29. Limitations = no periods greater than 31 days
  30. Introduce yourself