SlideShare a Scribd company logo
1 of 22
Download to read offline
Using OpenStack
In A Traditional Hosting Environment

   Jun Park (Ph.D.), Sr. Systems Architect
     Mike Wilson, Sr. Systems Architect
               EIG/Bluehost

         OpenStack Summit 2013
•   Scale @10x what we were in about a year’s time
•   Needed a system to manage it all
•   Provisioning these systems had to be automated
•   System had to scale to multiple data centers
•   Btw, we have 2 months to come up with this




EIG/Bluehost                                    2
High level requirements
• Centralized management, horizontal scalability.
• Abstractions for physical and logical deployments of
  devices and resources.
• Open-source project with lots of support and momentum
• Typical Cloud features (migration, imaging, provisioning,
  etc.)

   Easy transition into future cloud offering with support for
                      industry standard APIs
                                 =
                           OpenStack!


EIG/Bluehost                                                     3
Bluehost Environment
• Scale
    – More than 10,000 physical
      servers
    – Adding hundreds of nodes
      per day
• Network
    – Public network directly attached to VMs
    – Plan on adding private network later
• OpenStack/Folsom Components
    –   Nova (compute, api, scheduler)
    –   Quantum (network)
    –   AMQP: Qpid (messaging)
    –   Mysql (db)

EIG/Bluehost                                    4
Outline
• Scalability/Stability
• Rethink Quantum Network
• Operational Issues

• Wrap-up/Conclusions




EIG/Bluehost                5
Why Difficult To Scale up?
• Components that don’t scale
    – Messaging system           Look at that line!

    – Mysql server
    – Heavy APIs
• Hard to Diagnose
    – No simulator/emulator for high scalability testing
    – No quantitative guide as to how to scale up
    – No detailed error message or even not at all
    – Requires detailed knowledge of codebase

EIG/Bluehost                                               6
Nova Enhancements
• Monitoring/troubleshooting
    – Added service ping (similar to grizzly)
    – Additional task_states
    – Better errors to instance_faults
• Functionality
    – Added LVM resize and injection
    – Added stop_soft and stop_hard similar to reboot
• Stability/Scalability
    – Some bug fixes to OVS VIF driver
    – Added custom scheduler
EIG/Bluehost                                            7
MySQL/Innodb Concurrency
• Nova behavior
    –   Direct connection to DB (addressed in grizzly though)
    –   Too MANY queries; much more read than write
    –   Some queries often return huge number of results
    –   Periodicity of loads due to periodic tasks
• Mysql behavior
    – innodb_thread_concurrency = num of threads that can
      execute queries in the queue
    – innodb_concurrency_tickets = number of rows you can
      return before requeue (default 500!)
    – With default settings your mysql server will fall over with a
      few thousand compute nodes

EIG/Bluehost                                                      8
So the answer is just increase concurrency and tickets right?

                                 Sadly, no.




Tuning concurrency and tickets is a must! But we still requeue.
We don’t know why, but we have a workaround.

Our workaround:
Send most read queries to a cluster of mysql slaves.
I say most because many queries would be sensitive to
possible slave replication lag

EIG/Bluehost                                                      9
Messaging System
• Qpid Chosen
    –   Speed and easy cluster ability
    –   Incredibly unstable at large scale (Even at small scale)
    –   Possibly poorly configured
    –   Older release in CentOS6
• Problem
    – Broker bottleneck
    – Unnecessarily complex
• Recommendation
    – AVOID BROKER!
    – ZeroMQ as replacement


EIG/Bluehost                                                       10
BlueHost OpenStack

 Group of Controllers                 mysql master
• Nova-api                                                  Read-only mysql slave
                                                             Read-only mysql slave
• Nova-scheduler                                               Read-only mysql slaves
• Quantum-server
• Keystone

                Load Balanced


                                      Server Farm
                                > 10,000 physical servers
                                                                      Host
                                                                BH-enhanced OVS
 Qpid servers
                                                                 Quantum Plugin
                                                               BH-enhanced Nova-
EIG/Bluehost                                                        compute
Rethink Quantum Network
• Problem
    – Quantum API only for DB; no efficient API for Nova
    – No API-around design for actual network objects
    – Premature OpenvSwitch
      (OVS) quantum plugin

• Our approach
    – Adding intelligence to OVS plugin
    – Optimizing API
        • E.g., get_instance_nw_info() in nova; Python-level join operation
    – Expand current implicit communication with nova-
      compute (although not ideal)

EIG/Bluehost                                                                  12
OpenvSwitch (OVS)
• Support
    – OpenFlow 1.0 (1.2, 1.3 in experiment)
    – Various Hypervisors including KVM
    – Merged into main stream since Linux 3.3

• Functionality
    – Filtering rules and associated actions
        • E.g., anti-IP spoofing, DMAC filtering
    – Replacement or superset of ebtables/iptables/linuxbridge
    – QoS or Linux traffic control (TC)
    – Various third party controllers for full-fledged OF
      functionality

EIG/Bluehost                                                 13
Quantum With Nova-Compute
 Implicit communication
   with nova-compute
                                1. Nova boot API                Why nova-compute
   Controller                                                    still create tap?

Nova-api server                                                            Host
                           2. Call quantum-api to create port
                                                                     Nova-compute
Quantum-server                             4. Create OVS tap
                                                                   6. Do the rest of steps
                                5. Set up external-ids on tap
                                                                   for creating a new VM
    3. Create port in DB
                           1. Monitor any change of external-
                           ids of the existing taps
Quantum                    2. Call quantum-api to get port info     Quantum-agent
  DB
                           3. Modify external-ids on tap &
                           setup OVS flows
EIG/Bluehost                                                                          14
BH-Enhanced OVS Quantum Plugin
• Idiosyncratic Requirements
    – Focus on a Zone; plan on addressing multiple-datacenters issue
    – Direct public IP using flat shared provider network with No NAT
    – Still required to be isolated while allowing intra-traffic among VMs
• Our Approach
    – Developing “Distributed OpenFlow controller” using OVS plugin
    – Either no-tag or 24bit ID (e.g., QinQ, VxLAN, GRE), but NO VLAN
    – Caveats: Neither API-around approach nor Virtual Appliance
• New Features
    –   Anti-IP/ARP spoofing OF flows
    –   Multiple-IPs per public port
    –   Network bandwidth throttling (QoS)
    –   Optimal Intra-traffic flows
EIG/Bluehost                                                                 15
BH-Enhanced OVS Quantum Plugin
                                                                  • DMAC matching for
                                                                    internal traffic to VMs
•    Deploy QoS for      • DMAC matching for incoming
     outgoing packets      packets                                          •   Anti-IP
                         • TPA matching in ARP query                            spoofing: SRC
                                                                                IP matching for
               Tag=1 (MTU size bug)                                             outgoing
       VM1                                     pair of veth                     packets
                tap1          br-int-eth0                     phy-br-eth0
              10 Mbps

                         br-int                                br-eth0              eth0
              50 Mbps
                                                                             Public Networks

       VM2      tap2         Int-loopback                     phy-loopback
                                              pair of veth
               Tag=2
                         • DMAC matching for
    EIG/Bluehost           internal traffic to VMs                                         16
Operational Issues
• Reboot hosts
    – Problem
        • Circular dependencies between libvirtd and nova-compute
        • OVS bug, allows to add non-existing tap interfaces
    – Solution
        • A simple workaround to restart services in rc.local
• Restart services
    – Problem
        • Quantum recreates taps, resulting in network hiccups
    – Solution
        • Check out existing taps, no touch when unnecessary
EIG/Bluehost                                                        17
Operational Issues
• Monitor Health
    – Problem
        • Hard to track down
    – Solution
        • Adding health check APIs
• XML customization
    – Problem
        • No way to modify XML on the fly
        • Not even allowed for static customization
    – Solution
        • Add more parameters
EIG/Bluehost                                          18
Wrap-up/Conclusions




EIG/Bluehost                         19
Problem vs. Solution
                Problem                                      Solution
               Monitoring/troubleshootin
                                                  Service ping, task_states, etc.
                           g
                        No LVM                           Add LVM driver
  Nova
                 Overloaded scheduler                   Custom scheduler
                  OVS VIF driver issue                       Fix bugs
                       Overloaded                 Read-only Mysql slave server
  Mysql
                      Innodb issue                       Optimized confs
                    Instability issue                     Clustered qpid
  Qpid
                       Future plan                     No broker, ZeroMQ
                        Heavy API                         Optimized API
Quantum
                 Premature OVS plugin        Add features (Anti-IP, No-tag flows, QoS)
               Reboot, restart of services            Simple workarounds
Operations
                          XML                   XML customization (cpu custom)

EIG/Bluehost                                                                        20
For True OpenStack Success
Scalability
   Scalable Messaging System & DB Infra

Networks
  Good Networks Abstraction

          So … We don’t do live demo until …

EIG/Bluehost                                   21
We’re sorry, we haven’t had time to contribute a whole lot YET. But here’s
      some code:

      https://github.com/upoopoo for BH nova
      https://github.com/JunPark for BH Quantum (as in live production)

      Browse the branches, many have to do with features discussed today




EIG/Bluehost                                                                       22

More Related Content

What's hot

Kafka Tutorial: Advanced Producers
Kafka Tutorial: Advanced ProducersKafka Tutorial: Advanced Producers
Kafka Tutorial: Advanced ProducersJean-Paul Azar
 
Zero-Copy Event-Driven Servers with Netty
Zero-Copy Event-Driven Servers with NettyZero-Copy Event-Driven Servers with Netty
Zero-Copy Event-Driven Servers with NettyDaniel Bimschas
 
Integrating PostgreSql with RabbitMQ
Integrating PostgreSql with RabbitMQIntegrating PostgreSql with RabbitMQ
Integrating PostgreSql with RabbitMQGavin Roy
 
HBase: Extreme Makeover
HBase: Extreme MakeoverHBase: Extreme Makeover
HBase: Extreme MakeoverHBaseCon
 
VMworld 2013: Extreme Performance Series: Monster Virtual Machines
VMworld 2013: Extreme Performance Series: Monster Virtual Machines VMworld 2013: Extreme Performance Series: Monster Virtual Machines
VMworld 2013: Extreme Performance Series: Monster Virtual Machines VMworld
 
The State of HBase Replication
The State of HBase ReplicationThe State of HBase Replication
The State of HBase ReplicationHBaseCon
 
HadoopCon- Trend Micro SPN Hadoop Overview
HadoopCon- Trend Micro SPN Hadoop OverviewHadoopCon- Trend Micro SPN Hadoop Overview
HadoopCon- Trend Micro SPN Hadoop OverviewYafang Chang
 
VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...
VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...
VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...VMworld
 
HBase: Where Online Meets Low Latency
HBase: Where Online Meets Low LatencyHBase: Where Online Meets Low Latency
HBase: Where Online Meets Low LatencyHBaseCon
 
Alfresco scalability and performnce
Alfresco   scalability and performnceAlfresco   scalability and performnce
Alfresco scalability and performncePaul Hampton
 
Manage your switches like servers
Manage your switches like serversManage your switches like servers
Manage your switches like serversCumulus Networks
 
Percona XtraDB Cluster SF Meetup
Percona XtraDB Cluster SF MeetupPercona XtraDB Cluster SF Meetup
Percona XtraDB Cluster SF MeetupVadim Tkachenko
 
Improving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberImproving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberYing Zheng
 
Rabbitmq & Kafka Presentation
Rabbitmq & Kafka PresentationRabbitmq & Kafka Presentation
Rabbitmq & Kafka PresentationEmre Gündoğdu
 
12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQLKonstantin Gredeskoul
 
Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network 09...
Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network  09...Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network  09...
Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network 09...Cary Hayward
 
Modern Distributed Messaging and RPC
Modern Distributed Messaging and RPCModern Distributed Messaging and RPC
Modern Distributed Messaging and RPCMax Alexejev
 

What's hot (20)

Kafka Tutorial: Advanced Producers
Kafka Tutorial: Advanced ProducersKafka Tutorial: Advanced Producers
Kafka Tutorial: Advanced Producers
 
Zero-Copy Event-Driven Servers with Netty
Zero-Copy Event-Driven Servers with NettyZero-Copy Event-Driven Servers with Netty
Zero-Copy Event-Driven Servers with Netty
 
Integrating PostgreSql with RabbitMQ
Integrating PostgreSql with RabbitMQIntegrating PostgreSql with RabbitMQ
Integrating PostgreSql with RabbitMQ
 
Fastest Servlets in the West
Fastest Servlets in the WestFastest Servlets in the West
Fastest Servlets in the West
 
HBase: Extreme Makeover
HBase: Extreme MakeoverHBase: Extreme Makeover
HBase: Extreme Makeover
 
VMworld 2013: Extreme Performance Series: Monster Virtual Machines
VMworld 2013: Extreme Performance Series: Monster Virtual Machines VMworld 2013: Extreme Performance Series: Monster Virtual Machines
VMworld 2013: Extreme Performance Series: Monster Virtual Machines
 
The State of HBase Replication
The State of HBase ReplicationThe State of HBase Replication
The State of HBase Replication
 
HadoopCon- Trend Micro SPN Hadoop Overview
HadoopCon- Trend Micro SPN Hadoop OverviewHadoopCon- Trend Micro SPN Hadoop Overview
HadoopCon- Trend Micro SPN Hadoop Overview
 
VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...
VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...
VMworld 2013: Big Data: Virtualized SAP HANA Performance, Scalability and Bes...
 
Kafka: Internals
Kafka: InternalsKafka: Internals
Kafka: Internals
 
HBase: Where Online Meets Low Latency
HBase: Where Online Meets Low LatencyHBase: Where Online Meets Low Latency
HBase: Where Online Meets Low Latency
 
Alfresco scalability and performnce
Alfresco   scalability and performnceAlfresco   scalability and performnce
Alfresco scalability and performnce
 
Manage your switches like servers
Manage your switches like serversManage your switches like servers
Manage your switches like servers
 
Percona XtraDB Cluster SF Meetup
Percona XtraDB Cluster SF MeetupPercona XtraDB Cluster SF Meetup
Percona XtraDB Cluster SF Meetup
 
Improving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberImproving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at Uber
 
Rabbitmq & Kafka Presentation
Rabbitmq & Kafka PresentationRabbitmq & Kafka Presentation
Rabbitmq & Kafka Presentation
 
Alfresco tuning part2
Alfresco tuning part2Alfresco tuning part2
Alfresco tuning part2
 
12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL
 
Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network 09...
Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network  09...Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network  09...
Power of Open SDN- The Vendor Neutral Approach to Optimizing Your Network 09...
 
Modern Distributed Messaging and RPC
Modern Distributed Messaging and RPCModern Distributed Messaging and RPC
Modern Distributed Messaging and RPC
 

Similar to Blue host using openstack in a traditional hosting environment

Using OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting EnvironmentUsing OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting EnvironmentOpenStack Foundation
 
Managing Open vSwitch Across a Large Heterogenous Fleet
Managing Open vSwitch Across a Large Heterogenous FleetManaging Open vSwitch Across a Large Heterogenous Fleet
Managing Open vSwitch Across a Large Heterogenous Fleetandyhky
 
IPVS for Docker Containers
IPVS for Docker ContainersIPVS for Docker Containers
IPVS for Docker ContainersBob Sokol
 
[En] IPVS for Docker Containers
[En] IPVS for Docker Containers[En] IPVS for Docker Containers
[En] IPVS for Docker ContainersAndrey Sibirev
 
Integrating OpenStack to Existing infrastructure
Integrating OpenStack to Existing infrastructureIntegrating OpenStack to Existing infrastructure
Integrating OpenStack to Existing infrastructurelaurabeckcahoon
 
Network virtualization with open stack quantum
Network virtualization with open stack quantumNetwork virtualization with open stack quantum
Network virtualization with open stack quantumMiguel Lavalle
 
Implementing an IPv6 Enabled Environment for a Public Cloud Tenant
Implementing an IPv6 Enabled Environment for a Public Cloud TenantImplementing an IPv6 Enabled Environment for a Public Cloud Tenant
Implementing an IPv6 Enabled Environment for a Public Cloud TenantShixiong Shang
 
Open stack journey from folsom to grizzly
Open stack journey from folsom to grizzlyOpen stack journey from folsom to grizzly
Open stack journey from folsom to grizzlyopenstackindia
 
Understanding and deploying Network Virtualization
Understanding and deploying Network VirtualizationUnderstanding and deploying Network Virtualization
Understanding and deploying Network VirtualizationSDN Hub
 
Boyan Krosnov - Building a software-defined cloud - our experience
Boyan Krosnov - Building a software-defined cloud - our experienceBoyan Krosnov - Building a software-defined cloud - our experience
Boyan Krosnov - Building a software-defined cloud - our experienceShapeBlue
 
Meetup open stack_grizzly
Meetup open stack_grizzlyMeetup open stack_grizzly
Meetup open stack_grizzlyeNovance
 
Openstack HA
Openstack HAOpenstack HA
Openstack HAYong Luo
 
Openstack Quantum yahoo meetup 1 23-13
Openstack Quantum yahoo meetup 1 23-13Openstack Quantum yahoo meetup 1 23-13
Openstack Quantum yahoo meetup 1 23-13Dan Wendlandt
 
Capacity Management/Provisioning (Cloud's full, Can't build here)
Capacity Management/Provisioning (Cloud's full, Can't build here)Capacity Management/Provisioning (Cloud's full, Can't build here)
Capacity Management/Provisioning (Cloud's full, Can't build here)andyhky
 
Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...Fwdays
 
Open stack ha design & deployment kilo
Open stack ha design & deployment   kiloOpen stack ha design & deployment   kilo
Open stack ha design & deployment kiloSteven Li
 
Integrating OpenStack To Existing Infrastructure
Integrating OpenStack To Existing InfrastructureIntegrating OpenStack To Existing Infrastructure
Integrating OpenStack To Existing InfrastructureHui Cheng
 
OpenStack Networking and Automation
OpenStack Networking and AutomationOpenStack Networking and Automation
OpenStack Networking and AutomationAdam Johnson
 

Similar to Blue host using openstack in a traditional hosting environment (20)

Using OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting EnvironmentUsing OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting Environment
 
Neutron scaling
Neutron scalingNeutron scaling
Neutron scaling
 
Managing Open vSwitch Across a Large Heterogenous Fleet
Managing Open vSwitch Across a Large Heterogenous FleetManaging Open vSwitch Across a Large Heterogenous Fleet
Managing Open vSwitch Across a Large Heterogenous Fleet
 
IPVS for Docker Containers
IPVS for Docker ContainersIPVS for Docker Containers
IPVS for Docker Containers
 
[En] IPVS for Docker Containers
[En] IPVS for Docker Containers[En] IPVS for Docker Containers
[En] IPVS for Docker Containers
 
Integrating OpenStack to Existing infrastructure
Integrating OpenStack to Existing infrastructureIntegrating OpenStack to Existing infrastructure
Integrating OpenStack to Existing infrastructure
 
Network virtualization with open stack quantum
Network virtualization with open stack quantumNetwork virtualization with open stack quantum
Network virtualization with open stack quantum
 
Implementing an IPv6 Enabled Environment for a Public Cloud Tenant
Implementing an IPv6 Enabled Environment for a Public Cloud TenantImplementing an IPv6 Enabled Environment for a Public Cloud Tenant
Implementing an IPv6 Enabled Environment for a Public Cloud Tenant
 
Open stack journey from folsom to grizzly
Open stack journey from folsom to grizzlyOpen stack journey from folsom to grizzly
Open stack journey from folsom to grizzly
 
Understanding and deploying Network Virtualization
Understanding and deploying Network VirtualizationUnderstanding and deploying Network Virtualization
Understanding and deploying Network Virtualization
 
Boyan Krosnov - Building a software-defined cloud - our experience
Boyan Krosnov - Building a software-defined cloud - our experienceBoyan Krosnov - Building a software-defined cloud - our experience
Boyan Krosnov - Building a software-defined cloud - our experience
 
Meetup open stack_grizzly
Meetup open stack_grizzlyMeetup open stack_grizzly
Meetup open stack_grizzly
 
Openstack HA
Openstack HAOpenstack HA
Openstack HA
 
Openstack Quantum yahoo meetup 1 23-13
Openstack Quantum yahoo meetup 1 23-13Openstack Quantum yahoo meetup 1 23-13
Openstack Quantum yahoo meetup 1 23-13
 
Capacity Management/Provisioning (Cloud's full, Can't build here)
Capacity Management/Provisioning (Cloud's full, Can't build here)Capacity Management/Provisioning (Cloud's full, Can't build here)
Capacity Management/Provisioning (Cloud's full, Can't build here)
 
Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...
 
Open stack ha design & deployment kilo
Open stack ha design & deployment   kiloOpen stack ha design & deployment   kilo
Open stack ha design & deployment kilo
 
Integrating OpenStack To Existing Infrastructure
Integrating OpenStack To Existing InfrastructureIntegrating OpenStack To Existing Infrastructure
Integrating OpenStack To Existing Infrastructure
 
CloudStack and SDN
CloudStack and SDNCloudStack and SDN
CloudStack and SDN
 
OpenStack Networking and Automation
OpenStack Networking and AutomationOpenStack Networking and Automation
OpenStack Networking and Automation
 

More from OpenStack Foundation

Sponsor Webinar - OpenStack Summit Vancouver 2018
Sponsor Webinar  - OpenStack Summit Vancouver 2018Sponsor Webinar  - OpenStack Summit Vancouver 2018
Sponsor Webinar - OpenStack Summit Vancouver 2018OpenStack Foundation
 
OpenStack Summits 101: A Guide For Attendees
OpenStack Summits 101: A Guide For AttendeesOpenStack Summits 101: A Guide For Attendees
OpenStack Summits 101: A Guide For AttendeesOpenStack Foundation
 
OpenStack Marketing Plan - Community Presentation
OpenStack Marketing Plan - Community PresentationOpenStack Marketing Plan - Community Presentation
OpenStack Marketing Plan - Community PresentationOpenStack Foundation
 
OpenStack 5th Birthday - User Group Parties
OpenStack 5th Birthday - User Group PartiesOpenStack 5th Birthday - User Group Parties
OpenStack 5th Birthday - User Group PartiesOpenStack Foundation
 
Liberty release: Preliminary marketing materials & messages
Liberty release: Preliminary marketing materials & messagesLiberty release: Preliminary marketing materials & messages
Liberty release: Preliminary marketing materials & messagesOpenStack Foundation
 
OpenStack Foundation 2H 2015 Marketing Plan
OpenStack Foundation 2H 2015 Marketing PlanOpenStack Foundation 2H 2015 Marketing Plan
OpenStack Foundation 2H 2015 Marketing PlanOpenStack Foundation
 
OpenStack Summit Tokyo Sponsor Webinar
OpenStack Summit Tokyo Sponsor Webinar OpenStack Summit Tokyo Sponsor Webinar
OpenStack Summit Tokyo Sponsor Webinar OpenStack Foundation
 
Neutron Updates - Liberty Edition
Neutron Updates - Liberty Edition Neutron Updates - Liberty Edition
Neutron Updates - Liberty Edition OpenStack Foundation
 
Searchlight Updates - Liberty Edition
Searchlight Updates - Liberty EditionSearchlight Updates - Liberty Edition
Searchlight Updates - Liberty EditionOpenStack Foundation
 
Congress Updates - Liberty Edition
Congress Updates - Liberty EditionCongress Updates - Liberty Edition
Congress Updates - Liberty EditionOpenStack Foundation
 
Release Cycle Management Updates - Liberty Edition
Release Cycle Management Updates - Liberty EditionRelease Cycle Management Updates - Liberty Edition
Release Cycle Management Updates - Liberty EditionOpenStack Foundation
 
OpenStack Day CEE 2015: Real-World Use Cases
OpenStack Day CEE 2015: Real-World Use CasesOpenStack Day CEE 2015: Real-World Use Cases
OpenStack Day CEE 2015: Real-World Use CasesOpenStack Foundation
 

More from OpenStack Foundation (20)

Sponsor Webinar - OpenStack Summit Vancouver 2018
Sponsor Webinar  - OpenStack Summit Vancouver 2018Sponsor Webinar  - OpenStack Summit Vancouver 2018
Sponsor Webinar - OpenStack Summit Vancouver 2018
 
OpenStack Summits 101: A Guide For Attendees
OpenStack Summits 101: A Guide For AttendeesOpenStack Summits 101: A Guide For Attendees
OpenStack Summits 101: A Guide For Attendees
 
OpenStack Marketing Plan - Community Presentation
OpenStack Marketing Plan - Community PresentationOpenStack Marketing Plan - Community Presentation
OpenStack Marketing Plan - Community Presentation
 
OpenStack 5th Birthday - User Group Parties
OpenStack 5th Birthday - User Group PartiesOpenStack 5th Birthday - User Group Parties
OpenStack 5th Birthday - User Group Parties
 
Liberty release: Preliminary marketing materials & messages
Liberty release: Preliminary marketing materials & messagesLiberty release: Preliminary marketing materials & messages
Liberty release: Preliminary marketing materials & messages
 
OpenStack Foundation 2H 2015 Marketing Plan
OpenStack Foundation 2H 2015 Marketing PlanOpenStack Foundation 2H 2015 Marketing Plan
OpenStack Foundation 2H 2015 Marketing Plan
 
OpenStack Summit Tokyo Sponsor Webinar
OpenStack Summit Tokyo Sponsor Webinar OpenStack Summit Tokyo Sponsor Webinar
OpenStack Summit Tokyo Sponsor Webinar
 
Cinder Updates - Liberty Edition
Cinder Updates - Liberty Edition Cinder Updates - Liberty Edition
Cinder Updates - Liberty Edition
 
Glance Updates - Liberty Edition
Glance Updates - Liberty EditionGlance Updates - Liberty Edition
Glance Updates - Liberty Edition
 
Heat Updates - Liberty Edition
Heat Updates - Liberty EditionHeat Updates - Liberty Edition
Heat Updates - Liberty Edition
 
Neutron Updates - Liberty Edition
Neutron Updates - Liberty Edition Neutron Updates - Liberty Edition
Neutron Updates - Liberty Edition
 
Nova Updates - Liberty Edition
Nova Updates - Liberty EditionNova Updates - Liberty Edition
Nova Updates - Liberty Edition
 
Sahara Updates - Liberty Edition
Sahara Updates - Liberty EditionSahara Updates - Liberty Edition
Sahara Updates - Liberty Edition
 
Searchlight Updates - Liberty Edition
Searchlight Updates - Liberty EditionSearchlight Updates - Liberty Edition
Searchlight Updates - Liberty Edition
 
Trove Updates - Liberty Edition
Trove Updates - Liberty EditionTrove Updates - Liberty Edition
Trove Updates - Liberty Edition
 
OpenStack: five years in
OpenStack: five years inOpenStack: five years in
OpenStack: five years in
 
Swift Updates - Liberty Edition
Swift Updates - Liberty EditionSwift Updates - Liberty Edition
Swift Updates - Liberty Edition
 
Congress Updates - Liberty Edition
Congress Updates - Liberty EditionCongress Updates - Liberty Edition
Congress Updates - Liberty Edition
 
Release Cycle Management Updates - Liberty Edition
Release Cycle Management Updates - Liberty EditionRelease Cycle Management Updates - Liberty Edition
Release Cycle Management Updates - Liberty Edition
 
OpenStack Day CEE 2015: Real-World Use Cases
OpenStack Day CEE 2015: Real-World Use CasesOpenStack Day CEE 2015: Real-World Use Cases
OpenStack Day CEE 2015: Real-World Use Cases
 

Recently uploaded

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 

Recently uploaded (20)

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 

Blue host using openstack in a traditional hosting environment

  • 1. Using OpenStack In A Traditional Hosting Environment Jun Park (Ph.D.), Sr. Systems Architect Mike Wilson, Sr. Systems Architect EIG/Bluehost OpenStack Summit 2013
  • 2. Scale @10x what we were in about a year’s time • Needed a system to manage it all • Provisioning these systems had to be automated • System had to scale to multiple data centers • Btw, we have 2 months to come up with this EIG/Bluehost 2
  • 3. High level requirements • Centralized management, horizontal scalability. • Abstractions for physical and logical deployments of devices and resources. • Open-source project with lots of support and momentum • Typical Cloud features (migration, imaging, provisioning, etc.) Easy transition into future cloud offering with support for industry standard APIs = OpenStack! EIG/Bluehost 3
  • 4. Bluehost Environment • Scale – More than 10,000 physical servers – Adding hundreds of nodes per day • Network – Public network directly attached to VMs – Plan on adding private network later • OpenStack/Folsom Components – Nova (compute, api, scheduler) – Quantum (network) – AMQP: Qpid (messaging) – Mysql (db) EIG/Bluehost 4
  • 5. Outline • Scalability/Stability • Rethink Quantum Network • Operational Issues • Wrap-up/Conclusions EIG/Bluehost 5
  • 6. Why Difficult To Scale up? • Components that don’t scale – Messaging system Look at that line! – Mysql server – Heavy APIs • Hard to Diagnose – No simulator/emulator for high scalability testing – No quantitative guide as to how to scale up – No detailed error message or even not at all – Requires detailed knowledge of codebase EIG/Bluehost 6
  • 7. Nova Enhancements • Monitoring/troubleshooting – Added service ping (similar to grizzly) – Additional task_states – Better errors to instance_faults • Functionality – Added LVM resize and injection – Added stop_soft and stop_hard similar to reboot • Stability/Scalability – Some bug fixes to OVS VIF driver – Added custom scheduler EIG/Bluehost 7
  • 8. MySQL/Innodb Concurrency • Nova behavior – Direct connection to DB (addressed in grizzly though) – Too MANY queries; much more read than write – Some queries often return huge number of results – Periodicity of loads due to periodic tasks • Mysql behavior – innodb_thread_concurrency = num of threads that can execute queries in the queue – innodb_concurrency_tickets = number of rows you can return before requeue (default 500!) – With default settings your mysql server will fall over with a few thousand compute nodes EIG/Bluehost 8
  • 9. So the answer is just increase concurrency and tickets right? Sadly, no. Tuning concurrency and tickets is a must! But we still requeue. We don’t know why, but we have a workaround. Our workaround: Send most read queries to a cluster of mysql slaves. I say most because many queries would be sensitive to possible slave replication lag EIG/Bluehost 9
  • 10. Messaging System • Qpid Chosen – Speed and easy cluster ability – Incredibly unstable at large scale (Even at small scale) – Possibly poorly configured – Older release in CentOS6 • Problem – Broker bottleneck – Unnecessarily complex • Recommendation – AVOID BROKER! – ZeroMQ as replacement EIG/Bluehost 10
  • 11. BlueHost OpenStack Group of Controllers mysql master • Nova-api Read-only mysql slave Read-only mysql slave • Nova-scheduler Read-only mysql slaves • Quantum-server • Keystone Load Balanced Server Farm > 10,000 physical servers Host BH-enhanced OVS Qpid servers Quantum Plugin BH-enhanced Nova- EIG/Bluehost compute
  • 12. Rethink Quantum Network • Problem – Quantum API only for DB; no efficient API for Nova – No API-around design for actual network objects – Premature OpenvSwitch (OVS) quantum plugin • Our approach – Adding intelligence to OVS plugin – Optimizing API • E.g., get_instance_nw_info() in nova; Python-level join operation – Expand current implicit communication with nova- compute (although not ideal) EIG/Bluehost 12
  • 13. OpenvSwitch (OVS) • Support – OpenFlow 1.0 (1.2, 1.3 in experiment) – Various Hypervisors including KVM – Merged into main stream since Linux 3.3 • Functionality – Filtering rules and associated actions • E.g., anti-IP spoofing, DMAC filtering – Replacement or superset of ebtables/iptables/linuxbridge – QoS or Linux traffic control (TC) – Various third party controllers for full-fledged OF functionality EIG/Bluehost 13
  • 14. Quantum With Nova-Compute Implicit communication with nova-compute 1. Nova boot API Why nova-compute Controller still create tap? Nova-api server Host 2. Call quantum-api to create port Nova-compute Quantum-server 4. Create OVS tap 6. Do the rest of steps 5. Set up external-ids on tap for creating a new VM 3. Create port in DB 1. Monitor any change of external- ids of the existing taps Quantum 2. Call quantum-api to get port info Quantum-agent DB 3. Modify external-ids on tap & setup OVS flows EIG/Bluehost 14
  • 15. BH-Enhanced OVS Quantum Plugin • Idiosyncratic Requirements – Focus on a Zone; plan on addressing multiple-datacenters issue – Direct public IP using flat shared provider network with No NAT – Still required to be isolated while allowing intra-traffic among VMs • Our Approach – Developing “Distributed OpenFlow controller” using OVS plugin – Either no-tag or 24bit ID (e.g., QinQ, VxLAN, GRE), but NO VLAN – Caveats: Neither API-around approach nor Virtual Appliance • New Features – Anti-IP/ARP spoofing OF flows – Multiple-IPs per public port – Network bandwidth throttling (QoS) – Optimal Intra-traffic flows EIG/Bluehost 15
  • 16. BH-Enhanced OVS Quantum Plugin • DMAC matching for internal traffic to VMs • Deploy QoS for • DMAC matching for incoming outgoing packets packets • Anti-IP • TPA matching in ARP query spoofing: SRC IP matching for Tag=1 (MTU size bug) outgoing VM1 pair of veth packets tap1 br-int-eth0 phy-br-eth0 10 Mbps br-int br-eth0 eth0 50 Mbps Public Networks VM2 tap2 Int-loopback phy-loopback pair of veth Tag=2 • DMAC matching for EIG/Bluehost internal traffic to VMs 16
  • 17. Operational Issues • Reboot hosts – Problem • Circular dependencies between libvirtd and nova-compute • OVS bug, allows to add non-existing tap interfaces – Solution • A simple workaround to restart services in rc.local • Restart services – Problem • Quantum recreates taps, resulting in network hiccups – Solution • Check out existing taps, no touch when unnecessary EIG/Bluehost 17
  • 18. Operational Issues • Monitor Health – Problem • Hard to track down – Solution • Adding health check APIs • XML customization – Problem • No way to modify XML on the fly • Not even allowed for static customization – Solution • Add more parameters EIG/Bluehost 18
  • 20. Problem vs. Solution Problem Solution Monitoring/troubleshootin Service ping, task_states, etc. g No LVM Add LVM driver Nova Overloaded scheduler Custom scheduler OVS VIF driver issue Fix bugs Overloaded Read-only Mysql slave server Mysql Innodb issue Optimized confs Instability issue Clustered qpid Qpid Future plan No broker, ZeroMQ Heavy API Optimized API Quantum Premature OVS plugin Add features (Anti-IP, No-tag flows, QoS) Reboot, restart of services Simple workarounds Operations XML XML customization (cpu custom) EIG/Bluehost 20
  • 21. For True OpenStack Success Scalability  Scalable Messaging System & DB Infra Networks  Good Networks Abstraction So … We don’t do live demo until … EIG/Bluehost 21
  • 22. We’re sorry, we haven’t had time to contribute a whole lot YET. But here’s some code: https://github.com/upoopoo for BH nova https://github.com/JunPark for BH Quantum (as in live production) Browse the branches, many have to do with features discussed today EIG/Bluehost 22

Editor's Notes

  1. Love!30mins talk + 10 minsqna during the main talk we have short answers, have longer answers in qna. May have off-line chats for more details.Take a quick poll: How many people with openstack and cloudstack?Time: Tell our story… main tasks remaining …
  2. Quick poll, how many nodes?
  3. Quick poll : how many nodes?
  4. LOVE!!!Just a small step for us to try to help community and share what we’ve done.
  5. Quick poll: how many people know about OVS plugin? How many people use OVS in production?
  6. The main motivation of quantum network is to pull out network functionality from nova-compute as much as possible. Because of this, nova-compute still needs to know all the details of OVS. No code reuse, confusing, and causing inconsistency, succinct failures when VMs do not have network connectivity. With some garbled taps, quantum-agent thinks it needs to do something even for them.
  7. Let me introduce our BH-enhanced OVS plugin. NO NAT, WHY? Avoid single point of failure, easy to diagnose, meet some specific software requirement such as cPanel.
  8. Vlan tagging No tag or double-tag or VxLAN whatever…Avoiding O(n^2) solution….
  9. Let’s get this straightened out!Structurally means most distributed structure among all other cloud platforms.