Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

PyCon Russia 2014 - Auto Scale in the Cloud

1.404 Aufrufe

Veröffentlicht am

Horizontal scaling in the Cloud is the way to adapt resources to load of systems. The Cloud allows users to scale virtually indefinitely, or enough for their needs.
This way the number of servers follows trend of requests, and TCO (Total Cost of Owneship) of IT infrastructure can be reduced. Nonetheless companies can avoid dealing with capacity planning and pre-provisioning issues.

This talk will show how to use Python and Rackspace/OpenStack API and SDK to implement an event-based scaling solution (software released under the open-source Apache License: stay tuned).

Veröffentlicht in: Software

PyCon Russia 2014 - Auto Scale in the Cloud

  1. 1. An introduction Scale in the Cloud Created by: Simone Soldateschi Modified Date: 2014-06-02 Classification: Public Conference
  2. 2. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Who am I? Simone Soldateschi • Java, C/C++, PHP, Python developer • More than 8 years experience as SysAdm/SysEng • Developer Support Engineer at Rackspace • Task automation enthusiast • MTB’ing, triathlon, photo, manga @soldasimo simonesoldateschi 2
  3. 3. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Who are Rackspace? Founded in 1998 in San Antonio, TX by three guys that wanted to create a hosting company Home of Fanatical Support /o/ Second biggest Public Cloud in the world OpenStack Project co-founder
  4. 4. RACKSPACE® HOSTING | WWW.RACKSPACE.COM To be recognized as one of the world’s great service companies. “ ” Rackspace Vision 4
  5. 5. RACKSPACE® HOSTING | WWW.RACKSPACE.COM • Python SDK, Cloud • Auto Scaling • Management System • Control law • Garçon, all together now! Roadmap 5
  6. 6. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Install SDK $ mkvirtualenv pyconru New python executable in pyconru/bin/python Installing setuptools, pip...done. 6 (pyconru)$ pip install pyrax ipython Downloading/unpacking pyrax Downloading pyrax-1.8.1-py2.py3-none-any.whl (316kB): 316kB downloaded Downloading/unpacking ipython Downloading ipython-2.1.0-py2-none-any.whl (2.8MB): 2.8MB downloaded … Successfully installed pyrax ipython … Cleaning up… See: https://github.com/rackspace/pyrax
  7. 7. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Authentication 7 # authenticate pyrax.set_setting('identity_type', 'rackspace') pyrax.set_credentials(os.getenv('OS_AUTH_USER'), os.getenv('OS_AUTH_APIKEY'), region=os.getenv('OS_AUTH_REGION')) print “authenticated: %s” % pyrax.identity.authenticated
  8. 8. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Authentication Check credentials: (pyconru)$ python -m pyconru.basic (DEBUG) OS_AUTH_USER: foo (DEBUG) OS_AUTH_APIKEY: **** (WARNING) OS_AUTH_REGION undefined, using default 'LON' (DEBUG) authenticated: True (INFO) identity token: cfe6d60f070947bf**************** 8 Define environment variables: (pyconru)$ export OS_AUTH_USER=foo (pyconru)$ export OS_AUTH_KEY=bar (pyconru)$ export OS_AUTH_REGION=LON
  9. 9. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Cloud components 9
  10. 10. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Cloud components 10
  11. 11. RACKSPACE® HOSTING | WWW.RACKSPACE.COM • Python SDK, Cloud • Auto Scaling • Management System • Control law • Garçon, all together now! Roadmap 11
  12. 12. simone.soldateschi@rackspace.co.uk Vertical scaling 2 GB 2 CORES 8 GB 8 CORES
  13. 13. RACKSPACE® HOSTING | WWW.RACKSPACE.COM What is Autoscaling?
  14. 14. RACKSPACE® HOSTING | WWW.RACKSPACE.COM What is Autoscaling? WASTED $$$
  15. 15. RACKSPACE® HOSTING | WWW.RACKSPACE.COM What is Autoscaling?
  16. 16. simone.soldateschi@rackspace.co.uk New Usage Models CLOUDSMART 16 Dedicated Servers are Pets • Great thought to their acquisition • Name them and know each one • Willing to pay big money for their care Cloud Servers are Livestock • Use them as long as they provide value • Acquire more of them when needed • Dispose of any that aren’t needed • Get rid of them if they become ill
  17. 17. RACKSPACE® HOSTING | WWW.RACKSPACE.COM http://www.flickr.com/photos/fischerfotos/7419253200/
  18. 18. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Traffic Patterns
  19. 19. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Traffic Patterns ON & OFF Analytics Banks/Tax Agencies Test environments
  20. 20. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Traffic Patterns FAST GROWTH Events Business Growth Slashdot Effect
  21. 21. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Traffic Patterns VARIABLE News & Media Event Registrations Rapid fire sales
  22. 22. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Traffic Patterns CONSISTENT HR Application Accounting/Finance E-mail
  23. 23. RACKSPACE® HOSTING | WWW.RACKSPACE.COM http://www.flickr.com/photos/maximalideal/3356408693/
  24. 24. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Autoscaling Methodologies Time Based Reactive Predictive
  25. 25. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Time Based Autoscaling Load Balancer Server Server
  26. 26. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Time Based Autoscaling Load Balancer Server Server 9:00am
  27. 27. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Time Based Autoscaling Load Balancer Server Server Nov 1st
  28. 28. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Time Based Autoscaling Load Balancer Server Server Server
  29. 29. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Time Based Autoscaling GOOD FOR On & Off Consistent
  30. 30. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Reactive Autoscaling Load Balancer Server 60% Server 60%
  31. 31. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Reactive Autoscaling Load Balancer Server 80% Server 80%
  32. 32. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Reactive Autoscaling Load Balancer Server 60% Server 60% Server 40%
  33. 33. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Reactive Autoscaling Load Balancer Server 30% Server 30% Server 30%
  34. 34. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Reactive Autoscaling Load Balancer Server 45% Server 45%
  35. 35. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Reactive Autoscaling GOOD FOR Fast Growth Variable
  36. 36. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Predictive Autoscaling Load Balancer Server Server
  37. 37. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Predictive Autoscaling Load Balancer Server Server Forecasted Traffic +30%
  38. 38. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Predictive Autoscaling Load Balancer Server Server Server
  39. 39. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Predictive Autoscaling GOOD FOR Fast Growth Variable
  40. 40. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto scaling - Schedule-based scaling
  41. 41. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto scaling - Schedule-based scaling
  42. 42. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto scaling - Event-based scaling
  43. 43. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto scaling - Event-based scaling SCALE UP
  44. 44. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto scaling - Event-based scaling COOL DOWN
  45. 45. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 45 Cooldown
  46. 46. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto scaling - Event-based scaling COOL DOWN
  47. 47. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto scaling - Event-based scaling SCALE DOWN
  48. 48. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto Scale – Use case FRONT END
  49. 49. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto Scale – Use case LB FRONT END Share nothing Stateless nodes
  50. 50. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto Scale – Use case LB FRONT END LB API BOSS WORKER
  51. 51. RACKSPACE® HOSTING | WWW.RACKSPACE.COM http://www.flickr.com/photos/samuraislice/3309481048/
  52. 52. RACKSPACE® HOSTING | WWW.RACKSPACE.COM • Python SDK, Cloud • Auto Scaling • Management System • Control law • Garçon, all together now! Roadmap 52
  53. 53. RACKSPACE® HOSTING | WWW.RACKSPACE.COM The basics That’s it! $ pip install ansible Installation on management host
  54. 54. RACKSPACE® HOSTING | WWW.RACKSPACE.COM The basics Install agent on managed hosts:
  55. 55. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Why use ansible? Desired state Go live!
  56. 56. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Desired State Write code to tell the computer how to set up itself! 56RACKSPACE® HOSTING | WWW.RACKSPACE.COM
  57. 57. RACKSPACE® HOSTING | WWW.RACKSPACE.COM • Python SDK, Cloud • Auto Scaling • Management System • Control law • Garçon, all together now! Roadmap 57
  58. 58. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Closed-Loop Control Law 58
  59. 59. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Event-based Auto Scale
  60. 60. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Auto Scale
  61. 61. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Closed-loop Control Law – Garçon implementation 61 ?
  62. 62. RACKSPACE® HOSTING | WWW.RACKSPACE.COM • Python SDK, Cloud • Auto Scaling • Management System • Control law • Garçon, all together now! Roadmap 62
  63. 63. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon How to integrate Cloud Monitoring and Auto Scale ? 63
  64. 64. simone.soldateschi@rackspace.co.uk Garçon - How? Garçon
  65. 65. simone.soldateschi@rackspace.co.uk Garçon - Overview Garçon cm2asd cfgmgmtd
  66. 66. simone.soldateschi@rackspace.co.uk Garçon - Go Live! cfgmgmtd Go Live!
  67. 67. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Closed-Loop Control Law – Garçon implementation 67 Garçon
  68. 68. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cm2asd 68 # fetch current list of servers l_current_servers = scaling_group_servers(scaling_group_id) Garçon
  69. 69. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cm2asd 69 for i in range(len(l_current_servers)-1, -1, -1): server_id = l_current_servers[i] s = get_server(server_id) Garçon if s.status != 'ACTIVE': # server not active l_current_servers.pop(i) continue m = get_server_metadata(s.id) try: if (m['aspoc.server_status'] != 'configured'): server not configured l_current_servers.pop(i) continue except KeyError: # server not configured l_current_servers.pop(i) continue
  70. 70. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cm2asd 70 Garçon
  71. 71. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cm2asd 71 # compute average system load for scaling group servers_avg_load = servers_average_load(l_checks, samples, sample_time) # compare current load against configured threshold if servers_avg_load >= threshold_high: # trigger scale_up_webhook r = requests.post(scale_up_webhook) if r.status_code != 202: logger.error('scale_up_webhook (%s) returned HTTP %d' % (scale_up_webhook, r.status_code)) Garçon
  72. 72. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cm2asd 72 if servers_avg_load <= threshold_low: # trigger scale_down_webhook r = requests.post(scale_down_webhook) if r.status_code != 202: logger.error('scale_down_webhook (%s) returned HTTP %d' % (scale_down_webhook, r.status_code)) Garçon
  73. 73. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cfgmgmtd 73 for s_id in l_current_servers: ... # server exists? try: cs = pyrax.cloudservers cs.servers.get(s_id) except: logging.warning('Auto Scale server (%s, %s) missing ' '(maybe deleted manually?)' % ('-', s_id)) continue ... try: # read server metadata m = get_server_metadata(s_id) ...
  74. 74. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cfgmgmtd 74 Use metadata try: if (server_status != 'configured' and server_status != 'configuring'): ... # run thread to configure server threading.Thread(target=configure_server, args=(s_id,)).start() No metadata? except KeyError: # CONFIGURE server (KeyError, no metadata) in thread threading.Thread(target=configure_server, args=(s_id, ansible_timeout,)).start() ? X
  75. 75. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cfgmgmtd, Ansible 75 Reset server’s password: # set server password password = generate_password(10, punctuation=False) set_server_password(server_id, password) Server’s info (e.g. IP address): # fetch server info ip = get_server_ipv4(server_id, MGMT_NETWORK)
  76. 76. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cfgmgmtd, Ansible 76 Leverage Configuration Management System: for playbook in list_ansible_playbooks(): ... cmd = ['ansible-playbook', playbook_filename, # '-vvvv', '-c', 'paramiko', '-i', inventory_file] errcode = run_cmd(cmd, logfilename=playbook_logfilename, timeout=ansible_timeout) playbook_logfilename = (os.path.join(LOG_DIR, '%s-%s' % (s.name, playbook))) playbook_logfile = open(playbook_logfilename, 'w') playbook_filename = (os.path.join((playbooks_base_dir), '%s/main.yml' % playbook))
  77. 77. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Garçon in-depth – cfgmgmtd, Monitoring System 77 Create checks for new server to be managed # cloud monitoring (agent_id := server_uuid) add_cm_cpu_check(server_id)) # good, set 'aspoc_server.status=configured' in metadata set_server_metadata(server_id, 'aspoc.server_status', 'configured')
  78. 78. RACKSPACE® HOSTING | WWW.RACKSPACE.COM • Python SDK, Cloud • Auto Scaling • Management System • Control law • Garçon, all together now! RECAP 78
  79. 79. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 79 Q&A @soldasimo simonesoldateschi
  80. 80. RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM RACKSPACE® HOSTING | 5 MILLINGTON ROAD | HAYES, UNITED KINGDOM UB3 4AZ UK SALES: +44 (0)20 8712 6507 | UK SUPPORT: 0800 988 0300 | WWW.RACKSPACE.CO.UK @soldasimo simonesoldateschi

×