SlideShare ist ein Scribd-Unternehmen logo
1 von 35
Downloaden Sie, um offline zu lesen
How to Build a Cluster
A High-Level Overview of the Key Issues
Ramsay Key – May 2017
Why should you care?
• Although a lot of computing has been commoditized and delivered via IT
and “the cloud”, it’s still useful to understand infrastructure:
• Building your own “bare-metal” servers for yourself or a customer
• Deploying your hardware into your customer’s data-center
• Deploying your software into your customer’s data-center
• Understanding considerations for non-traditional systems (i.e. vehicles, devices)
• Some components need to built regardless of “public cloud” or private infrastructure
Photo:Google - http://datacenterfrontier.com/google-building-four-story-data-centers/
The ideas and practices for building
this apply even if you have just a
single rack or half-rack
Outline
• Cluster Design
• Racks and Servers
• Networking
• Day-to-Day Operations
Cluster Design
Gathering Requirements
• Operations
• What is the intended operation of the system? Prototype? Production? 24x7? Call-ins?
• Type of system? Query? Analytics? Batch? Streaming?
• Could the system grow?
• Dataflow
• How much data? Ingest volume (bytes/records)? Query load? Timeframe?
• Compute
• Type of computing: CPU heavy? RAM heavy? File I/O heavy? Network I/O heavy?
• If these answers are not clear – try to derive upper and lower bounds on the needs
RAM goes a long way these days!
• All 4B IPv4 addresses
fit in 18GB of RAM
• 256GB of RAM can
hold 16B MD5 hashes
Design Considerations
• Reliability
• How reliable does your system need to be? How many “nines” of availability do you need?
• Failure
• How much redundancy is required? How much do you have? What about the facilities plant?
• Scalability
• Horizontal vs. Vertical. How much load can the system process? Scalability also includes people processes
• Backup
• Do you have a backup plan?
• Application Deployments
• How to deploy, manage, troubleshoot applications?
Helpful Philosophies
• Keep It Simple Stupid
• Very easy to create complex server infrastructure. Strive for simplicity
• Be Homogenous
• Heterogeneity complicates scaling, debugging, and logistics
• Expect Failure
• Components will fail. Disks fail all the time. More computers ➔ More failure probability
• Automate Everything
• If you can reproduce your infrastructure quickly and easily, it is a good sign it is healthy
Pets vs. Cattle
Financial Considerations
• Electricity availability/cost fundamentally dictates scale
• Appropriate accounting/purchasing allows hardware to be depreciated
• Can you “buy” your way out of scalability problems? (i.e. horizontal scaling)
• Capital Expenditure (CapEx) vs. Operational Expenditures (OpEx)
• CapEx = hardware, facilities
• OpEx = labor, support, maintenance
• Trade-off between CapEx and OpEx
• Clusters generally try to minimize OpEx via automation, homogeneity
• However, clusters don’t run and fix themselves – still need labor to support them
Racks and Servers
Rackspace & Power, Space, Cooling
• Datacenters have real physical constraints generally characterized as Power,
Space, and Cooling (PSC)
• Datacenters are laid out in “racks” (a.k.a. cabinets)
• Datacenters have different “Tiers” (1-4) for handling different failure levels
• Racks are standardized around 42 “rack units” height (a.k.a. “U”).
• “Rack servers” commonly come in 1U and 2U dimensions. Width standardized.
• 3U+ generally implied to be more unique hardware
• Prefer rack-servers over “blade centers”
Rack Considerations
• 1U servers are considered “dense”
• Need to pay attention to cooling and cabling
• Can be hard to fit more elaborate components in 1U (GPUs or large hard-drives)
• 2U servers are good all-around chassis
• May lose some density per U
• A good reference for an “all-up” rack is 40 servers and 2 “top-of-rack”
(TOR) switches
Power Distribution Units (PDU) – can run 2 for redundancy
1U servers
2 Top-of-rack (TOR) switches. Red and blue cables are
“bonded” and provide redundancy and performance
HID – Badge swipe access / alarm
Fiber uplinks to datacenter spine
From: https://techbloc.net/archives/970
1 “management” switch for administration. Separate
network for when main network is down. Yellow.
Server Selection
• For purchasing, best to work through a value-added reseller (VAR)
• Can assist with questions, delivery, coordination
• Consider redundant power supplies depending on production level
• Consider redundant network ports depending on production level
• Hot-swappable hard-drives make life easy (almost standard now)
• Make sure NICs will “PXEBoot” (i.e. network boot)
• Consider RAID (Redundant Array of Inexpensive Disks) level. Popular options:
• RAID10 a good mix between redundancy and performance
• JBOD = Just a bunch of disks – let applications manage redundancy
10 2TB hot-swappable hard-drives
Dual hot-swappable power supplies
Dual CPUs, 10 cores
256GB RAM
Fan bank for cooling
(Not viewable) – 4 10G NIC ports, 2 1G NIC port, 1 IPMI port
Typical
commodity server
– circa 2017
Generally don’t
need tools to
replace parts!
Operating Systems
• Linux is the OS of choice when building clusters
• Lots of tools for managing and tuning Linux clusters at scale
• Licensed software complicates scaling a cluster
• These days many excellent open-source alternatives exists
• CentOS (derivative of Redhat) and Ubuntu are both popular options
• CentOS generally about stability and security
• Popular with enterprises, IT, and operations people
• Ubuntu generally newer and modern (closer to latest constituent software releases)
• Popular with innovators, researchers, etc.
Provisioning
• Provisioning = building or rebuilding a node
• Typical flow is for the node to “PXEBoot” into a kickstart (Redhat/Centos)
or preseed (Ubuntu) an installer
• Node first boots via DHCP then downloads a kickstart/preseed file
• The kickstart/preseed file points to an installer and associated packages
• Foreman is a system that facilitates the “PXEBoot”, kickstart/preseed process
• Typical model is to put the bare minimum into kickstart/preseed and then let
a configuration management system take over
Example Kickstart File
Networking
Network Considerations
• Typical server network configuration would have:
• 1Gb IPMI port
• Allows interaction with basic server functions (power-on/power-off, etc.)
• 1Gb management port
• For server administration tasks, separate from data network
• 2 10Gb data ports (in a bonded configuration)
• For passing data between nodes
• Keep in mind that disk I/O speed may be slower than the network
Network Fabric
• Typical rack configuration has two “top-of-rack” (TOR) switches that connect all
the internal rack servers together, plus an “uplink” to the datacenter spine of “core”
switches so it can talk to the other racks
• Use two switches per rack for redundancy and throughput
• Spine typically runs at 25Gb, 40Gb, or 100Gb
http://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-7000-
series-switches/white-paper-c11-737022.docx/_jcr_content/renditions/white-
paper-c11-737022_3.jpg
Racks
Day-to-Day Operations
Configuration Management
• Configuration Management = tools that guarantee server configurations
• Popular cluster configuration management tools:
• Puppet: Most common, well-known, agent architecture
• Chef: An early alternative to Puppet, also agent architecture
• Ansible: Popular, agent-less architecture, integrates with networking gear well
• SaltStack: More recent alternative, both agent and agent-less architecture
• All tools have pros & cons - doesn’t matter so much what you use, just use one!
• Possible to entirely define your infrastructure within the tools
• Version control (git) the tool configurations and you get “infrastructure as code”
Example Puppet Manifest
Monitoring
• Bad things will happen in your cluster!
• The larger, the more complex the cluster, the more fantastic ways it can fail
• Absolutely need monitoring of your cluster
• Nagios is the most common tool for monitoring
• Many alternatives: ElastAlert, Zenoss, Prometheus, xymon, Zabbix, Sensu, …
• Tools send an alert (email, syslog, IM) when bad things happen
• Tools usually come with some defaults – and have pluggable architectures
nagios.com
Health & Status (Metrics)
• “Measure Anything. Measure Everything” – Etsy
• Instrument everything you can:
• Useful for performance tuning
• When bad things happen, these will be handy for identifying the root-cause
• Applications, operating systems, processes, disks, network, memory, etc.
• Ganglia is the most common tool for metrics
• Many excellent alternatives: Grafana, Prometheus, Logstash, Graphite, statsd, collectd, OpenTSDB, Timely
• Pick one and use it!
How to Build a Compute Cluster
Coordination Services
• Often useful to run a “coordination service” within a cluster
• Coordination service provides distributed reliable services for applications:
• Configuration (identifying masters)
• Naming (finding other services)
• Synchronization (tracking state)
• Typically present a “key-value” interface to clients
• Popular implementations: Zookeeper, etcd, Consul, etc.
Other Usefuls Tools
• Public-Key Infrastructure (PKI, ssh) should be used for authentication
• Avoid passwords – only root should have a password, if at all
• Use LDAP to manage user accounts
• Have some “database” that records which servers provide which functions
• genders is a simple, popular way to do this
• pdsh is a parallel shell command useful for running commands across a cluster
• dshbak cleans up the output
How to Build a Compute Cluster
Software Deployment Considerations
• Always a good idea to partition your cluster into:
• Production servers - stuff you really care about…doesn’t need to be 24x7 to be production
• Integration servers - last stop before being added to production
• Test servers - general developer playground similar to production systems
• Package, version, and install your software like a product
• Helps for automation and traceability
• Scripting languages (python, perl, etc.) can be risky to deploy because they can easily be
changed once installed
Future Considerations
• Virtualization
• Openstack, AWS, GCE, Azure, Rackspace
• Containers
• Docker, rkt, CoreOS, Swarm, Kubernetes
• “Serverless” computing
• Open Compute Project
• Software-Defined-Networking (SDN)

Weitere ähnliche Inhalte

Was ist angesagt?

Best Practices of HA and Replication of PostgreSQL in Virtualized Environments
Best Practices of HA and Replication of PostgreSQL in Virtualized EnvironmentsBest Practices of HA and Replication of PostgreSQL in Virtualized Environments
Best Practices of HA and Replication of PostgreSQL in Virtualized EnvironmentsJignesh Shah
 
Presentation architecting a cloud infrastructure
Presentation   architecting a cloud infrastructurePresentation   architecting a cloud infrastructure
Presentation architecting a cloud infrastructurexKinAnx
 
Managing Remote Teams
Managing Remote TeamsManaging Remote Teams
Managing Remote TeamsDell World
 
MySQL Infrastructure Testing Automation at GitHub
MySQL Infrastructure Testing Automation at GitHubMySQL Infrastructure Testing Automation at GitHub
MySQL Infrastructure Testing Automation at GitHubIke Walker
 
SUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigDataSUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigDataJuan Herrera Utande
 
Postgres & Red Hat Cluster Suite
Postgres & Red Hat Cluster SuitePostgres & Red Hat Cluster Suite
Postgres & Red Hat Cluster SuiteEDB
 
VMworld 2013: How SRP Delivers More Than Power to Their Customers
VMworld 2013: How SRP Delivers More Than Power to Their Customers VMworld 2013: How SRP Delivers More Than Power to Their Customers
VMworld 2013: How SRP Delivers More Than Power to Their Customers VMworld
 
My experience with embedding PostgreSQL
 My experience with embedding PostgreSQL My experience with embedding PostgreSQL
My experience with embedding PostgreSQLJignesh Shah
 
End of RAID as we know it with Ceph Replication
End of RAID as we know it with Ceph ReplicationEnd of RAID as we know it with Ceph Replication
End of RAID as we know it with Ceph ReplicationCeph Community
 
PostgreSQL and Benchmarks
PostgreSQL and BenchmarksPostgreSQL and Benchmarks
PostgreSQL and BenchmarksJignesh Shah
 
VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED
VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED
VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED VMworld
 
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best PracticesVMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best PracticesVMworld
 
MT58 High performance graphics for VDI: A technical discussion
MT58 High performance graphics for VDI: A technical discussionMT58 High performance graphics for VDI: A technical discussion
MT58 High performance graphics for VDI: A technical discussionDell EMC World
 
Deploying Maximum HA Architecture With PostgreSQL
Deploying Maximum HA Architecture With PostgreSQLDeploying Maximum HA Architecture With PostgreSQL
Deploying Maximum HA Architecture With PostgreSQLDenish Patel
 
Still All on One Server: Perforce at Scale
Still All on One Server: Perforce at Scale Still All on One Server: Perforce at Scale
Still All on One Server: Perforce at Scale Perforce
 
Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...
Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...
Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...Principled Technologies
 
(ATS4-PLAT06) Considerations for sizing and deployment
(ATS4-PLAT06) Considerations for sizing and deployment(ATS4-PLAT06) Considerations for sizing and deployment
(ATS4-PLAT06) Considerations for sizing and deploymentBIOVIA
 
Lessons from building large clusters
Lessons from building large clustersLessons from building large clusters
Lessons from building large clustersSteve Loughran
 

Was ist angesagt? (19)

Best Practices of HA and Replication of PostgreSQL in Virtualized Environments
Best Practices of HA and Replication of PostgreSQL in Virtualized EnvironmentsBest Practices of HA and Replication of PostgreSQL in Virtualized Environments
Best Practices of HA and Replication of PostgreSQL in Virtualized Environments
 
Presentation architecting a cloud infrastructure
Presentation   architecting a cloud infrastructurePresentation   architecting a cloud infrastructure
Presentation architecting a cloud infrastructure
 
Managing Remote Teams
Managing Remote TeamsManaging Remote Teams
Managing Remote Teams
 
MySQL Infrastructure Testing Automation at GitHub
MySQL Infrastructure Testing Automation at GitHubMySQL Infrastructure Testing Automation at GitHub
MySQL Infrastructure Testing Automation at GitHub
 
SUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigDataSUSE: Infraestructura definida por software para BigData
SUSE: Infraestructura definida por software para BigData
 
Postgres & Red Hat Cluster Suite
Postgres & Red Hat Cluster SuitePostgres & Red Hat Cluster Suite
Postgres & Red Hat Cluster Suite
 
VMworld 2013: How SRP Delivers More Than Power to Their Customers
VMworld 2013: How SRP Delivers More Than Power to Their Customers VMworld 2013: How SRP Delivers More Than Power to Their Customers
VMworld 2013: How SRP Delivers More Than Power to Their Customers
 
My experience with embedding PostgreSQL
 My experience with embedding PostgreSQL My experience with embedding PostgreSQL
My experience with embedding PostgreSQL
 
End of RAID as we know it with Ceph Replication
End of RAID as we know it with Ceph ReplicationEnd of RAID as we know it with Ceph Replication
End of RAID as we know it with Ceph Replication
 
PostgreSQL and Benchmarks
PostgreSQL and BenchmarksPostgreSQL and Benchmarks
PostgreSQL and Benchmarks
 
VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED
VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED
VMworld 2013: VMware Mirage Storage and Network Deduplication, DEMYSTIFIED
 
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best PracticesVMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
VMworld 2014: Advanced SQL Server on vSphere Techniques and Best Practices
 
MT58 High performance graphics for VDI: A technical discussion
MT58 High performance graphics for VDI: A technical discussionMT58 High performance graphics for VDI: A technical discussion
MT58 High performance graphics for VDI: A technical discussion
 
Deploying Maximum HA Architecture With PostgreSQL
Deploying Maximum HA Architecture With PostgreSQLDeploying Maximum HA Architecture With PostgreSQL
Deploying Maximum HA Architecture With PostgreSQL
 
Still All on One Server: Perforce at Scale
Still All on One Server: Perforce at Scale Still All on One Server: Perforce at Scale
Still All on One Server: Perforce at Scale
 
TechTalkThai-CiscoHyperFlex
TechTalkThai-CiscoHyperFlexTechTalkThai-CiscoHyperFlex
TechTalkThai-CiscoHyperFlex
 
Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...
Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...
Simplifying systems management with Dell OpenManage on 13G Dell PowerEdge ser...
 
(ATS4-PLAT06) Considerations for sizing and deployment
(ATS4-PLAT06) Considerations for sizing and deployment(ATS4-PLAT06) Considerations for sizing and deployment
(ATS4-PLAT06) Considerations for sizing and deployment
 
Lessons from building large clusters
Lessons from building large clustersLessons from building large clusters
Lessons from building large clusters
 

Ähnlich wie How to Build a Compute Cluster

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio, Inc.
 
HPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journeyHPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journeyPeter Clapham
 
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Community
 
Survey of Big Data Infrastructures
Survey of Big Data InfrastructuresSurvey of Big Data Infrastructures
Survey of Big Data Infrastructuresm.a.kirn
 
SCALE 16x on-prem container orchestrator deployment
SCALE 16x on-prem container orchestrator deploymentSCALE 16x on-prem container orchestrator deployment
SCALE 16x on-prem container orchestrator deploymentSteve Wong
 
A Closer Look at Apache Kudu
A Closer Look at Apache KuduA Closer Look at Apache Kudu
A Closer Look at Apache KuduAndriy Zabavskyy
 
Scale up is history! is scale out the future for storage
Scale up is history!  is scale out the future for storageScale up is history!  is scale out the future for storage
Scale up is history! is scale out the future for storageStarWind Software
 
start_your_datacenter_sds_v3
start_your_datacenter_sds_v3start_your_datacenter_sds_v3
start_your_datacenter_sds_v3David Byte
 
Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham
 
Data Lake and the rise of the microservices
Data Lake and the rise of the microservicesData Lake and the rise of the microservices
Data Lake and the rise of the microservicesBigstep
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learnJohn D Almon
 
Meta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinarMeta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinarMichael Hiskey
 
Big data and hadoop
Big data and hadoopBig data and hadoop
Big data and hadoopMohit Tare
 
Xldb2011 wed 1415_andrew_lamb-buildingblocks
Xldb2011 wed 1415_andrew_lamb-buildingblocksXldb2011 wed 1415_andrew_lamb-buildingblocks
Xldb2011 wed 1415_andrew_lamb-buildingblocksliqiang xu
 
StarlingX - Project Onboarding
StarlingX - Project OnboardingStarlingX - Project Onboarding
StarlingX - Project OnboardingShuquan Huang
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople
 
Best Practices for Deploying Enterprise Applications on UNIX
Best Practices for Deploying Enterprise Applications on UNIXBest Practices for Deploying Enterprise Applications on UNIX
Best Practices for Deploying Enterprise Applications on UNIXNoel McKeown
 
The Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentThe Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentTimothy Fitz
 
Meta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinarMeta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinarKognitio
 

Ähnlich wie How to Build a Compute Cluster (20)

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
 
HPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journeyHPC and cloud distributed computing, as a journey
HPC and cloud distributed computing, as a journey
 
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
 
Survey of Big Data Infrastructures
Survey of Big Data InfrastructuresSurvey of Big Data Infrastructures
Survey of Big Data Infrastructures
 
SCALE 16x on-prem container orchestrator deployment
SCALE 16x on-prem container orchestrator deploymentSCALE 16x on-prem container orchestrator deployment
SCALE 16x on-prem container orchestrator deployment
 
A Closer Look at Apache Kudu
A Closer Look at Apache KuduA Closer Look at Apache Kudu
A Closer Look at Apache Kudu
 
Scale up is history! is scale out the future for storage
Scale up is history!  is scale out the future for storageScale up is history!  is scale out the future for storage
Scale up is history! is scale out the future for storage
 
start_your_datacenter_sds_v3
start_your_datacenter_sds_v3start_your_datacenter_sds_v3
start_your_datacenter_sds_v3
 
Flexible compute
Flexible computeFlexible compute
Flexible compute
 
Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticians
 
Data Lake and the rise of the microservices
Data Lake and the rise of the microservicesData Lake and the rise of the microservices
Data Lake and the rise of the microservices
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learn
 
Meta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinarMeta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinar
 
Big data and hadoop
Big data and hadoopBig data and hadoop
Big data and hadoop
 
Xldb2011 wed 1415_andrew_lamb-buildingblocks
Xldb2011 wed 1415_andrew_lamb-buildingblocksXldb2011 wed 1415_andrew_lamb-buildingblocks
Xldb2011 wed 1415_andrew_lamb-buildingblocks
 
StarlingX - Project Onboarding
StarlingX - Project OnboardingStarlingX - Project Onboarding
StarlingX - Project Onboarding
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
 
Best Practices for Deploying Enterprise Applications on UNIX
Best Practices for Deploying Enterprise Applications on UNIXBest Practices for Deploying Enterprise Applications on UNIX
Best Practices for Deploying Enterprise Applications on UNIX
 
The Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentThe Hard Problems of Continuous Deployment
The Hard Problems of Continuous Deployment
 
Meta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinarMeta scale kognitio hadoop webinar
Meta scale kognitio hadoop webinar
 

Kürzlich hochgeladen

Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 

Kürzlich hochgeladen (20)

Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 

How to Build a Compute Cluster

  • 1. How to Build a Cluster A High-Level Overview of the Key Issues Ramsay Key – May 2017
  • 2. Why should you care? • Although a lot of computing has been commoditized and delivered via IT and “the cloud”, it’s still useful to understand infrastructure: • Building your own “bare-metal” servers for yourself or a customer • Deploying your hardware into your customer’s data-center • Deploying your software into your customer’s data-center • Understanding considerations for non-traditional systems (i.e. vehicles, devices) • Some components need to built regardless of “public cloud” or private infrastructure
  • 3. Photo:Google - http://datacenterfrontier.com/google-building-four-story-data-centers/ The ideas and practices for building this apply even if you have just a single rack or half-rack
  • 4. Outline • Cluster Design • Racks and Servers • Networking • Day-to-Day Operations
  • 6. Gathering Requirements • Operations • What is the intended operation of the system? Prototype? Production? 24x7? Call-ins? • Type of system? Query? Analytics? Batch? Streaming? • Could the system grow? • Dataflow • How much data? Ingest volume (bytes/records)? Query load? Timeframe? • Compute • Type of computing: CPU heavy? RAM heavy? File I/O heavy? Network I/O heavy? • If these answers are not clear – try to derive upper and lower bounds on the needs
  • 7. RAM goes a long way these days! • All 4B IPv4 addresses fit in 18GB of RAM • 256GB of RAM can hold 16B MD5 hashes
  • 8. Design Considerations • Reliability • How reliable does your system need to be? How many “nines” of availability do you need? • Failure • How much redundancy is required? How much do you have? What about the facilities plant? • Scalability • Horizontal vs. Vertical. How much load can the system process? Scalability also includes people processes • Backup • Do you have a backup plan? • Application Deployments • How to deploy, manage, troubleshoot applications?
  • 9. Helpful Philosophies • Keep It Simple Stupid • Very easy to create complex server infrastructure. Strive for simplicity • Be Homogenous • Heterogeneity complicates scaling, debugging, and logistics • Expect Failure • Components will fail. Disks fail all the time. More computers ➔ More failure probability • Automate Everything • If you can reproduce your infrastructure quickly and easily, it is a good sign it is healthy
  • 11. Financial Considerations • Electricity availability/cost fundamentally dictates scale • Appropriate accounting/purchasing allows hardware to be depreciated • Can you “buy” your way out of scalability problems? (i.e. horizontal scaling) • Capital Expenditure (CapEx) vs. Operational Expenditures (OpEx) • CapEx = hardware, facilities • OpEx = labor, support, maintenance • Trade-off between CapEx and OpEx • Clusters generally try to minimize OpEx via automation, homogeneity • However, clusters don’t run and fix themselves – still need labor to support them
  • 13. Rackspace & Power, Space, Cooling • Datacenters have real physical constraints generally characterized as Power, Space, and Cooling (PSC) • Datacenters are laid out in “racks” (a.k.a. cabinets) • Datacenters have different “Tiers” (1-4) for handling different failure levels • Racks are standardized around 42 “rack units” height (a.k.a. “U”). • “Rack servers” commonly come in 1U and 2U dimensions. Width standardized. • 3U+ generally implied to be more unique hardware • Prefer rack-servers over “blade centers”
  • 14. Rack Considerations • 1U servers are considered “dense” • Need to pay attention to cooling and cabling • Can be hard to fit more elaborate components in 1U (GPUs or large hard-drives) • 2U servers are good all-around chassis • May lose some density per U • A good reference for an “all-up” rack is 40 servers and 2 “top-of-rack” (TOR) switches
  • 15. Power Distribution Units (PDU) – can run 2 for redundancy 1U servers 2 Top-of-rack (TOR) switches. Red and blue cables are “bonded” and provide redundancy and performance HID – Badge swipe access / alarm Fiber uplinks to datacenter spine From: https://techbloc.net/archives/970 1 “management” switch for administration. Separate network for when main network is down. Yellow.
  • 16. Server Selection • For purchasing, best to work through a value-added reseller (VAR) • Can assist with questions, delivery, coordination • Consider redundant power supplies depending on production level • Consider redundant network ports depending on production level • Hot-swappable hard-drives make life easy (almost standard now) • Make sure NICs will “PXEBoot” (i.e. network boot) • Consider RAID (Redundant Array of Inexpensive Disks) level. Popular options: • RAID10 a good mix between redundancy and performance • JBOD = Just a bunch of disks – let applications manage redundancy
  • 17. 10 2TB hot-swappable hard-drives Dual hot-swappable power supplies Dual CPUs, 10 cores 256GB RAM Fan bank for cooling (Not viewable) – 4 10G NIC ports, 2 1G NIC port, 1 IPMI port Typical commodity server – circa 2017 Generally don’t need tools to replace parts!
  • 18. Operating Systems • Linux is the OS of choice when building clusters • Lots of tools for managing and tuning Linux clusters at scale • Licensed software complicates scaling a cluster • These days many excellent open-source alternatives exists • CentOS (derivative of Redhat) and Ubuntu are both popular options • CentOS generally about stability and security • Popular with enterprises, IT, and operations people • Ubuntu generally newer and modern (closer to latest constituent software releases) • Popular with innovators, researchers, etc.
  • 19. Provisioning • Provisioning = building or rebuilding a node • Typical flow is for the node to “PXEBoot” into a kickstart (Redhat/Centos) or preseed (Ubuntu) an installer • Node first boots via DHCP then downloads a kickstart/preseed file • The kickstart/preseed file points to an installer and associated packages • Foreman is a system that facilitates the “PXEBoot”, kickstart/preseed process • Typical model is to put the bare minimum into kickstart/preseed and then let a configuration management system take over
  • 22. Network Considerations • Typical server network configuration would have: • 1Gb IPMI port • Allows interaction with basic server functions (power-on/power-off, etc.) • 1Gb management port • For server administration tasks, separate from data network • 2 10Gb data ports (in a bonded configuration) • For passing data between nodes • Keep in mind that disk I/O speed may be slower than the network
  • 23. Network Fabric • Typical rack configuration has two “top-of-rack” (TOR) switches that connect all the internal rack servers together, plus an “uplink” to the datacenter spine of “core” switches so it can talk to the other racks • Use two switches per rack for redundancy and throughput • Spine typically runs at 25Gb, 40Gb, or 100Gb http://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-7000- series-switches/white-paper-c11-737022.docx/_jcr_content/renditions/white- paper-c11-737022_3.jpg Racks
  • 25. Configuration Management • Configuration Management = tools that guarantee server configurations • Popular cluster configuration management tools: • Puppet: Most common, well-known, agent architecture • Chef: An early alternative to Puppet, also agent architecture • Ansible: Popular, agent-less architecture, integrates with networking gear well • SaltStack: More recent alternative, both agent and agent-less architecture • All tools have pros & cons - doesn’t matter so much what you use, just use one! • Possible to entirely define your infrastructure within the tools • Version control (git) the tool configurations and you get “infrastructure as code”
  • 27. Monitoring • Bad things will happen in your cluster! • The larger, the more complex the cluster, the more fantastic ways it can fail • Absolutely need monitoring of your cluster • Nagios is the most common tool for monitoring • Many alternatives: ElastAlert, Zenoss, Prometheus, xymon, Zabbix, Sensu, … • Tools send an alert (email, syslog, IM) when bad things happen • Tools usually come with some defaults – and have pluggable architectures
  • 29. Health & Status (Metrics) • “Measure Anything. Measure Everything” – Etsy • Instrument everything you can: • Useful for performance tuning • When bad things happen, these will be handy for identifying the root-cause • Applications, operating systems, processes, disks, network, memory, etc. • Ganglia is the most common tool for metrics • Many excellent alternatives: Grafana, Prometheus, Logstash, Graphite, statsd, collectd, OpenTSDB, Timely • Pick one and use it!
  • 31. Coordination Services • Often useful to run a “coordination service” within a cluster • Coordination service provides distributed reliable services for applications: • Configuration (identifying masters) • Naming (finding other services) • Synchronization (tracking state) • Typically present a “key-value” interface to clients • Popular implementations: Zookeeper, etcd, Consul, etc.
  • 32. Other Usefuls Tools • Public-Key Infrastructure (PKI, ssh) should be used for authentication • Avoid passwords – only root should have a password, if at all • Use LDAP to manage user accounts • Have some “database” that records which servers provide which functions • genders is a simple, popular way to do this • pdsh is a parallel shell command useful for running commands across a cluster • dshbak cleans up the output
  • 34. Software Deployment Considerations • Always a good idea to partition your cluster into: • Production servers - stuff you really care about…doesn’t need to be 24x7 to be production • Integration servers - last stop before being added to production • Test servers - general developer playground similar to production systems • Package, version, and install your software like a product • Helps for automation and traceability • Scripting languages (python, perl, etc.) can be risky to deploy because they can easily be changed once installed
  • 35. Future Considerations • Virtualization • Openstack, AWS, GCE, Azure, Rackspace • Containers • Docker, rkt, CoreOS, Swarm, Kubernetes • “Serverless” computing • Open Compute Project • Software-Defined-Networking (SDN)