Slides from live webinar hosted on February 16, 2017.
Deploying applications locally and bursting them to the cloud for compute may seem difficult, especially when working with high-performance, critical information. However, using cloudbursts to offset peaks in demand can bring big benefits and kudos from organizational leaders always looking to do more with less.
After this short webinar, you’ll be ready to:
- Explain what cloud bursting is and what workloads it is best for
- Identify efficiencies in applying cloud bursting to high-performance applications
- Understand how cloud computing services access your data and consume it during burst cycles
- Share three real-world use cases of companies leveraging cloud bursting for measurable efficiencies
- Have seen a demonstration of how it works
Presenters will build an actionable framework in just thirty minutes and then take questions.
5. Cloud Bursting
• Extend existing filesystem(s) into public compute
environment
• Retain application compatibility
• Leverage on-demand compute at massive scale
• R/W caching to eliminate WAN latency
6. Cloud Bursting Benefits
Use as Much (Or As Little) Compute as is Needed
• Simple to scale up and down, bursting to the cloud with Avere brings balance and efficiency to
an infrastructure by eliminating the need to house and maintain idle IT resources but still gain
the ability to get the job done quickly.
No Need to Rewrite Enterprise Applications
• With the Avere vFXT, enterprise apps do not have to be rewritten to run in the compute cloud.
By creating a NAS-based performance storage tier in cloud computing, applications gain access
to the data they need, while overcoming inherent NAS verses object API incompatibilities.
Data is Easy to Move, Easy to Manage
• Expansion of compute resources no longer involves hardware ordering, shipping, installation,
networking, configuring, or testing.
8. AvereOS Overview
• Components
• Clustered/Scale-out NAS system
• High-performance tier via R+W caching
• Global Namespace
• Built for parallel access: scale a working set across all nodes
• Scalable system
• 3 to 50 nodes within a single cluster
• Add nodes for additional performance and/or data locality
• High-Availability
• Single management interface, regardless of cluster size
• Easy expansion
9. Avere Overview: Public Compute
• Complete port to public compute environments
• All features + functionality intact: same look/fit/finish/feel as physical product
• Instantiated via dedicated management micro-instance
• Or, can be configured via command line interface
• Valuable for orchestration workflows
• One command, one cluster (no matter the size)
11. Cloud Bursting - Networking Challenges
Quick Deployment
SSH Tunneling
Flexible Network
VPN over Internet
Best Performance
Dedicated 10gig Link
-- Performance ++
++ Deployment Ease/Speed --
11
10Gbit
12. • Compute Cloud Components
• Compute Instances
• Disks/SSDs
• Network Interfaces
• AvereOS Software Components
• Install OS onto instance disks
• Bootstrap a configuration
• Cluster the system together
• vFXT Cluster
• IP addresses for clients to mount
• Core filer storage configuration
• Filesystem namespace access to on-premises data
Cloud Bursting – The Cloud Puzzle Pieces
12
SW/OS
SSD
NET
13. Cloud Bursting – Avere OS Orchestration Options
• Create/Monitor/Destroy vFXT clusters
• Interaction with Cloud Provider APIs
• Assemble the cloud components
• Interactions with Avere APIs
• Configure the cluster for file access
• Avere vFXT Cluster Manager
• Cloud Compute Instance with UI
• Avere provides access to disk image
• Graphical User Interface
• API (XMLRPC based)
• Python Library Orchestration
• Provide your own Python environment
• Avere provides python package
• Command-line orchestration
• Programmatic Python calls directly into library
13
AvereOS
SSD
14. Cloud Bursting – I/O Performance Capabilities
• GCE Instance Types
• n1-highmem-8: capable of 8Gbit/sec raw network throughput
• n1-highmem-32: capable of 16Gbit/sec raw network throughput
• GCE Disk Types
• PersistentSSD: capable of 240MB/sec read, 240MB/sec write
• LocalSSD: capable of 1560MB/sec read, 1090MB/sec write
• vFXT Scale-Out Clustering
• Adding vFXT nodes to the cluster grows disk cache and disk/network throughput
• Balancing client activity across the cluster vFXT nodes yields optimal performance
• vFXT File/Directory Caching
• Read or Read+Write caching
• Performance vs. Consistency can be dialed-in
14
16. Use-case: Media and Entertainment
10.1.1.191:/vfx
NFS / SMB
VPN
Scenario
- Media company requires extra compute to complete a project, but cannot acquire new
hardware within a reasonable timeframe. Beg/borrow/steal isn’t an option.
- Avere vFXT cluster in EC2/GCE/Azure
- Render assets lifted from Local NAS to public compute environment
- Jobs run/data is processed, output written down to Local NAS
- vFXT cluster (and compute) powered down to reduce cost
- Data optionally resident in compute environment for next run
Local NAS /
Datacenter
17. Use-case: Financial Services and Technology
10.1.1.192:/marketdata
NFS / SMB
VPN
Scenario
- Financial Services company would like to process new models at massive scale for 1-2
hours per day, without changing researcher workflow
- Avere vFXT cluster in EC2/GCE/Azure
- Researcher generates new model, writing directly to Local NAS as always
- vFXT and 10,000-50,000 compute cores spun up to begin work
- Output written directly to Local NAS for additional interpretation
- Compute and vFXT assets powered down or removed to minimize cost
Local NAS /
Datacenter
18. Use-case: Life Sciences NGS
10.1.1.193:/genomes
NFS / SMB
VPN
Scenario
- A Drug Discovery company would like to process genetic data from drug trials at
massive scale in the compute cloud, without growing in-house compute infrastructure
- Avere vFXT cluster in EC2/GCE/Azure
- Researcher obtains new genomic models, writing directly to Local NAS as always
- vFXT and tens of thousands of compute cores spun up to begin work
- Output written directly via vFXT funnles back to Local NAS for additional analysis
- Compute and vFXT assets powered down or removed to minimize cost
Local NAS /
Datacenter
20. The Demo Environment
20
Infrastructure
• Network Connectivity
• Orchestrator
• vFXT cluster
• Compute Client Instances
Application
• Linux NFS grid application
• CPU intensive computation
• Read intensive I/O workload
On-Prem Datacenter
NAS
/on-prem-data
/on-prem-archive
Compute Cloud Clients have
real-time cached access to on-
premises data via GCE-hosted
vFXT Cluster
Clients on-premises create
the content and consume the
results, consisting of desktop
users, NAS storage and
compute grid
Google Cloud Project
vFXT
/ (root)
|--- /on-prem-data
|--- /on-prem-archive
compute-node##
GCE vFXT.py
Orchestrator
21. Pre-Flight Checklist
• Can the orchestrator reach the GCP Project network?
• Does the GCP Project network have a network path back to on-premises NAS
core filer?
• Can the GCE vfxt.py instance reach the googleapis.com endpoint?
• Is the vfxt.py python library installed on the GCE instance?
• Does the vfxt.py GCE instance have the proper permissions/credentials to
orchestrate the vFXT cluster creation?
• Determine the region/zone to build the cloud infrastructure
• Are the project quotas set up for the instances/storage we plan to deploy?
OK! Time for liftoff!!!
21
22. 22
April 24-27, 2017
Las Vegas, Nevada
Booth SL9627
May 15-17, 2017
Hilton Midtown
New York, New York
May 17-18, 2017
Navy Pier
Chicago, Illinois
23. Questions?
23
Damian Hasak
Solutions Architect Lead
Avere Systems
dhasak@averesystems.com
Bernhard Behn
Principal Technical Marketing Engineer
Avere Systems
bbehn@averesystems.com
www.averesystems.
com