Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Deep Dive: Using Hybrid Storage wi...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• AWS Storage Gateway highlights
• Custom...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data movement
OnlineOffline
Data security
and ma...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway
• Seamlessly connect on-prem...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage Gateway Hybrid Storage Solutions
U s e s...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling a Spectrum of Hybrid Use Cases
Backup |...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A Few of Our Customers
© 2017, Amazon Web Servic...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Stemcell Technologies’
Hybrid Storage Architectu...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STEMCELL Technologies
- Canada’s largest biotech...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
15%
On-premises
datacenter
Environment Snapshot
...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
5%
On-premises resources to
support local worklo...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fits our workload Cost-effective growth Low IT o...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Windows
Server
Our Primary Use Case
S3 EBS
Snaps...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Secondary Use Cases
S3
NFS Client
File Gateway
F...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architecture Detail
cached
volumes
S3 EBS
Snapsh...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Network Design – Volume Cache
Storage
Gateway
Se...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Cloudwatch Monitoring/Alerting
- Per-volu...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Backup & Restore
EBS Snapshots for backup
- Crea...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Gateway Sizing – Real World
Hardware
- Make it f...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s Next
ESXi Dedicated
Hardware
Upgrade
NFS ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway Deep Dive
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway Amazon S3 Standard &
Infrequ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s New for Storage Gateway
• File Gateway (N...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tape Gateway
• Virtual tape storage in S3 & Glac...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Block storage in S3 accessed via on-
premises ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Stored
volume Amazon EBS
snapshots
Custom...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached
volume Amazon EBS
snapshots
Custom...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached (Write Operation)
volume Amazon EB...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached (Read from Cache)
volume Amazon EB...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached (Read from Cloud)
volume Amazon EB...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Snaps & Clones: Backup & DR
Region
Storage Gatew...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
Customer Premi...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
A B
A
Ca...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
ASnapsho...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
A B C
A ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
A B C
A ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
O n - p r e m i s e s f i l e s t o...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
s3://{bucket name}/{folder1}/{file1}
s3://{bucke...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
C o n t r o l h o w y o u r d a t a...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
M e t a d a t a a n d w o r k l o a...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Reads, Writes, and Updates)
Custom...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Reads)
Customer Premises
File Gate...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Writes)
Customer Premises
File Gat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
Customer Premises
File Ga...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
File Gateway VM
A B CA B ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
A B CA B C D A B C
A B C ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
A B C D
A B C D E F
A B C...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
A B CA H I J E F
A B C D ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Content Distribution with File Gateway
Storage
G...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway: Automated Hybrid Workflows
with Cl...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
For Enterprise Storage Engineers
• Learn how to ...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn More in These Sessions
• STG317 – Chalk Ta...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Questions?
Nächste SlideShare
Wird geladen in …5
×

STG309_Deep Dive Using Hybrid Storage with AWS Storage Gateway to Solve On-Premises Storage Problems

1.253 Aufrufe

Veröffentlicht am

Enterprises of all sizes face continuing data growth and persistent requirements to back up and recover application data. The pains of recurring storage hardware purchasing, management, and failures are still acute for many IT organizations. Some also need to integrate on-premises datasets with in-cloud workloads, such as big data processing and analytics. Learn how to use AWS Storage Gateway to connect on-premises applications to AWS storage services using standard storage protocols, such as NFS, iSCSI, and VTL. Storage Gateway enables hybrid cloud storage solutions for backup and disaster recovery, file sharing, in-cloud processing, or bulk ingest for migration. We discuss use cases with real-life customer stories, and offer best practices.

  • Als Erste(r) kommentieren

STG309_Deep Dive Using Hybrid Storage with AWS Storage Gateway to Solve On-Premises Storage Problems

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Deep Dive: Using Hybrid Storage with AWS Storage Gateway to Solve On - Premises Storage Problems A d a m L e g g e t t , M a n a g e r , I T , S t e m c e l l T e c h n o l o g i e s A s a K a l a v a d e , G e n e r a l M a n a g e r , A W S P a u l R e e d , S r . P r o d u c t M a n a g e r , A W S E v e r e t t D o l g n e r , B u s i n e s s D e v e l o p m e n t M a n a g e r , A W S N o v e m b e r 2 8 , 2 0 1 7 S T G 3 0 9
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • AWS Storage Gateway highlights • Customer Stemcell Technologies’ hybrid architecture • Diving deep • Volume gateway • File Gateway
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data movement OnlineOffline Data security and management Complete Set of Data Building Blocks Amazon EFS Amazon EBS AWS Snow family AWS Storage Gateway Family AWS Direct Connect Amazon EFS File Sync Amazon S3 Transfer Acceleration Storage Partners Amazon Kinesis Data Streams Amazon Kinesis Video Streams Amazon S3 Amazon Glacier AWS KMS AWS IAM AWS CloudWatch AWS CloudTrail AWS CloudFormation AWS Lambda Amazon Macie AWS QuickSight
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Storage Gateway • Seamlessly connect on-premises applications with AWS • Leverages cloud storage scalability, reliability, durability, and economics • Use cases span entire customer journey to the cloud • Key benefits • Integration with AWS storage, management, and monitoring • No changes to existing applications • Local cache for low-latency access to data stored in AWS • Optimized data transfer
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage Gateway Hybrid Storage Solutions U s e s t a n d a r d s t o r a g e p r o t o c o l s t o a c c e s s A W S s t o r a g e AWS Storage Gateway Amazon EBS snapshots Amazon S3 & S3 – infrequent access Amazon Glacier AWS Identity and Access Management (IAM) Amazon CloudWatch Files Volumes Tapes On-premises AWSCloud AWS Key Management Service (KMS) AWS CloudTrail
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enabling a Spectrum of Hybrid Use Cases Backup | DR | Archive Enterprise Data Centers Amazon EBS Amazon S3 Amazon Glacier App. serversFile servers Research Sites AWS Storage Gateway DevicesDatabasesMultimedia content Analytics | File Services | Production Tiering | Data Processing Data Distribution | Backup | DR | Archive | Migration Amazon EC2 AWS Lambda Amazon CloudFront Amazon Athena Amazon EMR Backup serversUsers Remote Offices Small-Medium Businesses
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A Few of Our Customers © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Stemcell Technologies’ Hybrid Storage Architecture A d a m L e g g e t t , I T M a n a g e r
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. STEMCELL Technologies - Canada’s largest biotechnology company - Privately-held, based in Vancouver, BC - ~1100 staff, offices in eight countries - “Scientists Helping Scientists™” - Specialty products for life science research - Cell culture, cell isolation and accessory products - Quality focused
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 15% On-premises datacenter Environment Snapshot 45% IaaS / PaaS 40% SaaS
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 5% On-premises resources to support local workloads Environment Roadmap (2018) 55% IaaS / PaaS 40% SaaS
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fits our workload Cost-effective growth Low IT overhead Fits our strategy Why Storage Gateway? - Image and instrument acquisition/analysis - Low latency & local - 2-3 months of “hot” files - Reduce capex $ - Scale without buying disk - Small branch to Corp HQ - Use commodity servers - No storage admin needed - Easy backup/restore/DR - No change to business workflows, no retraining - Our business is science - Cost efficient IT services - Grow really quickly!! - Compliance and sovereignty
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Windows Server Our Primary Use Case S3 EBS Snapshot Storage Gateway Service DFS Volume Gateway Cached … Site A Site X - 1 gateway/site - 4-10 departments per gateway - *High-throughput workloads have dedicated gateway Windows Server Volume Gateway Cached
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Secondary Use Cases S3 NFS Client File Gateway File Gateway - Used for visual media team archive/offload - Low throughput requirement - Run local VM on (cheap) SATA disk - No LUN/disk size restriction Volume Gateway – Stored - Used EBS Snapshots for backup of on- premises Oracle databases - Now replicating to AWS warm-site - Have moved to native S3 backups File Gateway Architecture
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Architecture Detail cached volumes S3 EBS Snapshot Storage Gateway Service iSCSI Windows Server SMB HTTPS Region x4 SMB2 Mapped Drives VMDK Upload Buffer VMDK Cache VMDK OS DiskPoolA DiskPoolB DiskPoolC VMware ESXi on Cisco UCS Data Center DFS
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Network Design – Volume Cache Storage Gateway Service iSCSI Windows Server vSwitch-STORAGEvSwitch-SG vSwitch-Server Volume Cached Gateway 3 network zones - SG provisioned with 2 NICs; VMXNet3 - SG:eth0; northbound connection to SG service via https, segmented zone - SG:eth1 for iSCSI - No default gateway, no exit path. iSCSI dedicated traffic - Windows server configured with 2 NICs as well; allows iSCSI tuning and client segmentation SMBHTTPS Windows Client eth0 eth1
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Cloudwatch Monitoring/Alerting - Per-volume monitoring - cachehitpercentage (80%) - cachepercentdirty (2.5%) - cachepercentused - Monitor the upload buffer for performance issues - Gateway read/write times - Shows performance issues with hardware - Correlate with VMware - Tune alerts based on workload - Review monthly cache averages for sizing/growth
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Backup & Restore EBS Snapshots for backup - Created via Storage Gateway - 1h/4h/1d RPO (based on data classification) - Retained based on application/business requirements - Lambda script to clean-up old snaps Restore process in AWS - Warm standby gateway and file server in AWS - Re-attach snap; pull files - 10 min to attach and have data available - 30 min RTO; re-point DFS Real-world ‘gotcha’: VSS not supported Storage Gateway Service Volume Gateway AMI on EC2 EC2 Instance Storage Gateway EBS Snapshot Warm standby Windows Server on EC2
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Gateway Sizing – Real World Hardware - Make it fast, make it dedicated! - Lots of CPU, avoid contention - Memory not a huge difference greater that 16 GB - SSD or SAS with good controllers - Building NVMe-based GW Volumes and Disk - Split disk pools for Upload Buffer, cache, and OS - Size Upload Buffer by your internet connection - We use 500 GB - 5 days worth of data (oversized) - Size cache by data ingest, use, and turn rate - R&D data is 3 months of on-cache data (12 TB) - General business apps: 1 mon. (500 GB) - Watch for disk contention! - We thin-provision disk…but its not a shared pool
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What’s Next ESXi Dedicated Hardware Upgrade NFS File Gateway Now Near-term (three to six months) EC2 Cache Gateways via Direct Connect Directional File Gateways for Primary Cloud Analytics Workload
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Storage Gateway Deep Dive
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Storage Gateway Amazon S3 Standard & Infrequent Access Files (NFS v3 / v4) Volumes (iSCSI) Tapes (iSCSI VTL) AWS Storage Gateway Data stored in Amazon S3 with • Native S3 management (File) • Snapshots to EBS (Volume) • Archive to Amazon Glacier (Tape) Service accessed through VM which provides • Protocol conversion and device emulation • Caching (read-through / write-back) • Optimized data transfer HTTPS Amazon EC2 Amazon Glacier Amazon EBS snapshots
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What’s New for Storage Gateway • File Gateway (November 2016) • Reduced pricing (December 2016) • 12 launches in 2017 • Available in all public regions including China (Beijing) • HIPAA-eligible (October 2017) • Tape • Additional backup apps • Improved performance enhancements (pre-fetch and faster retrievals) • Volume • Cloning for faster DR • File Gateway • Refresh cache for multi-site file sharing • Upload notifications (new!)
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tape Gateway • Virtual tape storage in S3 & Glacier accessed via gateway • Data compressed in-transit and at-rest • Up to 1 PB total tape storage per gateway, unlimited archive capacity • Supports leading backup applications • Three- to five-hour tape retrieval from Amazon Glacier Media Changer Virtual Tapes stored in Amazon S3 Archived Tapes in Amazon Glacier Backup Server iSCSI Tape Gateway Tape Gateway VM Tape Drive HTTPS Customer Premises Region Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management Local Cache
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Block storage in S3 accessed via on- premises gateway appliance VM • Compresses data in-transit and at-rest • Up to 1PB of total volume storage per gateway appliance • Create EBS snapshots of on-premises volumes • Create volumes from EBS snapshots • Two modes: Volume Stored and Volume Cached O n - p r e m i s e s v o l u m e s b a c k e d b y A m a z o n S 3 w i t h E B S s n a p s h o t s Volume Gateway Application Server Amazon EBS snapshots Storage Gateway Service Volume Gateway VM Volume Stored in Amazon S3 HTTPSiSCSI Customer Premises Region
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume Stored volume Amazon EBS snapshots Customer Premises iSCSI Storage Gateway VM Region stored volumeApp Server complete volume stored in the cloud and on-prem Amazon CloudWatch Storage Gateway Service Supporting storage can be SAN, VMDK, VHD upload buffer
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume Cached volume Amazon EBS snapshots Customer Premises App Server Storage Gateway VM Region cache virtual volume upload buffer complete volume stored in the cloud virtual volume stored on-prem backed by cache and cloud Amazon CloudWatch Storage Gateway Service iSCSI
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume Cached (Write Operation) volume Amazon EBS snapshots Customer Premises App Server iSCSI Storage Gateway VM Region cache virtual volume upload buffer compress Amazon CloudWatch Storage Gateway Service
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume Cached (Read from Cache) volume Amazon EBS snapshots Customer Premises App Server iSCSI Storage Gateway VM Region cache virtual volume upload buffer Amazon CloudWatch Storage Gateway Service
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume Cached (Read from Cloud) volume Amazon EBS snapshots Customer Premises App Server iSCSI Storage Gateway VM Region cache virtual volume upload buffer Amazon CloudWatch Storage Gateway Service
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Snaps & Clones: Backup & DR Region Storage Gateway Service Amazon EBS snapshots EC2 instances Win Server Availability Zone #2 EC2 instance Win Server EBS Volume Volume GW Application Server Volume Gateway VM Customer Premises iSCSI Availability Zone #1 Clone • Gateway volumes are continually written to S3 • Point in time protection via EBS snaps • Ability to recover to EBS from snaps • Ability to clone the latest volume to another gateway in the event of a disaster Volume
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS Snapshots with Volume Gateway Customer Premises App Server Storage Gateway VM Region virtual volume Storage Gateway Service cache upload buffer GW Volume Amazon EBS snapshots A B C A A A B C B
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS Snapshots with Volume Gateway A B C A B A Call snapshot via API to GW service Snapshot Customer Premises App Server Storage Gateway VM Region virtual volume Storage Gateway Service cache upload buffer GW Volume Amazon EBS snapshots
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS Snapshots with Volume Gateway A B C ASnapshot A B C Call snapshot via API to GW serviceCustomer Premises App Server Storage Gateway VM Region virtual volume Storage Gateway Service cache upload buffer GW Volume Amazon EBS snapshots
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS Snapshots with Volume Gateway A B C A B C A B C I’d like to snap this volume please … Customer Premises App Server Storage Gateway VM Region virtual volume Storage Gateway Service cache upload buffer GW Volume Amazon EBS snapshots
  36. 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EBS Snapshots with Volume Gateway A B C A B C A B C Snapshot completeCustomer Premises App Server Storage Gateway VM Region virtual volume Storage Gateway Service cache upload buffer GW Volume Amazon EBS snapshots
  37. 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway O n - p r e m i s e s f i l e s t o r a g e m a i n t a i n e d a s o b j e c t s i n A m a z o n S 3 • Storage in your S3 buckets with 1:1 mapping from files-to-objects • File metadata stored in object metadata • Full control over how your data is accessed and stored Customer Premises Glacier HTTPS NFS v3 / v4.1 Application Server Objects in your S3 bucket Storage Gateway VM
  38. 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. s3://{bucket name}/{folder1}/{file1} s3://{bucket name}/{folder1}/{file2} s3://{bucket name}/{folder1}/{file3} s3://{bucket name}/{folder1}/{file4} s3://{bucket name}/{folder2}/{file1} s3://{bucket name}/{folder2}/{file2} File Gateway O n e - t o - o n e m a p p i n g f r o m f i l e s - t o - o b j e c t s nfs://{IP}/{share name}/{folder1}/{file1} nfs://{IP}/{share name}/{folder1}/{file2} nfs://{IP}/{share name}/{folder1}/{file3} nfs://{IP}/{share name}/{folder1}/{file4} nfs://{IP}/{share name}/{folder2}/{file1} nfs://{IP}/{share name}/{folder2}/{file2} File Gateway VM NFS client Objects in your S3 bucket
  39. 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway C o n t r o l h o w y o u r d a t a i s a c c e s s e d a n d s t o r e d NFS/file options export • IAM role for access • Storage class • Object encryption with KMS • Guess MIME type ** NEW! ** S3 options per bucket • List of allowed clients • Read-only/read-write exports • User squashing • Default object ownership (uid/gid) • Default object permissions File Gateway VM NFS client Objects in your S3 bucket
  40. 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway M e t a d a t a a n d w o r k l o a d a u t o m a t i o n Metadata and object inventory Data cache File share File Gateway VM RefreshCache API Amazon CloudWatch Events NFS client Objects in your S3 bucket ls /foo/*
  41. 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Reads, Writes, and Updates) Customer Premises File Gateway VM Region share Storage Gateway Service NFS Client File Stored as objects index cache
  42. 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Reads) Customer Premises File Gateway VM Region share Storage Gateway Service NFS Client File Stored as objects ranged read operations if required bytes are not in cache read operation from cache index cache
  43. 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Writes) Customer Premises File Gateway VM Region share Storage Gateway Service NFS Client File Stored as objects multipart upload write to cache first index cache
  44. 44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Updates) Customer Premises File Gateway VM Region NFS Client File Stored as objects A B C A B C
  45. 45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Updates) File Gateway VM A B CA B C D A B C Customer Premises Region NFS Client File Stored as objects
  46. 46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Updates) A B CA B C D A B C A B C D Customer Premises Region NFS Client File Stored as objects File Gateway VM
  47. 47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Updates) A B C D A B C D E F A B C D E F Customer Premises Region NFS Client File Stored as objects File Gateway VM
  48. 48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway (Updates) A B CA H I J E F A B C D E F A H I J E F Customer Premises Region NFS Client File Stored as objects File Gateway VM
  49. 49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Content Distribution with File Gateway Storage Gateway VM S3 Bucket Read-only NFS client Cross region replication Los Angeles HQ Denver site NFS client S3 Bucket US-West – Oregon Region Melbourne site Sydney, Australia Region Storage Gateway VM Read-only NFS client Storage Gateway VM EC2Storage Gateway AMI NFS Client RefreshCache API File Upload Notification NFS client M u l t i - s i t e a c c e s s , 1 b u c k e t , a u t o m a t e dM u l t i - r e g i o n a c c e s sA c c e s s i n - c l o u d v i a N F SI n g e s t v i a S n o w b a l l E d g e , a c c e s s w i t h F i l e G a t e w a y
  50. 50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway: Automated Hybrid Workflows with CloudWatch Events Customer Premises File Gateway VM File share Storage Gateway ServiceNFS Client File stored as objects in S3 Cache Devices Users Amazon CloudWatch File Upload Notification Amazon Athena CloudWatch Event starts a query-in- place job Athena writes results set back to S3 Data flow Messaging flow Query-in-place operation RefreshCache
  51. 51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. For Enterprise Storage Engineers • Learn how to architect and manage highly available solutions on AWS storage services • Advance toward AWS certifications • Help your organization migrate to the cloud faster Online at www.aws.training • Access 100+ new digital training courses including advanced training on storage • Deep dives on S3, EFS, and EBS • Migrating and tiering storage to AWS (hybrid solutions) At re:Invent • Visit Hands-on Labs at the Venetian • Attend a proctored “Introduction to EFS” Spotlight Lab on Thursday at 3pm at the Venetian • Meet storage experts at the Ask the Experts in Hands-on Labs room at the Venetian New Storage Training
  52. 52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Learn More in These Sessions • STG317 – Chalk Talk: Deploying HA Windows File Services w/ Cloud Tiering – today @12:15pm • STG320 – Workshop: Hybrid Storage Unleashed, featuring Storage Gateway – today @2:30pm • STG201 – Storage State of the Union, tomorrow, Wed. @11:30am • ENT322 – Disaster Recovery with AWS: Tiered Approaches – tomorrow, Wed. @1:45pm • STG310 – Files in AWS, featuring Celgene Corp – tomorrow, Wed @4:00pm • STG204 – Migrating Large Scale Data Sets to the Cloud – Thur. @3:15pm • STG203 – Get rid of tape & modernize backup w/ AWS, featuring Southern Oregon University (Monday – catch the replay on YouTube)
  53. 53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Questions?

×