Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

BeeGFS Enterprise Deployment

SC19 BoF talk about our Deployment of a BeeGFS storage system

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

BeeGFS Enterprise Deployment

  1. 1. Posix file systems at Fred Hutch 2012: Scale Out NAS - 100s of TB, - Scaleout NAS replacing single filers - Consolidated HPC defines life sciences performance requirements 2014: BeeGFS Scratch - Up to 500TB Scratch - No redundancy - 100% uptime for 3 years - 10 research groups - 2 month project 2019: BeeGFS for all - Migrating to redundant deployment with HA - 150 research groups - 700M files, 2 PiB - Higher Expectations - Backup to cloud - 6 month project © Fred Hutchinson Cancer Research Center 0
  2. 2. BeeGFS Enterprise deployment – what does this mean? © Fred Hutchinson Cancer Research Center 1
  3. 3. “Fast File” service architecture - 3 racks each in 2 different DC pods - buddy mirrors of Meta and Storage - Clustered SMB/NFS NAS Gateway © Fred Hutchinson Cancer Research Center 2 - scale to 8 PiB usable capacity - Management server on Vmware - Flash Storage Pool
  4. 4. Standard Hardware - Supermicro standard workhorse - 34 x 14 TB drives = 476 TB RAW - WD HC530 and Seagate Exos X14 (mirror) - Micron SATA SSD for ZFS caching © Fred Hutchinson Cancer Research Center 3 - Intel Optane low latency NVMe for metadata - Metadata: cores, more cores - Cisco Nexus 9300 – 100G
  5. 5. A word about ZFS (ZoL) …..just use it! - 3 RAID groups with 11 drives each, one spare - RAIDZ3 most efficient, best resilver times - Resilver times 50% improved in v.0.8 - Large 14TB drives not a problem - Easy to use encryption in v.0.8 - Faster with LZ4 compression than without - 3.5 GB/s disk throughput - Comes with Ubuntu ! © Fred Hutchinson Cancer Research Center 4
  6. 6. Reliable monitoring! - Drives fail all the time - Benchmark while re- silvering ! - Notification to where you hangout (Slack?) © Fred Hutchinson Cancer Research Center 5
  7. 7. SMB, NFS, SFTP gateway - Clustered Samba 4.11.2 with CTDB - Latest Winbind scales to 100k users - CTDB also manages clustered NFS 4.2 - Kerberized NFS is finally working well - /fast -rw,fsid=101,no_subtree_check *(sec=krb5i:krb5p,root_squash) @cluster(sec=sys,no_root_squash) - Read only mounts to Backup/DR solution (ObjectiveFS) - S3 (minio) and http/WebDav are planned. © Fred Hutchinson Cancer Research Center 6
  8. 8. A shoutout to Samba+ from Sernet - Repositories with latest Samba for your OS - 6 Samba core developers on Staff - Outstanding support - HA with CTDB is stable & easy to install but hard to troubleshoot - Latest Winbind is much better ! © Fred Hutchinson Cancer Research Center 7
  9. 9. Some eye-candy with Prometheus & Grafana © Fred Hutchinson Cancer Research Center 8
  10. 10. More eye-candy with Prometheus & Grafana © Fred Hutchinson Cancer Research Center 9
  11. 11. Even more eye-candy with Prometheus & Grafana © Fred Hutchinson Cancer Research Center 10
  12. 12. Optane: 400k+ metadata read iops, 80-100k write iops © Fred Hutchinson Cancer Research Center 11
  13. 13. How much does it cost per month !? - Hardware cost for 5 years, divide by 60 - Support costs and license fees by month - Capacity is total file system space in TebiByte - Don’t forget sales tax ! - $1 per TiB per month for hardware, $2 mirrored - $1 per TiB per month for BeeGFS support - = $3 per TiB per month without compression - 1/3 LZ4 compression = $2 / TiB / month total cost mirrored © Fred Hutchinson Cancer Research Center 12
  14. 14. What does Fred Hutch want next from BeeGFS ? - ZFS based snapshots in BeeGFS - HA for Management server - --rebalance feature to ease cluster growth - Delta-sync for Metadata server - More explicit support for Ubuntu © Fred Hutchinson Cancer Research Center 13
  15. 15. Thank you ! Questions petersen@fredhutch.org Slides https://www.slideshare.net/dirkpetersen © Fred Hutchinson Cancer Research Center 14

×