Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently. The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined. Watch the video: https://wp.me/p3RLHQ-lBX Learn more: https://www.zettar.com/ and https://rice2020oghpc.rice.edu/program-2/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Recomendados
Más contenido relacionado
Was ist angesagt?
Was ist angesagt?(20)
Similar a Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Similar a Zettar: Moving Massive Amounts of Data across Any Distance Efficiently(20)
Más de inside-BigData.com
Más de inside-BigData.com(20)
Último
Último(20)
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
- 1. 1 Chin Fang, Ph.D. Founder & CEO, Zettar Inc. © 2020 Zettar Inc. All rights reserved Moving Massive Amounts of Data across Any Distance Efficiently
- 2. 2 The Consequence of The Lack of Data Mobility 90% of the world’s data was generated in the past two years (i.e. since 2016) Only about 1% of it is utilized, processed and acted upon Navin Shenoy , EVP, DCG, Intel Corp. 1 2 Innovation Summit, August 2018 © 2020 Zettar Inc. All rights reserved
- 3. 3 State of The Art and Common Misconceptions Firewalls everything Proprietary protocols better than TCP 2 © 2020 Zettar Inc. All rights reserved 1 AI, ML, DL and LOSF4 Create data lakes5 Virtualization everywhere 3
- 4. 4 Data Movement Solution Categories in The Modern Times 1 Data traffic aggregation solution 2 3 Data transport solution: co-design mandatory Data access solution • Source data mostly stays. • Interactive usage oriented • Desired data fetched in small amounts, possibly from many storage pools • Coupled data flows. One or more storage pools at each end • Predominately machine generated, batch-processed • Source data must be moved or replicated, mostly at scale • Independent data flows aggregated • Mainly a network oriented operation (SD-WAN) • Multiple independent storage pools to multiple storage pools on both ends Example s: © 2020 Zettar Inc. All rights reserved Most dominant & challenging!
- 5. 5 Picture credit: Wikipedia A Co-Design Endeavor! A Water Transport Analogy © 2020 Zettar Inc. All rights reserved Get these ready first!
- 6. 6 Project 1: Investigate the impact of using K/V SSDs © 2020 Zettar Inc. All rights reserved 2 clusters 90+ Gbps 2 1 2 storage node & transfer node / cluster 3
- 7. 7 Project 2: Poland-Singapore Data Transfer Production Trial over CAE-1 © 2020 Zettar Inc. All rights reserved Picture credits: left – ICM, Warsaw, Poland; right – Vincenzo Capone, GÉANT
- 8. 8 2019 – 2020 Q1 - Parallel Streaming © 2019 Zettar Inc. All rights reserved One pump on each end, single pipe, easy!Two pumps on each end, two pipes, hmm Very easy to ensure the sequential order of different flow elements on the receiving end No guarantee of the sequential order of different flow elements on the receiving end anymore ?
- 9. 9 Chin Fang, Ph.D. Founder & CEO, Zettar Inc. Thank you! © 2020 Zettar Inc. All rights reserved
- 10. 10 The Concept of Alpha – WAN Connections Are Different! © 2020 Zettar Inc. All rights reserved Picture credit: Dr. Les Cottrell, SLAC National Accelerator Laboratory
- 11. 11 Network Connectivity Between Asia and Europe – Early 2019 © 2020 Zettar Inc. All rights reserved Picture credit: NSCC, Singapore
- 12. 12 CAE-1 for Asia and Europe – August 2019 © 2020 Zettar Inc. All rights reserved Picture credit: NSCC, Singapore 100Gbp s
- 13. 13 Computational Power Source – Two Supermicro Server Models © 2020 Zettar Inc. All rights reserved SYS-7049P-TR SYS-2028U-TN24R4T+
- 14. 14 Initial Link Bandwidth Qualification 1: iperf3 © 2020 Zettar Inc. All rights reserved
- 15. 15 Link Bandwidth Qualification 2: Data Mover Memory-to-Memory © 2020 Zettar Inc. All rights reserved
- 16. 16 Storage Throughput Qualification: Parallel Filesystem Parallel Tasks © 2020 Zettar Inc. All rights reserved
- 17. 17 Actual Storage-to-Storage Transfer Bandwidth utilization © 2020 Zettar Inc. All rights reserved