Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
May 6, 2019 1
Improving Data Transfer and Delivery using Globus
Recent Upgrades to The Atmospheric Radiation
Measurement (...
ARM’s Vision
2
To provide a detailed & accurate description
of the earth atmosphere in diverse climate
regimes to resolve ...
ARM Data Flow – The Big Picture
Data Growth
1 PB
3
ARM Data – Disaster Recovery
Offsite Data backup
ARM Data files that are copied into the ORNL HPSS system at ORNL are also...
Data Transfer and Staging to Facilitate Data Science
5
Data Transfer and Staging to Facilitate Data Science
6
Data Pipeline and Software Architecture
May 6, 2019 7
Data Processing
Storage &
Data
Model
Querying Analytics Scientific
U...
Data Retrieval, Packaging, and Delivery
§ Merging
§ DQR filtering
§ Conversion
Retrieval
Future
capability
Data-
streams
H...
Data Discovery Tool
9
10
§ Based on big data analysis platform
(NoSQL)
§ ARM HPC Clusters for data
processing
§ Provides an interactive web
inte...
Questions?
5/6/19 11
ARM Data Services
Giri Prakash
palanisamyg@ornl.gov
Nächste SlideShare
Wird geladen in …5
×

Recent Upgrades to ARM Data Transfer and Delivery Using Globus

16 Aufrufe

Veröffentlicht am

This presentation was given at the 2019 GlobusWorld Conference in Chicago, IL by Giri Prakash from the ARM Data Center at Oak Ridge National Laboratory.

Veröffentlicht in: Technologie
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Gehören Sie zu den Ersten, denen das gefällt!

Recent Upgrades to ARM Data Transfer and Delivery Using Globus

  1. 1. May 6, 2019 1 Improving Data Transfer and Delivery using Globus Recent Upgrades to The Atmospheric Radiation Measurement (ARM) Facility Data Center Architecture GIRI PRAKASH, ZACH PRICE, JOSEPH OLATT, AND JITU KUMAR ARM Data Center, Oak Ridge National Laboratory Globus World, May 01, 2019 palanisamyg@ornl.gov
  2. 2. ARM’s Vision 2 To provide a detailed & accurate description of the earth atmosphere in diverse climate regimes to resolve the uncertainties in climate and earth system models toward the development of sustainable solutions for the Nation’s energy & environmental challenges.
  3. 3. ARM Data Flow – The Big Picture Data Growth 1 PB 3
  4. 4. ARM Data – Disaster Recovery Offsite Data backup ARM Data files that are copied into the ORNL HPSS system at ORNL are also copied over to the HPSS system at ANL. The Globus-URL-copy program and the ESnet network between the two labs are utilized for this purpose. ■ Date copying to ANL started: 03-26-2018 ■ Total size transferred: 188.03 TB ■ Total number of files transferred: 3,938,465 4
  5. 5. Data Transfer and Staging to Facilitate Data Science 5
  6. 6. Data Transfer and Staging to Facilitate Data Science 6
  7. 7. Data Pipeline and Software Architecture May 6, 2019 7 Data Processing Storage & Data Model Querying Analytics Scientific Users Data Pipeline Software Architecture Interface Visualization Analytics Output Spark ARM HPC Computing Clusters JupyterLab Relational Database NoSQL Database • Supports fast analysis of voluminous data • Hides architectural complexities • Stage data in HPC • Metadata • Order History • Data from multiple instruments Frontend Analytic Server Backend Dr.Bhargavi Krishna, Yuping Lu, and Dr.Jitu Kumar 7
  8. 8. Data Retrieval, Packaging, and Delivery § Merging § DQR filtering § Conversion Retrieval Future capability Data- streams HPSS Online copy Link to data access Data quality Access to plots DOI based citation guidance Publication request Discovery UI & Web services NetCDF data extractions Data staging order Live Data WS 8
  9. 9. Data Discovery Tool 9
  10. 10. 10 § Based on big data analysis platform (NoSQL) § ARM HPC Clusters for data processing § Provides an interactive web interface for users to find simulations of interest through examination of the LES performance relative to select ARM observations § Allows user to visualize LASSO data bundle diagnostics and skill scores on the fly using plots and tables § Globus as a delivery option Cassandra D3 & NodeJS Spark Data Discovery for LASSO
  11. 11. Questions? 5/6/19 11 ARM Data Services Giri Prakash palanisamyg@ornl.gov

×