Sharing data: Sanger Experiences

•Als ODP, PDF herunterladen•

1 gefällt mir•829 views

Sharing large amounts of data is easier said than done. This talk gives an overview of our experiences doing big-data science over wide-area networks.

Technologie

Data Sharing: Sanger Experiences Guy Coates Wellcome Trust Sanger Institute [email_address]

Cloud Experiments ,[object Object],[object Object],[object Object],[object Object]

First attempt: 14 Mbits/s out of 2Gbit/s link.

Department of Enegy Office of Science. ,[object Object]

Tune your TCP stack. ,[object Object],[object Object]

Cambridge -> EC2 Dublin: 25 Mbytes/s (200 Mbits/s) ,[object Object],[object Object],[object Object],[object Object]

What about the Physicists? ,[object Object]

Dedicated 10GigE networking between CERN and the 10 Tier 1 centres. ,[object Object],[object Object]

Constant monitoring / bandwidth tests to ensure it stays working.

We need a bigger networks: ,[object Object]

Sanger Production Pipeline ,[object Object]

Sequencing data flow Sanger Sequencer Analysis/ alignment Internal repository EBI EGA / SRA (EBI)

Data movement between Sanger/EBI ,[object Object]

Empfohlen

Sanger HPC infrastructure Report (2007)Guy Coates

Cloud ExperiencesGuy Coates

Storage for next-generation sequencingGuy Coates

Next generation genomics: Petascale data in the life sciencesGuy Coates

Next-generation sequencing: Data mangementGuy Coates

Life sciences big data use casesGuy Coates

Future Architectures for genomicsGuy Coates

Managing Genomics Data at the Sanger Instituteinside-BigData.com

Empfohlen

Sanger HPC infrastructure Report (2007)Guy Coates

Cloud ExperiencesGuy Coates

Storage for next-generation sequencingGuy Coates

Next generation genomics: Petascale data in the life sciencesGuy Coates

Next-generation sequencing: Data mangementGuy Coates

Life sciences big data use casesGuy Coates

Future Architectures for genomicsGuy Coates

Managing Genomics Data at the Sanger Instituteinside-BigData.com

Clouds, Grids and DataGuy Coates

Cluster Filesystems and the next 1000 human genomesGuy Coates

How HPC and large-scale data analytics are transforming experimental scienceinside-BigData.com

Challenges and Opportunities of Big Data GenomicsYasin Memari

HDFS IssuesSteve Loughran

A Step to the Clouded Solution of Scalable Clinical Genome Sequencing (BDT308...Amazon Web Services

My other computer_is_a_datacentreSteve Loughran

PUC Masterclass Big DataArjen de Vries

Spark Summit EU talk by Erwin Datema and Roeland van HamSpark Summit

RedisKishor Parkhe

Roots tech 2013 Big Data at Ancestry (3-22-2013) - no animationsWilliam Yetman

Hadoop TechnologyAtul Kushwaha

HadoopZubair Arshad

getFamiliarWithHadoopAmirReza Mohammadi

Hw09 Protein AlignmentCloudera, Inc.

Introduction to Big Data and Science Clouds (Chapter 1, SC 11 Tutorial)Robert Grossman

Hadoop for Bioinformatics: Building a Scalable Variant StoreUri Laserson

The Gordon Data-intensive Supercomputer. Enabling Scientific DiscoveryIntel IT Center

A Survey on Approaches for Frequent Item Set Mining on Apache HadoopIJTET Journal

Module 01 - Understanding Big Data and Hadoop 1.x,2.xNPN Training

Clouds: All fluff and no substance?Guy Coates

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal

Weitere ähnliche Inhalte

Was ist angesagt?

Clouds, Grids and DataGuy Coates

Cluster Filesystems and the next 1000 human genomesGuy Coates

How HPC and large-scale data analytics are transforming experimental scienceinside-BigData.com

Challenges and Opportunities of Big Data GenomicsYasin Memari

HDFS IssuesSteve Loughran

A Step to the Clouded Solution of Scalable Clinical Genome Sequencing (BDT308...Amazon Web Services

My other computer_is_a_datacentreSteve Loughran

PUC Masterclass Big DataArjen de Vries

Spark Summit EU talk by Erwin Datema and Roeland van HamSpark Summit

RedisKishor Parkhe

Roots tech 2013 Big Data at Ancestry (3-22-2013) - no animationsWilliam Yetman

Hadoop TechnologyAtul Kushwaha

HadoopZubair Arshad

getFamiliarWithHadoopAmirReza Mohammadi

Hw09 Protein AlignmentCloudera, Inc.

Introduction to Big Data and Science Clouds (Chapter 1, SC 11 Tutorial)Robert Grossman

Hadoop for Bioinformatics: Building a Scalable Variant StoreUri Laserson

The Gordon Data-intensive Supercomputer. Enabling Scientific DiscoveryIntel IT Center

A Survey on Approaches for Frequent Item Set Mining on Apache HadoopIJTET Journal

Module 01 - Understanding Big Data and Hadoop 1.x,2.xNPN Training

Was ist angesagt? (20)

Clouds, Grids and Data

Cluster Filesystems and the next 1000 human genomes

How HPC and large-scale data analytics are transforming experimental science

Challenges and Opportunities of Big Data Genomics

HDFS Issues

A Step to the Clouded Solution of Scalable Clinical Genome Sequencing (BDT308...

My other computer_is_a_datacentre

PUC Masterclass Big Data

Spark Summit EU talk by Erwin Datema and Roeland van Ham

Redis

Roots tech 2013 Big Data at Ancestry (3-22-2013) - no animations

Hadoop Technology

Hadoop

getFamiliarWithHadoop

Hw09 Protein Alignment

Introduction to Big Data and Science Clouds (Chapter 1, SC 11 Tutorial)

Hadoop for Bioinformatics: Building a Scalable Variant Store

The Gordon Data-intensive Supercomputer. Enabling Scientific Discovery

A Survey on Approaches for Frequent Item Set Mining on Apache Hadoop

Module 01 - Understanding Big Data and Hadoop 1.x,2.x

Ähnlich wie Sharing data: Sanger Experiences

Clouds: All fluff and no substance?Guy Coates

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal

Hadoop and Voldemort @ LinkedInHadoop User Group

Network Engineering for High Speed Data SharingGlobus

Hadoop World 2011: Hadoop Network and Compute Architecture Considerations - J...Cloudera, Inc.

Datastage parallell jobs vs datastage server jobsshanker_uma

Vikram Andem Big Data Strategy @ IATA Technology Roadmap IT Strategy Group

Modernizing upstream workflows with aws storage - john malloryAmazon Web Services

Big Data - Need of Converged Data PlatformGeekNightHyderabad

Enabling big data & AI workloads on the object store at DBS Alluxio, Inc.

Big Data, Mob Scale.darach

Big Events, Mob Scale - Darach Ennis (Push Technology)jaxLondonConference

Lessons from lhcdrsm79

Inroduction to Big DataOmnia Safaan

Seminar Presentation HadoopVarun Narang

Coates bosc2010 clouds-fluff-and-no-substanceBOSC 2010

Data Mobility ExhibitionGlobus

Infrastructure Strategies 2007Dr. Jimmy Schwarzkopf

Google Cloud Computing on Google Developer 2008 Dayprogrammermag

Yaron Haviv, Iguaz.io - OpenStack and BigData - OpenStack Israel 2015Cloud Native Day Tel Aviv

Ähnlich wie Sharing data: Sanger Experiences (20)

Clouds: All fluff and no substance?

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Hadoop and Voldemort @ LinkedIn

Network Engineering for High Speed Data Sharing

Hadoop World 2011: Hadoop Network and Compute Architecture Considerations - J...

Datastage parallell jobs vs datastage server jobs

Vikram Andem Big Data Strategy @ IATA Technology Roadmap

Modernizing upstream workflows with aws storage - john mallory

Big Data - Need of Converged Data Platform

Enabling big data & AI workloads on the object store at DBS

Big Data, Mob Scale.

Big Events, Mob Scale - Darach Ennis (Push Technology)

Lessons from lhc

Inroduction to Big Data

Seminar Presentation Hadoop

Coates bosc2010 clouds-fluff-and-no-substance

Data Mobility Exhibition

Infrastructure Strategies 2007

Google Cloud Computing on Google Developer 2008 Day

Yaron Haviv, Iguaz.io - OpenStack and BigData - OpenStack Israel 2015

Kürzlich hochgeladen

🐬 The future of MySQL is Postgres 🐘RTylerCroy

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech

How to convert PDF to text with Nanonetsnaman860154

Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Slack Application Development 101 Slidespraypatel2

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies

Real Time Object Detection Using Open CVKhem

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1

Histor y of HAM Radio presentation slidevu2urc

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

Kürzlich hochgeladen (20)

🐬 The future of MySQL is Postgres 🐘

Data Cloud, More than a CDP by Matt Robison

Driving Behavioral Change for Information Management through Data-Driven Gree...

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

How to Troubleshoot Apps for the Modern Connected Worker

Advantages of Hiring UIUX Design Service Providers for Your Business

How to convert PDF to text with Nanonets

Powerful Google developer tools for immediate impact! (2023-24 C)

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Slack Application Development 101 Slides

The Codex of Business Writing Software for Real-World Solutions 2.pptx

GenCyber Cyber Security Day Presentation

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Real Time Object Detection Using Open CV

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Boost Fertility New Invention Ups Success Rates.pdf

Histor y of HAM Radio presentation slide

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Sharing data: Sanger Experiences

1. Data Sharing: Sanger Experiences Guy Coates Wellcome Trust Sanger Institute [email_address]

6. First attempt: 14 Mbits/s out of 2Gbit/s link.

10.

11.

12.

13.

14. Constant monitoring / bandwidth tests to ensure it stays working.

15. See HEPIX talks for gory details.

16.

17. Is it the only thing we need to do?

18.

19. Sequencing data flow Sanger Sequencer Analysis/ alignment Internal repository EBI EGA / SRA (EBI)

20.

21. 10Gbit/s (1.2 Gbyte/s) link between EBI and Sanger.

22. We share a data-centre.

23. Physically near, so we do not need to worry about WAN issues.

24.

25.

26.

27.

28.

29. Do we need to move the data?

30. Centralised data Data Sequencing Centre + DCC Sequencing centre Sequencing centre Sequencing centre Sequencing centre

31.

32.

33. 3 month lead time.

34. ~$1.5M capex.

35. Federation: A Better way: Collaborations are short term: 18 months-3 years. Sequencing centre Sequencing centre Sequencing centre Sequencing centre Federated access

36. Federation software: Unstructured data (flat files) Data size per Genome Structured data (databases) BioMart IRODS (data grid software) Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)

37.

38. Put VMs on compute that is “attached” to the data. Data CPU CPU CPU CPU Data CPU CPU CPU CPU VM

39.

40. We need cross site teams who can troubleshoot all potential trouble spots.

41. Teams need application & systems people.

42.

43. Gen-Tao Chiang

44. Pete Clapham

45. Simon Kelley