SlideShare ist ein Scribd-Unternehmen logo
1 von 123
Downloaden Sie, um offline zu lesen
Genomes on Rails
  has_many :sequences
Hello
➊
Previously

    ➋
Production

    ➌
 Process
➊ Previously
The human genome


   15 years to decode
     3 billion letters
$3 billion
$3 billion ++
Race for the prize
Open data
Open source
Perl
Lots of Perl
Lots of Perl
 ~4500 modules
Onwards!
40 species
Map evolutionary
     space
Compare genomes
compare species
Compare genomes
compare species
Compare genomes

 compa re indi viduals
More Perl
~1500 modules
Quantum leap!
1000 personal
  genomes
beyond 23andme
1000 personal
  genomes
Hypertension
Diabetes
Coronary heart disease
Bipolar disorder
Malaria
➋ Production
Register projects


Register samples


  Sample prep


  Sequencing


    Analysis
Change!
Flexible data capture
Virtual fields
Sample


   Name
  Organism
Concentration
class Sample < ActiveRecord::Base
  has_many :descriptors
  has_many :descriptor_values
end
Key value pairs
Faster than you’d think
Change!
V1               V2


   Sample          Sample


   Name             Name
  Organism        Organism
Concentration   Concentration
                   Origin
                Quality metric
Rationalize!
V1               V2


   Sample          Sample


   Name             Name
  Organism        Organism
Concentration   Concentration
                   Origin
                Quality metric
Mapping!
V1               V3


   Sample          Sample


   Name             Name
  Organism         Species
Concentration   Concentration
   Origin          Origin
                Quality metric
Pipeline management
Workflow

 Task 1        Task 2        Task 3

  Name          Name         Name
Operator     Serial number   Passed
Instrument        Kit
Throughput!
320Tb 450 CPU
320Tb 450 CPU   Archive
75   Tb
pilot study!
Multiple apps
Multiple instances
Loosely coupled
Loose coupling is hard
Deployment
Maintenance
Monitoring
Hard to maintain
  separation
Support novel science
Single code base
nginx reverse proxy
fairnginx
Mongrel
Fast deployment
Automate everything
Play well with others!



 Interoperability!
Legacy databases
RESTful services
Generate API stubs
SCALE!
Trillionics
2   X
150Tb per week
Over 6 months
More hardware
400 additional nodes
additional 360 Tb
Towards a
Virtual Institute
Lots of data
Lots of data, lots of
      people
Lots of data, lots of
people, lots of compute
Lots of data, lots of
people, lots of compute,
      lots of uses
Lots of data, lots of
 people, lots of compute,
lots of uses, lots and lots
   and lots and lots...
➌ Process
Concept Requirements Development   Product
takes too lon
                                   g
Concept Requirements Development       Product
takes too lon
                                    g
Concept Requirements Development        Product




       the se change
Plan                 Development




 REVIEW        Concept




What we need              Get ready
Focused
Project owner is key
Weekly releases
More flexible
Less time
Better transparency
Less software
Sequencing informatics
Thank you
GREENISGOOD.CO.UK

Weitere ähnliche Inhalte

Ähnlich wie Genomes On Rails

Services For Science April 2009
Services For Science April 2009Services For Science April 2009
Services For Science April 2009
Ian Foster
 
Utility HPC: Right Systems, Right Scale, Right Science
Utility HPC: Right Systems, Right Scale, Right ScienceUtility HPC: Right Systems, Right Scale, Right Science
Utility HPC: Right Systems, Right Scale, Right Science
Chef Software, Inc.
 

Ähnlich wie Genomes On Rails (20)

Introducing VSWarehouse - A Scalable Genetic Data Warehouse for VarSeq
Introducing VSWarehouse - A Scalable Genetic Data Warehouse for VarSeqIntroducing VSWarehouse - A Scalable Genetic Data Warehouse for VarSeq
Introducing VSWarehouse - A Scalable Genetic Data Warehouse for VarSeq
 
Rare Variant Analysis Workflows: Analyzing NGS Data in Large Cohorts
Rare Variant Analysis Workflows: Analyzing NGS Data in Large CohortsRare Variant Analysis Workflows: Analyzing NGS Data in Large Cohorts
Rare Variant Analysis Workflows: Analyzing NGS Data in Large Cohorts
 
Cassava genome hub
Cassava genome hubCassava genome hub
Cassava genome hub
 
Services For Science April 2009
Services For Science April 2009Services For Science April 2009
Services For Science April 2009
 
2014 nicta-reproducibility
2014 nicta-reproducibility2014 nicta-reproducibility
2014 nicta-reproducibility
 
BioIT Europe 2010 - BioCatalogue
BioIT Europe 2010 - BioCatalogueBioIT Europe 2010 - BioCatalogue
BioIT Europe 2010 - BioCatalogue
 
Sharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reportsSharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reports
 
DCC Keynote 2007
DCC Keynote 2007DCC Keynote 2007
DCC Keynote 2007
 
Next generation sequencing & microarray-- Genotypic Technology
Next generation sequencing & microarray-- Genotypic TechnologyNext generation sequencing & microarray-- Genotypic Technology
Next generation sequencing & microarray-- Genotypic Technology
 
Ngs part i 2013
Ngs part i 2013Ngs part i 2013
Ngs part i 2013
 
So you want to do a: RNAseq experiment, Differential Gene Expression Analysis
So you want to do a: RNAseq experiment, Differential Gene Expression AnalysisSo you want to do a: RNAseq experiment, Differential Gene Expression Analysis
So you want to do a: RNAseq experiment, Differential Gene Expression Analysis
 
Utility HPC: Right Systems, Right Scale, Right Science
Utility HPC: Right Systems, Right Scale, Right ScienceUtility HPC: Right Systems, Right Scale, Right Science
Utility HPC: Right Systems, Right Scale, Right Science
 
Dr Justin Schonfeld - Bioinformatics Applications
Dr Justin Schonfeld - Bioinformatics ApplicationsDr Justin Schonfeld - Bioinformatics Applications
Dr Justin Schonfeld - Bioinformatics Applications
 
Zero ETL analytics with LLAP in Azure HDInsight
Zero ETL analytics with LLAP in Azure HDInsightZero ETL analytics with LLAP in Azure HDInsight
Zero ETL analytics with LLAP in Azure HDInsight
 
BioThings API: Building a FAIR API Ecosystem for Biomedical Knowledge
BioThings API: Building a FAIR API Ecosystem for Biomedical KnowledgeBioThings API: Building a FAIR API Ecosystem for Biomedical Knowledge
BioThings API: Building a FAIR API Ecosystem for Biomedical Knowledge
 
Solr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance studySolr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance study
 
Making NGS Data Analysis Clinically Practical: Repeatable and Time-Effective ...
Making NGS Data Analysis Clinically Practical: Repeatable and Time-Effective ...Making NGS Data Analysis Clinically Practical: Repeatable and Time-Effective ...
Making NGS Data Analysis Clinically Practical: Repeatable and Time-Effective ...
 
Beyond Transparency: Success & Lessons From tambisBoston2003
Beyond Transparency: Success & Lessons From tambisBoston2003Beyond Transparency: Success & Lessons From tambisBoston2003
Beyond Transparency: Success & Lessons From tambisBoston2003
 
iMicrobe and iVirus: Extending the iPlant cyberinfrastructure from plants to ...
iMicrobe and iVirus: Extending the iPlant cyberinfrastructure from plants to ...iMicrobe and iVirus: Extending the iPlant cyberinfrastructure from plants to ...
iMicrobe and iVirus: Extending the iPlant cyberinfrastructure from plants to ...
 
Prometheus lightning talk (Devops Dublin March 2015)
Prometheus lightning talk (Devops Dublin March 2015)Prometheus lightning talk (Devops Dublin March 2015)
Prometheus lightning talk (Devops Dublin March 2015)
 

Mehr von Matt Wood

Mehr von Matt Wood (12)

Genomics in the Cloud
Genomics in the CloudGenomics in the Cloud
Genomics in the Cloud
 
How to make Friendfeeds and influence people
How to make Friendfeeds and influence peopleHow to make Friendfeeds and influence people
How to make Friendfeeds and influence people
 
Genomes On Rails
Genomes On RailsGenomes On Rails
Genomes On Rails
 
Into The Wonderful
Into The WonderfulInto The Wonderful
Into The Wonderful
 
Extreme Informatics
Extreme InformaticsExtreme Informatics
Extreme Informatics
 
What can Bioinformaticians learn from YouTube?
What can Bioinformaticians learn from YouTube?What can Bioinformaticians learn from YouTube?
What can Bioinformaticians learn from YouTube?
 
The A to Z of developing for the web
The A to Z of developing for the webThe A to Z of developing for the web
The A to Z of developing for the web
 
Introduction to Scrum
Introduction to ScrumIntroduction to Scrum
Introduction to Scrum
 
30 Minutes With Rails
30 Minutes With Rails30 Minutes With Rails
30 Minutes With Rails
 
Subversion Best Practices
Subversion Best PracticesSubversion Best Practices
Subversion Best Practices
 
Lucene
LuceneLucene
Lucene
 
Introduction to the Semantic Web
Introduction to the Semantic WebIntroduction to the Semantic Web
Introduction to the Semantic Web
 

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 

Genomes On Rails