Petascale Genomics (Strata Singapore 20151203)
âąAls PPTX, PDF herunterladenâą
6 gefĂ€llt mirâą1,512 views
Scalable genomics with Hadoop and Spark
Empfohlen
Weitere Àhnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (9)
Ăhnlich wie Petascale Genomics (Strata Singapore 20151203)
Ăhnlich wie Petascale Genomics (Strata Singapore 20151203) (20)
Mehr von Uri Laserson
Mehr von Uri Laserson (6)
KĂŒrzlich hochgeladen
KĂŒrzlich hochgeladen (20)
Petascale Genomics (Strata Singapore 20151203)
- 1. 1© Cloudera, Inc. All rights reserved. Scaling Up Genomics with Hadoop and Spark Uri Laserson | @laserson | 14 November 2015 Petascale Genomics
- 2. 2© Cloudera, Inc. All rights reserved. We come in peace. Pioneer plaque
- 3. 3© Cloudera, Inc. All rights reserved. What is genomics?
- 4. 4© Cloudera, Inc. All rights reserved. Organism
- 5. 5© Cloudera, Inc. All rights reserved. Organism Cell
- 6. 6© Cloudera, Inc. All rights reserved. Organism Cell Genome
- 7. 7© Cloudera, Inc. All rights reserved.
- 8. 8© Cloudera, Inc. All rights reserved.
- 9. 9© Cloudera, Inc. All rights reserved. Reference chromosome
- 10. 10© Cloudera, Inc. All rights reserved. Reference chromosome Location
- 11. 11© Cloudera, Inc. All rights reserved. â⊠decoding the Book of Lifeâ
- 12. 12© Cloudera, Inc. All rights reserved. ...atatggaaccaaaaaagagcccgcatcgccaaggcaatcctaagccaaaagaacaaagctggaggcatcacactacctgacttcaaactatactaca agcctacagtaaccaaaacagcatggtactggtaccaaaacagagatatagatcaatggaacagaacagagccctcagaaataacgccgcatatctacaa ctatctgatctttgacgaacctgagaaaaacaagcaatggggaaaggattccctatttaataaatggtgctgggaaaactggctagccatatgtagaaag ctgaaactggatcccttccttacaccttatacaaaaatcaattcaagatggattaaagacttaaacgttagacctaaaaccataaaaaccctagaagaaa acctaggcagtaccattcaggacataggcatgggcaaggacttcatgtccaaaacaccaaaagcaatggcaacaaaagacaaaattgacaaatgggatct aattaaactaaagagcttctgcacagcaaaagaaactaccatcagagtgaacaggaaacctacaaaatgggagaaaattttcgcaacctactcatctgac aaagggctaatatccagaatctacaatgaactcaaacaaatttacaagaaaaaaacaaacaaccccatcaaaaagtgggcaaaggacatgaacagacact tctcaaatgaagacatttatgcagccaaaaaacacatgaaaaaatgctcatcatcactggccatcagagaaatgcaaatcaaaaccacaatgagatacca tctcacaccagttagaatggcaatcattaaaaagtcaggaaacaacaggtgctggagaggatgtggagaaataggaacacttttacactgttggtgggac tgtaaactagttcaaccattgtggaagtcagtgtggtgattcctcagggatctagaactagaaataccatttgacccagccatcccattactgggtatat acccaaaggactataaatcatgctgctataaagacacatgcacacgtatgtttattgcggcattattcacaatagcaaagacttggaaccaacccaaatg tccaacaatgataaactggattaagaaaatgtggcacatatacaccatggaatactctgcagccataaaaaaggatgagttcatgtcctttgtagggaca tggatgaaattggaaatcatcattctcagtaaactatcgcaagaataaaaaaccaaacaccgcatattctcactcataggtgggaattgaacaatgagat cacatggacacaggaagaggaatatcacactctggggactgtggtggggtggggggaggggggagggatagcattgggagatatacctaatgctagatga cgagttagtgggtgcagcgcaccagcatggcacatgtatacatatgtaactaacctgcacattgtgcacatgtaccctaaaacttaaagtataataaaaa aataaaaaaaataaagtgtgtgtgtgtatgactttaattaacttgatcacccacacacacacaaacactgaccaaaattaatatcaagtcaggtctgtct gaatgtaaagccaacagcaaacatccctctctccaaatggaaaagaaacagggggttatgggcagctacactgctaaatgttaaaactttatttttaaat gtggccataaaaatcactaaataaaattgataatatatgtttttgatgaataaattttatatatgtctacactggaaactatatagcaataaaaactaac catgtacaactaaactcataaatttcataaacataataagtaaaagaagccagacaaaaagtagtgtatactgttaaattccatttatataaaagttcaa aaaagccaaaaagaaactatgctgttaaaagtaaggattatagttactattcagggaagagagtagtggctggaaagaaacataaagggggtctctgaag tggaataatgttctgttttttgatctgggtattagggtgtttaatttcggaaaattattttatctttatacttattgtattattgattttttgcttaaca aattactcaaaacttagaggtttaaaaaaaattaattattgtattaatttctctgggccaggaattggagagagcttagctgggtagttctggttcaaaa tttctcatgagattaccgtcaagctgttggagggggctgcatcatctgaaggcttgaccgaggctagaggatctactttcaagatggcccactcacatgg ctgttggcaagaagtttcagtttctcactagcttctagcaggaggccataatttctcaccacatagatctctctatagggctactcgagtgtcctcacag caaggtagctggctttcttcagagccaagtgactcaaaggcaaagaggaagtcactatgccatttatgacctagttttggaactcacactttgttccgaa ttgaccttccatcactttctagtcattaggatttaagtcactaactctgatccatagtcaaggggagtaaaatttggctttattgttggaggatggagta gcaaagaatttgttgacacattttaaaactaccatacttaaacagttcatttttctgaatatgcttcaattagaagttaaaatgatgcaattttaaaaca ttgtttcaaatgaacactgttagggagagaagtgcttcttctccatatctaatgtttcttccatatttagggagttccattagtttaacactttaag...
- 13. 13© Cloudera, Inc. All rights reserved.
- 14. 14© Cloudera, Inc. All rights reserved.
- 15. 15© Cloudera, Inc. All rights reserved.
- 16. 16© Cloudera, Inc. All rights reserved.
- 17. 17© Cloudera, Inc. All rights reserved.
- 18. 18© Cloudera, Inc. All rights reserved. >read1 TTGGACATTTCGGGGTCTCAGATT >read2 AATGTTGTTAGAGATCCGGGATTT >read3 GGATTCCCCGCCGTTTGAGAGCCT >read4 AGGTTGGTACCGCGAAAAGCGCAT
- 19. 19© Cloudera, Inc. All rights reserved. >read1 TTGGACATTTCGGGGTCTCAGATT >read2 AATGTTGTTAGAGATCCGGGATTT >read3 GGATTCCCCGCCGTTTGAGAGCCT >read4 AGGTTGGTACCGCGAAAAGCGCAT Bioinformatics!
- 20. 20© Cloudera, Inc. All rights reserved. >read1 TTGGACATTTCGGGGTCTCAGATT >read2 AATGTTGTTAGAGATCCGGGATTT >read3 GGATTCCCCGCCGTTTGAGAGCCT >read4 AGGTTGGTACCGCGAAAAGCGCAT Bioinformatics!
- 21. 21© Cloudera, Inc. All rights reserved. Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Pipelines!
- 22. 22© Cloudera, Inc. All rights reserved. ##fileformat=VCFv4.1 ##fileDate=20090805 ##source=myImputationProgramV3.1 ##reference=file:///seq/references/1000GenomesPilot-NCBI36.fasta ##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x> ##phasing=partial ##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of Samples With Data"> ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth"> ##INFO=<ID=AF,Number=A,Type=Float,Description="Allele Frequency"> ##INFO=<ID=AA,Number=1,Type=String,Description="Ancestral Allele"> ##INFO=<ID=DB,Number=0,Type=Flag,Description="dbSNP membership, build 129"> ##INFO=<ID=H2,Number=0,Type=Flag,Description="HapMap2 membership"> ##FILTER=<ID=q10,Description="Quality below 10"> ##FILTER=<ID=s50,Description="Less than 50% of samples have data"> ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype"> ##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality"> ##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth"> ##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality"> #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT NA00001 NA00002 NA00003 20 14370 rs6054257 G A 29 PASS NS=3;DP=14;AF=0.5;DB;H2 GT:GQ:DP:HQ 0|0:48:1:51,51 1|0:48:8:51,51 1/1:43:5:.,. 20 17330 . T A 3 q10 NS=3;DP=11;AF=0.017 GT:GQ:DP:HQ 0|0:49:3:58,50 0|1:3:5:65,3 0/0:41:3 20 1110696 rs6040355 A G,T 67 PASS NS=2;DP=10;AF=0.333,0.667 GT:GQ:DP:HQ 1|2:21:6:23,27 2|1:2:0:18,2 2/2:35:4 20 1230237 . T . 47 PASS NS=3;DP=13;AA=T GT:GQ:DP:HQ 0|0:54:7:56,60 0|0:48:4:51,51 0/0:61:2 Compressed text files (non-splittable) Semi-structured Poorly specified
- 23. 23© Cloudera, Inc. All rights reserved. ##fileformat=VCFv4.1 ##fileDate=20090805 ##source=myImputationProgramV3.1 ##reference=file:///seq/references/1000GenomesPilot-NCBI36.fasta ##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x> ##phasing=partial ##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of Samples With Data"> ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth"> ##INFO=<ID=AF,Number=A,Type=Float,Description="Allele Frequency"> ##INFO=<ID=AA,Number=1,Type=String,Description="Ancestral Allele"> ##INFO=<ID=DB,Number=0,Type=Flag,Description="dbSNP membership, build 129"> ##INFO=<ID=H2,Number=0,Type=Flag,Description="HapMap2 membership"> ##FILTER=<ID=q10,Description="Quality below 10"> ##FILTER=<ID=s50,Description="Less than 50% of samples have data"> ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype"> ##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality"> ##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth"> ##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality"> #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT NA00001 NA00002 NA00003 20 14370 rs6054257 G A 29 PASS NS=3;DP=14;AF=0.5;DB;H2 GT:GQ:DP:HQ 0|0:48:1:51,51 1|0:48:8:51,51 1/1:43:5:.,. 20 17330 . T A 3 q10 NS=3;DP=11;AF=0.017 GT:GQ:DP:HQ 0|0:49:3:58,50 0|1:3:5:65,3 0/0:41:3 20 1110696 rs6040355 A G,T 67 PASS NS=2;DP=10;AF=0.333,0.667 GT:GQ:DP:HQ 1|2:21:6:23,27 2|1:2:0:18,2 2/2:35:4 20 1230237 . T . 47 PASS NS=3;DP=13;AA=T GT:GQ:DP:HQ 0|0:54:7:56,60 0|0:48:4:51,51 0/0:61:2 Compressed text files (non-splittable) Semi-structured Poorly specified Global sort order
- 24. 24© Cloudera, Inc. All rights reserved. C HPC (scheduler) POSIX filesystem Java HPC (Queue) POSIX filesystem C++ Single-node SQLite Itâs file formats all the way down!
- 25. 25© Cloudera, Inc. All rights reserved. Dedup
- 26. 26© Cloudera, Inc. All rights reserved. /** * Main work method. Reads the BAM file once and collects sorted information about * the 5' ends of both ends of each read (or just one end in the case of pairs). * Then makes a pass through those determining duplicates before re-reading the * input file and writing it out with duplication flags set correctly. */ protected int doWork() { // build some data structures buildSortedReadEndLists(useBarcodes); generateDuplicateIndexes(useBarcodes); final SAMFileWriter out = new SAMFileWriterFactory().makeSAMOrBAMWriter(outputHeader, true, OUTPUT); final CloseableIterator<SAMRecord> iterator = headerAndIterator.iterator; while (iterator.hasNext()) { final SAMRecord rec = iterator.next(); if (!rec.isSecondaryOrSupplementary()) { if (recordInFileIndex == nextDuplicateIndex) { rec.setDuplicateReadFlag(true); // Now try and figure out the next duplicate index if (this.duplicateIndexes.hasNext()) { nextDuplicateIndex = this.duplicateIndexes.next(); } else { // Only happens once we've marked all the duplicates nextDuplicateIndex = -1; } } else { Method Code
- 27. 27© Cloudera, Inc. All rights reserved. /** * Main work method. Reads the BAM file once and collects sorted information about * the 5' ends of both ends of each read (or just one end in the case of pairs). * Then makes a pass through those determining duplicates before re-reading the * input file and writing it out with duplication flags set correctly. */ protected int doWork() { // build some data structures buildSortedReadEndLists(useBarcodes); generateDuplicateIndexes(useBarcodes); final SAMFileWriter out = new SAMFileWriterFactory().makeSAMOrBAMWriter(outputHeader, true, OUTPUT); final CloseableIterator<SAMRecord> iterator = headerAndIterator.iterator; while (iterator.hasNext()) { final SAMRecord rec = iterator.next(); if (!rec.isSecondaryOrSupplementary()) { if (recordInFileIndex == nextDuplicateIndex) { rec.setDuplicateReadFlag(true); // Now try and figure out the next duplicate index if (this.duplicateIndexes.hasNext()) { nextDuplicateIndex = this.duplicateIndexes.next(); } else { // Only happens once we've marked all the duplicates nextDuplicateIndex = -1; } } else { Method Code
- 28. 28© Cloudera, Inc. All rights reserved. @Option(shortName = "MAX_FILE_HANDLES", doc = "Maximum number of file handles to keep open when spilling " + "read ends to disk. Set this number a little lower than the " + "per-process maximum number of file that may be open. This " + "number can be found by executing the 'ulimit -n' command on " + "a Unix system.") public int MAX_FILE_HANDLES_FOR_READ_ENDS_MAP = 8000;
- 29. 29© Cloudera, Inc. All rights reserved. @Option(shortName = "MAX_FILE_HANDLES", doc = "Maximum number of file handles to keep open when spilling " + "read ends to disk. Set this number a little lower than the " + "per-process maximum number of file that may be open. This " + "number can be found by executing the 'ulimit -n' command on " + "a Unix system.") public int MAX_FILE_HANDLES_FOR_READ_ENDS_MAP = 8000; Dedup Method/Algo Code Platform
- 30. 30© Cloudera, Inc. All rights reserved. Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation
- 31. 31© Cloudera, Inc. All rights reserved. Itâs pipelines all the way down! Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation
- 32. 32© Cloudera, Inc. All rights reserved. Itâs pipelines all the way down! Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 1 Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 2 Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 3
- 33. 33© Cloudera, Inc. All rights reserved. Manually running pipelines on HPC $ bsub âq shared_12h python split_genotypes.py $ bsub âq shared_12h âR mem=4g python query_agg.py genotypes_1.vcf agg1.csv $ bsub âq shared_12h âR mem=4g python query_agg.py genotypes_2.vcf agg2.csv $ bsub âq shared_12h âR mem=4g python query_agg.py genotypes_3.vcf agg3.csv $ bsub âq shared_12h âR mem=4g python query_agg.py genotypes_4.vcf agg4.csv $ bsub âq shared_12h python merge_maf.py
- 34. 34© Cloudera, Inc. All rights reserved.
- 35. 35© Cloudera, Inc. All rights reserved. Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Alignment Dedup Recalibrate QC/Filter Alignment Dedup Recalibrate QC/Filter
- 36. 36© Cloudera, Inc. All rights reserved. Node 1 Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 2 Node 3 Alignment Dedup Recalibrate QC/Filter Alignment Dedup Recalibrate QC/Filter Node 4
- 37. 37© Cloudera, Inc. All rights reserved. Node 1 Alignment Dedup QC/Filter Variant Calling Variant Annotation Node 2 Node 3 Alignment Dedup QC/Filter Alignment Dedup QC/Filter Node 4 Recalibrate
- 38. 38© Cloudera, Inc. All rights reserved. Why Are We Still Defining File Formats By Hand? ⹠Instead of defining custom file formats for each data type and access pattern⊠⹠Parquet creates a compressed format for each Avro-defined data model ⹠Improvements over existing formats ⹠~20% for BAM ⹠~90% for VCF
- 39. 39© Cloudera, Inc. All rights reserved. YARN-managed Hadoop cluster Spark executors đ=1 đ đ đ(đđđ|đđđ, đđ) đ=1 đ đ đ(đđđ|đđđ, đđ) đ=1 đ đ đ(đđđ|đđđ, đđ)Partial sums đ=1 đ đ=1 đ đ đ(đđđ|đđđ, đđ) Driver Application code ContEst Algorithm
- 40. 40© Cloudera, Inc. All rights reserved. Hadoop provides layered abstractions for data processing HDFS (scalable, distributed storage) YARN (resource management) MapReduce Impala (SQL) Solr (search) Spark ADAMquince guacamole ⊠bdg-formats(Avro/Parquet)
- 41. 41© Cloudera, Inc. All rights reserved. ⹠Hosted at Berkeley and the AMPLab ⹠Apache 2 License ⹠Contributors from both research and commercial organizations ⹠Core spatial primitives, variant calling ⹠Avro and Parquet for data models and file formats Spark + Genomics = ADAM
- 42. 42© Cloudera, Inc. All rights reserved. Core Genomics Primitives: Spatial Join
- 43. 43© Cloudera, Inc. All rights reserved. Executing query in Hadoop: interactive Spark shell (ADAM) def inDbSnp(g: Genotype): Boolean = true or false def isDeleterious(g: Genotype): Boolean = g.getPolyPhen val samples = sc.textFile("path/to/samples").map(parseJson(_)).collect() val dbsnp = sc.textFile("path/to/dbSNP").map(_.split(",")).collect() val dnaseRDD = sc.adamBEDFeatureLoad("path/to/dnaseâ) val genotypesRDD = sc.adamLoad("path/to/genotypes") val filteredRDD = genotypesRDD .filter(!inDbSnp(_)) .filter(isDeleterious(_)) .filter(isFramingham(_)) val joinedRDD = RegionJoin.partitionAndJoin(sc, filteredRDD, dnaseRDD) val maf = joinedRDD .keyBy(x => (x.getVariant, getPopulation(x))) .groupByKey() .map(computeMAF(_)) maf.saveAsNewAPIHadoopFile("path/to/output") apply predicates load data join data group-by aggregate (MAF) persist data
- 44. 44© Cloudera, Inc. All rights reserved. Executing query in Hadoop: distributed SQL SELECT g.chr, g.pos, g.ref, g.alt, s.pop, MAF(g.call) FROM genotypes g INNER JOIN samples s ON g.sample = s.sample INNER JOIN dnase d ON g.chr = d.chr AND g.pos >= d.start AND g.pos < d.end LEFT OUTER JOIN dbsnp p ON g.chr = p.chr AND g.pos = p.pos AND g.ref = p.ref AND g.alt = p.alt WHERE s.study = "framingham" p.pos IS NULL AND g.polyphen IN ( "possibly damaging", "probably damaging" ) GROUP BY g.chr, g.pos, g.ref, g.alt, s.pop apply predicates âloadâ and join data group-by aggregate (UDAF)
- 45. 45© Cloudera, Inc. All rights reserved. ADAM preliminary performance
- 46. 46© Cloudera, Inc. All rights reserved. 1. Somebody will build on your code 2. You should have assembled a team to build your software 3. If you choose the right license, more people will use and build on your software. 4. Making software free for commercial use shows you are not against companies. 5. You should maintain your software indefinitely 6. Your âstable URLâ can exist forever 7. You should make your software âidiot proofâ 8. You used the right programming language for the task. Lior Pachter https://liorpachter.wordpress.com/2015/07/10/the-myths-of-bioinformatics-software/ âMyths of Bioinformatics Softwareâ
- 47. 47© Cloudera, Inc. All rights reserved.
- 48. 48© Cloudera, Inc. All rights reserved. Acknowledgements UCBerkeley Matt Massie Frank Nothaft Michael Heuer Tamr Timothy Danford MSSM Jeff Hammerbacher Ryan Williams Cloudera Tom White Sandy Ryza
- 49. 49© Cloudera, Inc. All rights reserved. Thank you @laserson laserson@cloudera.com
Hinweis der Redaktion
- Before we dive in, let me ask a couple of questions: Biologists? Spark experts? There are always at least three different constituencies in the room: * biologists * programmers * someone thinking about how to build a business around this Gonna tell you a lot of lies today. Wonât satisfy everyone. Where I skip over the truth, maybe there will be at least a breadcrumb of truth left over. This will not be a very technical talk.
- Scared/pissed off some bio people in the past. Bioinformatics is a field with a long history, thirty or more years as a separate discipline. At the same time, the fundamental technology is changing. So if I talk about âproblems of bioinformaticsâ today, itâs OK because WE COME IN PEACE! Bioinformatics software development has been *remarkably* effective, for decades. If there are problems to be solved, these are the result of new technologies, new ambitions of scale.
- What even is genomics? Who here has heard the terms âchromosomeâ and âgeneâ before, and could explain the difference? So before we dive into the main part of the talk, Iâm going to spend a few minutes discussing some of the basic biological concepts.
- Fundamentally, weâre interested in studying individuals (and populations of individuals) [ADVANCE] But each individual is actually a population: of cells [ADVANCE] But each of those cells has, ideally, an identical genome. The genome is a collection of 23 linear molecules. These are called âpolymers,â theyâre built (like Legos) out of a small number of repeated interlocking parts â these are the A, T, G, and C youâve probably heard about. The content of the genome is determined by the linear order in which these letters are arranged. (Linear is important!)
- Fundamentally, weâre interested in studying individuals (and populations of individuals) [ADVANCE] But each individual is actually a population: of cells [ADVANCE] But each of those cells has, ideally, an identical genome. The genome is a collection of 23 linear molecules. These are called âpolymers,â theyâre built (like Legos) out of a small number of repeated interlocking parts â these are the A, T, G, and C youâve probably heard about. The content of the genome is determined by the linear order in which these letters are arranged. (Linear is important!)
- Fundamentally, weâre interested in studying individuals (and populations of individuals) [ADVANCE] But each individual is actually a population: of cells [ADVANCE] But each of those cells has, ideally, an identical genome. The genome is a collection of 23 linear molecules. These are called âpolymers,â theyâre built (like Legos) out of a small number of repeated interlocking parts â these are the A, T, G, and C youâve probably heard about. The content of the genome is determined by the linear order in which these letters are arranged. (Linear is important!)
- Without losing much, assume that our genomes are contained on just a single chromosome. Now, not only do all the cells in your body have identical genomes⊠[ADVANCE] But individual humans have genomes that are very similar to each other. So similar that I can define âthe sameâ chromosome between individuals⊠and that means⊠[ADVANCE] That we can define a âbaseâ or a âreferenceâ chromosome. Now that there is a reference that all of us adhere to⊠[ADVANCE] We can define a concept of âlocationâ across chromosomes. This is possibly the most important concept in genome informatics, the idea that DNA defines a common linear coordinate system. This also means that we can talk about differences between individuals in terms of diffs to a common reference genome. But where does this reference genome come from?
- Without losing much, assume that our genomes are contained on just a single chromosome. Now, not only do all the cells in your body have identical genomes⊠[ADVANCE] But individual humans have genomes that are very similar to each other. So similar that I can define âthe sameâ chromosome between individuals⊠and that means⊠[ADVANCE] That we can define a âbaseâ or a âreferenceâ chromosome. Now that there is a reference that all of us adhere to⊠[ADVANCE] We can define a concept of âlocationâ across chromosomes. This is possibly the most important concept in genome informatics, the idea that DNA defines a common linear coordinate system. This also means that we can talk about differences between individuals in terms of diffs to a common reference genome. But where does this reference genome come from?
- Without losing much, assume that our genomes are contained on just a single chromosome. Now, not only do all the cells in your body have identical genomes⊠[ADVANCE] But individual humans have genomes that are very similar to each other. So similar that I can define âthe sameâ chromosome between individuals⊠and that means⊠[ADVANCE] That we can define a âbaseâ or a âreferenceâ chromosome. Now that there is a reference that all of us adhere to⊠[ADVANCE] We can define a concept of âlocationâ across chromosomes. This is possibly the most important concept in genome informatics, the idea that DNA defines a common linear coordinate system. This also means that we can talk about differences between individuals in terms of diffs to a common reference genome. But where does this reference genome come from?
- Without losing much, assume that our genomes are contained on just a single chromosome. Now, not only do all the cells in your body have identical genomes⊠[ADVANCE] But individual humans have genomes that are very similar to each other. So similar that I can define âthe sameâ chromosome between individuals⊠and that means⊠[ADVANCE] That we can define a âbaseâ or a âreferenceâ chromosome. Now that there is a reference that all of us adhere to⊠[ADVANCE] We can define a concept of âlocationâ across chromosomes. This is possibly the most important concept in genome informatics, the idea that DNA defines a common linear coordinate system. This also means that we can talk about differences between individuals in terms of diffs to a common reference genome. But where does this reference genome come from?
- Here is Bill Clinton (and Craig Venter and Francis Collins), announcing in June of 2000 the ârough draftâ of the Human Genome â this is the Human Genome Project. Took >10 years and $2 billion What did this actually do?
- An ASCII text file with a linear sequence of 3 billion ACGTs This is the reference. Now go cure cancer. If this looks uninterpretable, it is!
- Anyone recognize this? Want to make an analogy. Difficult to understand. How do I make it more comprehensible?
- Mapmakers work to add ANNOTATIONS to the map. Annotations are keyed by geo coordinates. Points, lines, and polygons in 2d space
- And often, itâs only the annotations that are interesting, so mapmakers focus on *annotation* of the maps themselves. The core technologies are 2D planar and spherical geometry, geometric operations composed out of latitudes and longitudes. This is what we want to do for the genome. What does the annotated map of the genome look like?
- Chromosome on top. Highlighted red portion is what weâre zoomed in on. See the scale: total of about 600,000 bases (ACGTs) arranged from left to right. Multiple annotation âtracksâ are overlaid on the genome sequence, marking functional elements, positions of observed human differences, similarity to other animals. In part itâs the product of numerous additional large biology annotation projects (e.g., HapMap project, 1000 Genomes, ENCODE). Lot's of bioinformatics is computing these elements, or evaluating models on top of the elements. How are these annotations actually generated? Shift gears and talk about the technology.
- DNA SEQUENCING If satellites provide images of the world for cartography, sequencers are the microscopes that give you âimagesâ of the genome. Over past decade, massive EXPONENTIAL increase in throughput (much faster than Mooreâs law)
- Get sample Extract DNA (possibly other manipulations) Dump into sequencer Spits out text file (actually looks just like that) But how to get from the text file to an annotation track that reconstructs a genome or shows position of certain functional elements? [ADVANCE] Bioinformatics is the computational process to reconstruct the genomic information. But⊠[ADVANCE] Often considered simply a black box. What does it actually look like inside?
- Get sample Extract DNA (possibly other manipulations) Dump into sequencer Spits out text file (actually looks just like that) But how to get from the text file to an annotation track that reconstructs a genome or shows position of certain functional elements? [ADVANCE] Bioinformatics is the computational process to reconstruct the genomic information. But⊠[ADVANCE] Often considered simply a black box. What does it actually look like inside?
- Get sample Extract DNA (possibly other manipulations) Dump into sequencer Spits out text file (actually looks just like that) But how to get from the text file to an annotation track that reconstructs a genome or shows position of certain functional elements? [ADVANCE] Bioinformatics is the computational process to reconstruct the genomic information. But⊠[ADVANCE] Often considered simply a black box. What does it actually look like inside?
- Pipelines, of course. Example pipeline: raw sequencing data => a single individualâs âdiffâ from the reference. How are these typically structured? Each step is typically written as a standalone program â passing files from stage to stage These are written as part of a globally-distrbuted research program, by researchers and grad students around the world, who have to assume the lowest common denominator: command line and filesystem. This has important implications for scalability. What does one of these files look like?
- Text is highly inefficient Compresses poorly Values must be parsed Text is semi-structured Flexible schemas make parsing difficult Difficult to make assumptions on data structure Text poorly separates the roles of delimiters and data Requires escaping of control characters (ASCII actually includes RS 0x1E and FS 0x1F, but theyâre never used)
- Imposes severe constraint: global sort invariant. => Many impls depend on this, even if itâs not necessary or conducive to distributed computing.
- Bioinformaticians LOVE hand-coded file formats. But only store several fundamental data types. Strong assumptions in the formats. Inconsistent implementations in multiple languages. Doesnât allow different storage backends. OK, we discussed what the data/files are like that are passed around. What about the computation itself?
- Letâs take one of the transformations in the pipeline. Basically a more complex version of a DISTINCT operation.
- Actual code from the standard Picard implementation of MarkDuplicates. Two things should be going on: Algorithm/Method overall Actual code implementation. Start by building some data structures from the input files. Then iterate over file and rewrite is as necessary.
- But what if we jump into one of these functions. Youâll find a dependence on⊠[ADVANCE]
- An input option related to Unix file handle limits? WTF? Why should this METHOD need know anything about the platform that this is running on? LEAKY ABSTRACTIONS
- Most bioinformatics tools make strong assumptions about their environments, and also the structure of the data (e.g., global sort), when it shouldnât be necessary. Ok, but thatâs not all⊠[ADVANCE]
- Weâve looked at the data and a bit of code for one of these tools. But this runs the pipeline on a single individual. But of course, itâs never one pipeline⊠[ADVANCE] Itâs a pipeline per person! But since each pipeline runs (essentially) serially, scaling it up is easy⊠[ADVANCE] Scale out! Typically managed with a pretty low-level job scheduler.
- Weâve looked at the data and a bit of code for one of these tools. But this runs the pipeline on a single individual. But of course, itâs never one pipeline⊠[ADVANCE] Itâs a pipeline per person! But since each pipeline runs (essentially) serially, scaling it up is easy⊠[ADVANCE] Scale out! Typically managed with a pretty low-level job scheduler.
- Weâve looked at the data and a bit of code for one of these tools. But this runs the pipeline on a single individual. But of course, itâs never one pipeline⊠[ADVANCE] Itâs a pipeline per person! But since each pipeline runs (essentially) serially, scaling it up is easy⊠[ADVANCE] Scale out! Typically managed with a pretty low-level job scheduler.
- MANUAL split and merge MANUAL resource request BABYSIT for failures/errors CUSTOM intermediate ser/de But this basically works and the parallelism is pretty simple. This architecture has kept up with the pace of sequencing for some time now. Pipelines. Managed by job schedulers. Passing files around. SO WHY AM I EVEN UP HERE TALKING? Two reasonsâŠ
- SCALE! New levels of ambition for large biology projects. 100k genomes at Genomics England in collaboration with National Health Service. Raw data for a single individual can be in the hundreds of GB But even before we hit that huge scale (which is soon)âŠ
- For latest algorithms, we donât want to analyze each sample separately. We want to use ALL THE DATA we generate. Well, these pipelines often include lots of aggregation, perhaps we can just⊠[ADVANCE] Do the easy thing! Not ideal, especially as the amount of data goes up (data transfer), number of files increases (we saw file handles). May start hitting the cracks. But even worse⊠[ADVANCE] God help you if you want to jointly use all the data in earlier part of the pipeline. 2 Problems: Large scale Using all data simultaneously
- For latest algorithms, we donât want to analyze each sample separately. We want to use ALL THE DATA we generate. Well, these pipelines often include lots of aggregation, perhaps we can just⊠[ADVANCE] Do the easy thing! Not ideal, especially as the amount of data goes up (data transfer), number of files increases (we saw file handles). May start hitting the cracks. But even worse⊠[ADVANCE] God help you if you want to jointly use all the data in earlier part of the pipeline. 2 Problems: Large scale Using all data simultaneously
- For latest algorithms, we donât want to analyze each sample separately. We want to use ALL THE DATA we generate. Well, these pipelines often include lots of aggregation, perhaps we can just⊠[ADVANCE] Do the easy thing! Not ideal, especially as the amount of data goes up (data transfer), number of files increases (we saw file handles). May start hitting the cracks. But even worse⊠[ADVANCE] God help you if you want to jointly use all the data in earlier part of the pipeline. 2 Problems: Large scale Using all data simultaneously How do we solve these problems?
- Things like global sort order are overly restrictive and leads to algos relying on it when itâs not necessary.
- A lot of the problems go away with a tool like Spark. Example of an algo. Bioinformatics loves evaluating probabilistic models on the genome annotations. We can easily extract parallelism at different parts of our pipelines. With easiest language, we can describe a high-level computation. Use higher level distributed computing primitives and let the system figure out all the platform issues for you: storage, job scheduling, fault tolerance, shuffles, serde.
- Layered abstractions. Use multiple storage engines with different characteristics. Multiple execution engines. Avro ties it all together. Application code/algos should only touch the top of the abstraction layer. Cheap scalable STORAGE at bottom Resource management middle EXECUTION engines that can run your code on the cluster and provide parallelism Consistent SERIALIZATION framework Scientist should NOT WORRY about lower levels (coordination, file formats, storage details, fault tolerance)
- Weâve implemented this vision with Spark, starting from the Amplab (same people that gave you Spark) into a project called ADAM The reason this works is that Spark naturally handles pipelines, and automatically performs shuffles when appropriate, but alsoâŠ
- In addition to some of the standard pipeline transformations, implemented the core spatial join operations (analogous to a geospatial library).
- Another computation for a statistical aggregate on genome variant data. Details not important. Spark data flow: Distributed data load High level joins/spatial computations that are parallelized as necessary. But really nice thing is because our data is stored using the Avro data model⊠[ADVANCE]
- You can execute the exact same computation using, for example, SQL! Pick the best tool for the job.
- Single-node performance improvements. Free scalability: fixed price, significant wall-clock improvements See most recent SIGMOD.
- Controversial and disagree with many. #8 similar to assuming primitive lowest common denominator For especially for the last âmythâ, being able to achieve the ambition that people are proposing will require moving beyond âanything is okâ to making some important technical decisions.
- Not to be outdone, Craig Venter proposes 1 million genomes at Human Longevity Inc.
- Cloudera is hiring. Including the data science team.