VariantSpark library for fast genomics analysis
•Als PPTX, PDF herunterladen•
1 gefällt mir•2,213 views
VariantSpark a customer Apache Spark library for genomic data. Customer wide random forest machine learning algorithm, designed for workloads with millions of features.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (19)
Ähnlich wie VariantSpark library for fast genomics analysis
Ähnlich wie VariantSpark library for fast genomics analysis (20)
Mehr von Lynn Langit
Mehr von Lynn Langit (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
VariantSpark library for fast genomics analysis
- 1. VariantSpark: a library for Genomics Transformational Bioinformatics | Denis C. Bauer | @allPowerde Lynn Langit
- 3. Natalie Twine Transformational Bioinformatics Team Transformational Bioinformatics | Denis C. Bauer | @allPowerde Denis Bauer Oscar Luo Rob Dunne Piotr SzulAidan O’BrienLaurence Wilson Adrian White Mia Champion Gaetan Burgio Collaborators David Levy News Software Dan Andrews Kaitao Lai Kaylene Simpson Iva Nikolic Ian Blair Kelly Williams
- 4. BMC Genomics 2015, 16:1052 PMID: 26651996 (IF=4) Cited 4 VariantSpark | Denis C. Bauer @allPowerde
- 5. Unsupervised ML : K-Means www.cloudaccess.eu 1000 x 40 Million variants Matrix * k-means Predict super population 4 14 ethnic groups and s u p e r populations VariantSpark | Denis C. Bauer @allPowerde * VariantSpark can also process phase 3 data: 3000 individuals and 80 million variants
- 6. Comparing K-Means Implementations 0 1000 2000 Python R H adoop Adam AD M IXTU R E VariantSpark method timeinseconds task binary−conversion clustering pre−processing 103 75 29 28 18 4 min VariantSpark | Denis C. Bauer @allPowerde
- 7. Supervised ML: Wide Random Forests Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 8. Transformational Bioinformatics | Denis C. Bauer | @allPowerde Genomic Research Workflow https://www.projectmine.com/about/ Focus
- 9. Performance – Faster and More Accurate VariantSpark is the only method to scale to 100% of the genome Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 10. Scaling to 50 M variables and 10 K samples Transformational Bioinformatics | Denis C. Bauer | @allPowerde 100K trees: 5 – 50h AWS: ~$215.50 100K trees: 200 – 2000h AWS: ~ $ 8620.00 • Yarn Cluster (12 workers) • 16 x Intel Xeon E5-2660@2.20GHz CPU • 128 GB of RAM • Spark 1.6.1 on YARN • 128 executors • 6GB / executor (0.75TB) • Synthetic dataset (mtry = 0.25) Whole Genome Range GWAS Range
- 12. Databricks & VariantSpark via a Jupyter notebook
- 13. Solving Important Questions… Cancer genomics?
- 14. DEMO: Who is a Hipster?
- 15. • Quickly access a managed Spark cluster - AWS EC2 / spot instances • Link to your data and perform whole genome analysis in real-time VariantSpark & Databricks Notebooks Transformational Bioinformatics | Denis C. Bauer | @allPowerde Jupyter Notebook Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 16. Joint-loci association test Hipster-Index = ((2 + GT[B6]) * (1.5 + GT[R1])) + ((0.5 + GT[C2]) * (1 + GT[B2])) Label = 1 if Hipster-Index>10 Genomic profile Label Samples(n=2500) Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 17. Try it out: VariantSpark Notebook https://databricks.com/blog/2017/07/26/breaking-the- curse-of-dimensionality-in-genomics-using-wide- random-forests.html
- 18. VariantSpark: a library for Genomics Transformational Bioinformatics | Denis C. Bauer | @allPowerde Lynn Langit
Hinweis der Redaktion
- http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002195
- http://www.cloudaccess.eu/blog/wp-content/uploads/2014/06/genetic_roots.png
- Chromosome 22; VM on Microsoft Azure with A7 Linux instance and 8 cores, 56GB memory running Ubuntu.
- https://academics.cloud.databricks.com/#notebook/170398/command/170419
- https://databricks.com/blog/2017/07/26/breaking-the-curse-of-dimensionality-in-genomics-using-wide-random-forests.html