Digital Marketing Plan, how digital marketing works
Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir
1. Share and analyse genomic data
at scale
with Spark, Adam, Tachyon & the Spark Notebook
by @DataFellas, Oct • 29th • 2015
2. Outline
• Sharp intro to Genomics data
• What are the Challenges
• Distributed Machine Learning to the rescue
• Projects: Distributed teams
• Research: Long process
• Towards Maximum Share for efficiency
3. Andy Petrella
Maths
Geospatial
Distributed Computing
Spark Notebook
Trainer Spark/Scala
Machine Learning
Xavier Tordoir
Physics
Bioinformatics
Distributed Computing
Scala (& Perl)
trainer Spark
Machine Learning
“There must be another way of doingthecredits” -- Robin Hood: Menin Tights (1993, Mel Brooks)
4. Analyse Genomic At Scale
Spark, Adam, Spark Notebook
• Sharp intro to Genomics data
• What are the Challenges
• Distributed Machine Learning to the rescue
5. What is genomics data?
DNA?
What makes us what we are…
… a complex biochemical soup.
With applications to medical diagnostics, drug response,
disease mechanisms
17. So what do we do with this?
Study variations between populations
Descriptive statistics
Machine Learning (Population stratification or Supervised
learning)
… and share and replay!
18. The Spark Notebook
… comes to the rescue.
Spark: easy APIs
Self described and consistent
Easily shared (code)
http://www.spark-notebook.io
22. So what do we do with this?
… and share and replay!
Code can be shared easily but we want better...
How do we share data produced by the notebook?
How do we publish the notebook as a service?
23. Share Genomic At Scale
Spark, Tachyon, Mesos, Shar3
• Projects: Distributed teams
• Research: Long process
• Towards Maximum Share for efficiency
24. Projects
Intrinsically involving many teams
geolocally distributed in different
countries or laboratories
with different skills in
Biology, Genetics, I.T., Medicine (, legal...)
28. Projects
Need proper data management between entities, yet
coping with:
amount of data
heterogeneity of people
distance between actors
constraints related to data
location
31. Research
Most reasons are quite obvious not have to be overlooked
Lots of measures and validation
Lots of control (including by Gov.)
Lots of actors
32. Research
As a matter of fact, research need
to be conducted on data and
to produce results
But both are highly exposed to reuse, so what if we lose
either of them?
33. Research
However, we can get into troubles instantly without even
losing them.
What if we don’t track the processes to go from one to the
other?
In any scientific process: confrontation, replay and
enhancement are key to move forward
34. This is misleading to think that sharing the code is enough.
Remind: we look for data and results, not for code.
The process includes the code, the context, the sources
and so on, and all should be part of the data
discovery/validation task
Research
35. Assess the risk factor associated with a disease given
mutations of a certain gene.
More than 50 years of data collecting and modelling.
Hundreds of researchers, each generation with new ideas.
Replaying old processes on new data,
new processes on old data
Research
36. Share share
share
All these facts relate to our capacity to share our work and
to collaborate.
We need to share efficiently and accurately
• data
• process
• results
38. Share share
share
“Create” Cluster
Find sources (context, quality, semantic, …)
Connect to sources (structure, schema/types,…)
Create distributed data pipeline/Model
Tune accuracy
Tune performances
Write results to Sinks
Access Layer
User Access
ops
data
ops data
sci
sci ops
sci
ops data
web ops data
web ops data sci
39. Share share
share
“Create” Cluster
Find sources (context, quality, semantic, …)
Connect to sources (structure, schema/types,…)
Create distributed data pipeline/Model
Tune accuracy
Tune performances
Write results to Sinks
Access Layer
User Access
ops
data
ops data
sci
sci ops
sci
ops data
web ops data
web ops data sci
42. That’s all folks
Thanks for listening/staying
Poke us on Twitter or via http://data-fellas.guru
@DataFellas
@Shar3_Fellas
@SparkNotebook
@Xtordoir & @Noootsab
Check also @TypeSafe: http://t.co/o1Bt6dQtgH