Ari Berman - Intel Big Data Seminar 9/6/2012

BIOTEAM
Enabling Science

Storage Infrastructure
Font: Optima Regular

and Data Management
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)
Light Blue #6699CC (CMYK 62, 22, 3, 0)

in Life Sciences
Ari E. Berman, Ph.D.
Senior Scientiﬁc Consultant, BioTeam, Inc.

©BioTeam, Inc. 2012 - http://www.bioteam.net

BIOTEAM A little about me
Enabling Science

• Ph.D. in Molecular Biology/Neuroscience

Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Trained in laboratory and bioinformatics
• 13 years experience as an IT infrastructure/
HPC geek/Perl monger
• Odd mix of skills led me to BioTeam
• Joined BioTeam in May

BIOTEAM Who is BioTeam?
Enabling Science

• Independent Consulting Practice

Colors:

• Made up of scientists vast experience in
Dark Blue #003399 (CMYK 96, 69, 3, 0)

software, HPC, and IT
• Unique cross-section of skill sets
• 10+ years of bridging the gap between
technology and science
• Functions as much as a think tank as a
consulting practice.


Why am I here talking
BIOTEAM
Enabling Science to you?

• We work on broad range of projects:

Pharma, Biotech, EDU, .gov, .mil, etc.
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• We are in a unique position: can see how
people are approaching current problems
• We work from a tech agnostic perspective:
we provide what’s best for the customer
• Our niche: 1000ft. overview of tech
problems in life sciences


BIOTEAM
Enabling Science
Why are we all here?


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

Big data in life-sciences: just when you thought
it was safe to go back into the datacenter...


BIOTEAM Big data: the tired story
Enabling Science

• Next-generation sequencing,
Mass spec, imaging, etc.

•
Colors:

High-throughput
Dark Blue #003399 (CMYK 96, 69, 3, 0)

experimentation

• Clinical research/standard
healthcare - personalized
medicine

• Un-natural expansion of
technology (sequencing)

• Now: we can get the data
fast, what do we do with it?


BIOTEAM Big data: the tired story
Enabling Science

• At this point, this is an old
problem

•
Colors:
Most sequencers generating
Dark Blue #003399 (CMYK 96, 69, 3, 0)

0.5TB/day

• Final genomes around
300GB

• High-volume quantitative
methods quickly produce
100’s of TBs of data

• The kicker: tight research
budgets


BIOTEAM Storing Big Data
Enabling Science

• Problem is less about storing
the data. We’ve solved
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)
storage.

• We can now put in
thousands of spindles in a
semi-affordable manner

• Lots of high-density boxes

• The petabyte challenge has
been met

• Now, it needs to work well

• And still be affordable


Today’s problem: Accessing
BIOTEAM
Enabling Science Big Data


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0) • In practice - get to 1.5PB,
500M files: metadata falls off
a cliff

• Directory listings take
minutes

• Sorting takes forever

• Forget about filesystem
profiling/optimization


BIOTEAM

•

Colors:
What’s being done?

•
Dark Blue #003399 (CMYK 96, 69, 3, 0)
SSDs thought to be our

savior

• Blazing fast, SLC, many in
parallel

• Parallel ﬁlesystems could
cache metadata on SSDs

• Reduce search time orders
of magnitude


BIOTEAM

•
Of course, it’s not that
Colors:
simple
Dark Blue #003399 (CMYK 96, 69, 3, 0)

•
Now, distribution and access
points of SSDs matter

• How they are addressed
matters

• How many small files on the
filesystem matters

• How the files are to be used
matters


BIOTEAM But wait: there’s more
Enabling Science

• A consistent array of disks

Colors:
no longer enough beyond
1.5PB
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Entire solutions of high-
speed disks not cost-
effective

• Distribution of ﬁle access
needs: some fast, some
archive

• Tiering of storage
infrastructure


BIOTEAM Tiering
Enabling Science

• Keep archival data on
slower, cheaper disks
Colors:
• No SSDs
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Keep fast access ﬁles on
smaller, high-speed disks with
many (possibly all) SSDs
(HPC, high throughput
needs)
• Mid-level tiers for
administrative needs
(documents, etc)
• Can even add a tape tier for
more permanent storage


BIOTEAM Managing Tiers
Enabling Science

•
Administratively difficult

•
Colors:
Can manage by different
Dark Blue #003399 (CMYK 96, 69, 3, 0)

mount points, quotas, user
education

• Better: policy engines

• Use with parallel file systems
(GPFS, OneFS, etc)

• Policy based automated
movement of files through
tiers, even to tape


BIOTEAM By golly, we’ve done it!
Enabling Science


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• If done correctly, single namespace
infrastructure can work well for all needs
• Can handle HPC to archive
• Can be done in a semi-affordable manner


BIOTEAM Now what?
Enabling Science

Colors: • Now, we’re faced with more problems

Dark Blue #003399 (CMYK 96, 69, 3, 0)

• For NIH, HIPAA laws, and general sanity,

need DR
• Need twice the space than you’ll use
• No other way to do it right now
• Use inexpensive, slow disk solutions to save
money on DR


BIOTEAM Now what?
Enabling Science


•
Colors:
Also: how to keep track
Dark Blue #003399 (CMYK 96, 69, 3, 0)

of data?

• At 1PB and 0.5 billion
ﬁles, creative directory
structures lose out

• Complexity too much
for anyone to handle


BIOTEAM Data Management
Enabling Science

•

Colors: One solution: databases
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Keep higher depth of
metadata (tagging,
descriptions)

• Cumbersome for the
general user to use: adds
complexity layer to user
experience


Enabling Science

•

Colors: Databases can work,
Dark Blue #003399 (CMYK 96, 69, 3, 0)
though

• iRODS is a good
example

• Put the metadata
database layer in-
between the ﬁlesystem
and the user


Enabling Science

• Others working on this
model as well

Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Cambridge Computer:
“As you approach
billions of files, file
exploring is no longer
feasible.”

• Need a new interface

• Rich metadata to keep
track of the files


BIOTEAM Wait, more metadata?
Enabling Science

• More metadata? wasn’t
this the original problem

Colors:

on large ﬁlesystems?
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Wouldn’t this make
matters worse?

• Depends on how it is
done.

• Current models have
metadata completely
separate from ﬁles


BIOTEAM Wait, more metadata?
Enabling Science

• And...

•

Colors:
Who’s going to go back
and type all of that
Dark Blue #003399 (CMYK 96, 69, 3, 0)

metadata in?

• No one - we kind of
need to start over

• ...or, need a way of
inferring metadata and
ﬁlling in the blanks from
existing data

• Still need legacy support
for systems


BIOTEAM Or: Middleware
Enabling Science

Colors:
• Use an interactive software product

Dark Blue #003399 (CMYK 96, 69, 3, 0)

between filesystem and user

• Can manage link between filesystem
and extended metadata
• Can enhance the scientific process:
manage data, analysis, results, and
facilitate collaboration


BIOTEAM What are scientists doing?
Enabling Science

Lab Scientist w/
Excel
•Accessible for most
scientists
•Flexible
Colors:
• Data maintenance
Dark Blue #003399 (CMYK 96, 69, 3, 0)
burden on lab scientists

•Quickly overwhelmed in
size and complexity
•Data publication by email


Enabling Science

Lab Scientist w/
Excel
scientists
•Flexible
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)
Lab Bioinformatician
•Quick development of web-

•Quickly overwhelmed in based system
size and complexity •Rapid turn around for
•Data publication by emailscientist needs
•Single point of failure
•Limited breadth of
experience
•Poor documentation, poor
transition


Enabling Science

Lab Scientist w/
Excel
scientists
•Flexible
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

size and complexity •Rapid turn around for Outsource custom
•Data publication by emailscientist needs software
•Limited breadth of •Stable, professional software
experience •Well documented, easier
transition
•Poor documentation, poor Communication barrier with
transition •
scientists
•Lack of domain knowledge
leaves large functionality
gaps
•Inﬂexible design leaves
software obsolete in a
matter


Enabling Science

Lab Scientist w/
Excel
scientists
•Flexible
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

size and complexity •Rapid turn around for Outsource custom
•Data publication by emailscientist needs software
•Limited breadth of •Stable, professional software
experience •Well documented, easier
transition
•Poor documentation, poor Communication barrier with
transition •
scientists
“Shrink-wrapped”
•Lack of domain knowledge
software.
leaves large functionality
gaps •Lab data management solutions
•Inﬂexible design leavesleverage many customers, years
software obsolete in a experience
of
matter •Year-to-year enhancement of
product
•High purchase price due to limited
market
•Mismatch to local lab expertise
and workﬂow
•Unused complexity

BIOTEAM LIMS
Enabling Science

• Laboratory Information Management
Colors:
System
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Many out there now: standard and custom
• Many focus markets
• Basespace: Illumina (NGS)
• Quartzy: general lab monkey
• MiniLIMS

BIOTEAM Disclaimer
Enabling Science


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• This will feel like a sales pitch
• Just want to illustrate how we’re tackling
information mangement problem


BIOTEAM The BioTeam Solution: MiniLIMS
Enabling Science

• An affordable software product that leverages

real world experience
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

•

Decades of combined software and informatics
expertise
• Years of LIMS customization
• $4995 license for academic labs

• Flexible architecture that adapts to new
processes and technologies
• Schema-less design allows real time changes to data
model
• Plugin architecture allows mix and match functionality


Enabling Science

Font: Optima Regular • Customization options that match lab
Colors:
resources
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• End user customizable system and Excel import/

export that empowers lab scientists
• Accessible source code and APIs for in-house
developers
• BioTeam consulting for labs without development
resources, or development teams that are stretched
thin


Enabling Science

End user conﬁgurable

Colors: Form and Page Display GC Mass Spec
Dark Blue #003399 (CMYK 96, 69, 3, 0)

Invoicing

PHP API
Data and Conﬁguration
Objects
Analysis tools

Data Broker
NGS

Schema-less MySQL
persistence

MiniLIMS Core MiniLIMS Plugins


MiniLIMS: Linking lab to
BIOTEAM datacenter
Enabling Science

Central auth
Customer workflow MiniLIMS Core
Reagent
inventory Uptime
User acct setup, login reporting Lab workflow MiniLIMS Plugin

Colors: MiniLIMS Custom
Dark Blue #003399 (CMYK 96, 69, 3, 0)
Sample receiving
Sample registration

Sample / library prep,
QC

Run / slide / flowcell
Sample status setup

Instrument console

Run monitoring

Results delivery, billing Analysis launch,
monitoring, results


BIOTEAM Simple/Flexible Concept
Enabling Science

Type Name

Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

Property Value


BIOTEAM Simple to query
Enabling Science


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)


BIOTEAM Customizations: plugins
Enabling Science


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)


BIOTEAM Customizations: plugins
Enabling Science


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

1. Select Bowtie 3. Click to run
2. Select the Fastq File &
the protocol
Name the experiment


BIOTEAM Workflows
Enabling Science


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)


BIOTEAM Moving forward: Appliance
Enabling Science

• Turnkey solution

•
MiniLIMS + Local
Analysis Engine
Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

• Plan is to link to cloud
resources: automatic
backup & link to hosted
MiniLIMS

• 16 cores, 96GB RAM,
18T redundant storage,
SSD for OS.

• Solution for any lab
needing LIMS


BIOTEAM How to enable science
Enabling Science

• Solidify storage infrastructure

Colors:
• Add tiered storage with
Dark Blue #003399 (CMYK 96, 69, 3, 0)

policy engine to move data

• Supply DR

• Enable metadata
acceleration: SSDs + cache

• Implement middleware for
rich metadata tracking

• Make it easy for the
scientists


BIOTEAM
Enabling Science


Colors:
Dark Blue #003399 (CMYK 96, 69, 3, 0)

Thank you!


Ari Berman - Intel Big Data Seminar 9/6/2012

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Ari Berman - Intel Big Data Seminar 9/6/2012

Ähnlich wie Ari Berman - Intel Big Data Seminar 9/6/2012 (16)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Ari Berman - Intel Big Data Seminar 9/6/2012

Hinweis der Redaktion