How New York Genome Center Manages the Massive Data
Generated from DNA Sequencing
Transcript of a sponsored discussion on how the drive to better diagnose diseases and develop
more effective treatments is aided by swift, cost efficient, and accessible big data analytics
infrastructure.
Listen to the podcast. Find it on iTunes. Get the mobile app. Sponsor: Hewlett
Packard Enterprise.
Dana Gardner: Hello, and welcome to the next edition of the HPE Discover Podcast Series.
I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host and
moderator for this ongoing discussion on IT innovation and how it’s making
an impact on people’s lives.
Our next big-data use case leadership discussion examines how the non-profit
New York Genome Center manages and analyzes up to 12 terabytes of data
generated each day from its genome sequence appliances. We’ll learn how to
better diagnose disease and develop more effective treatments aided by swift,
cost efficient, and accessible big-data analytics.
To hear how they exploit the vast data outputs to then speedily correlate for time sensitive
reporting, please join me in welcoming our guest.
Start Your
HPE Vertica
Community Edition Trial Now
We're here with Toby Bloom, Deputy Scientific Director for Informatics at the New York
Genome Center in New York. Welcome, Toby.
Toby Bloom: Hi. Thank you.
Gardner: First, tell us a little bit about your organization. It seems like it’s a unique institution, a
large variety of backers, consortium members. Tell us about it.
Bloom: New York Genome Center is about two-and-a-half years old. It was formed initially as a
collaboration among 12 of the large medical institutions in New York, Cornell, Columbia,
NewYork-Presbyterian Hospital, Mount Sinai, NYU, Einstein Montefiore, and Stony Brook
University. All of the big hospitals in New York decided that it would be better to have one
genome center than have to build 12 of them. So we were formed initially to be the center of
genomics in New York.
Gardner: And what does one do at a center of genomics?
Gardner

Bloom: We're a biomedical research facility that has a large capacity to sequence genomes and
use the resulting data output to analyze the genomes, find the causes of disease, and hopefully
treatments of disease, and have a big impact on healthcare and on how medicine
works now.
Gardner: When it comes to doing this well, it sounds like you are generating an
awesome amount of data. What sort of data is that and where does it come from?
Bloom: Right now, we have a number of genome sequencing instruments that
produce about 12 terabytes of raw data per day. That raw data is basically lots of
strings of As, Cs, Ts and Gs, DNA data from genomes from patients who we're sequencing.
Those can be patients who are sick and we are looking for specific treatment. They can be
patients in large research studies, where we're trying to use and correlate a large number of
genomes to find the similarities that show us the cause of the disease.
Gardner: When we look at a typical big-data environment like in a corporation, it’s often
transactional information. It might be outputs from sensors or machines. How is this a different
data problem when you are dealing with DNA sequences?
Lots of data
Bloom: Some of it’s the same problem, and some of it’s different. We're bringing in lots of
data. The raw data, I said, is probably about 12 terabytes a day right now. That could easily
double in the next year. But than we analyze the data, and I probably store three to four times
that much data in a day.
In a lot of environments, you start with the raw data, you analyze it, and you cook it down to
your answers. In our environment, it just gets bigger and bigger for a long time, before we get
the answers and can make it smaller. So we're dealing with very large amounts of data.
We do have one research project now that is taking in streaming data from devices, and we think
over time we'll likely be taking in data from things like cardiac monitors, glucose
monitors, and other kinds of wearable medical devices. Right now, we
are taking in data off apps on smartphones that are tracking movement
for some patients in a rheumatoid arthritis study we're doing.
We have to analyze a bunch of different kinds of data together. We’d like to
bring in full medical records for those patients and integrate it with the genomic data. So we do
have a wide variety of data that we have to integrate, and a lot of it is quite large.
Gardner: When you were looking for the technological platforms and solutions to accommodate
your specific needs, how did that pan out? What works? What doesn’t work? And where are you
in terms of putting in place the needed infrastructure?
Bloom

Bloom: The data that comes off the machines is in large files, and a lot of the complex analysis
we do, we do initially on those large files. I am talking about files that are from 150 to 500
gigabytes or maybe a terabyte each, and we do a lot of machine-learning analysis on those. We
do a bunch of Bayesian statistical analyses. There are a large number of methods we use to try to
extract the information from that raw data.
Start Your
HPE Vertica
Community Edition Trial Now
When we've figured out the variance and mutations in the DNA that we think are correlated with
the disease and that we were interested in looking at, we then want to load all of that into a
database with all of the other data we have to make it easy for researchers to use in a number of
different ways. We want to let them find more data like the data they have, so that they can get
statistical validation of their hypotheses.
We want them to be able to find more patients for cohorts, so they can sequence more and get
enough data. We need to be able to ask questions about how likely it is, if you have a given
genomic variant, you get a given disease. Or, if you have the disease, how likely it is that you
have this variant. You can only do that if it’s easy to find all of that data together in one place in
an organized way.
So we really need to load that data into a database and connect it to the medical records or the
symptoms and disease information we have about the patients and connect DNA data with RNA
data with epigenetic data with microbiome data. We needed a database to do that.
We looked at a number of different databases, but we had some very hard requirements to solve.
We were looking for one that could handle trillions of rows in a table without failing over, tens of
trillions of rows without falling over, and to be able to answer queries fast across multiple tables
with tens of trillions of rows. We need to be able to easily change and add new kinds of data to it,
because we're always finding new kinds of data we want to correlate. So there are things like
that.
Simple answer
We need to be able to load terabytes of data a day. But more than anything, I had a lot of
conversations with statisticians about why they don’t like databases, about why they keep asking
me for all of the data in comma-delimited files instead of databases. And the answer, when you
boiled it down, was pretty simple.
When you have statisticians who are looking at data with huge numbers of attributes and huge
numbers of patients, the kinds of statistical analysis they're doing means they want to look at
some much smaller combinations of the attributes for all of the patients and see if they can find

correlations, and then change that and look at different subsets. That absolutely requires a
column-oriented database. A row-oriented relational database will bring in the whole database to
get you that data. It takes forever, and it’s too slow for them.
So, we started from that. We must have looked at four or five different databases. Vertica was the
one that could handle the scale and the speed and was robust and reliable enough and is our
platform now. We're still loading in the first round of our data. We're still in the tens of billions of
rows, as opposed to trillions of rows, but we'll get there.
Gardner: You’re also in the healthcare field. So there are considerations around privacy,
governance, auditing, and, of course, price sensitivity, because you're a non-profit. How did that
factor into your decision? Is the use of off-the-shelf hardware a consideration, or off-the-shelf
storage? Are you looking at conversion infrastructure? How did you manage some of those cost
and regulatory issues?
Bloom: Regulatory issues are enormous. There are regulations on clinical data that we have to
deal with. There are regulations on research data that overlap and are not fully consistent with the
regulations on clinical data. We do have to be very careful about who has access to which sets of
data, and we have all of this data in one database, but that doesn’t mean any one person can
actually have access to all of that data.
We want it in one place, because over time, scientists integrate more and more data and get
permission to integrate larger and larger datasets, and we need that. There are studies we're doing
that are going to need over 100,000 patients in them to get statistical validity on the hypotheses.
So we want it all in one place.
What we're doing right now is keeping all of the access-control information about who can
access which datasets as data in the database, and we basically append clauses to every query to
filter down the data to the data that any particular user can use. Then we'll tell them the answers
for the datasets they have and how much data that’s there that they couldn’t look at, and if they
needed the information, how to go try to get access to that.
Gardner: So you're able to manage some of those very stringent requirements around access
control. How about that infrastructure cost equation?
Bloom: Infrastructure cost is a real issue, but essentially, what we're dealing with is, if we're
going to do the work we need to do and deal with the data we have to deal with, there are two
options. We spend it on capital equipment or we spend it on operating costs to build it ourselves.
In this case, not all cases, it seemed to make much more sense to take advantage of the
equipment and software, rather than trying to reproduce it and use our time and our personnel's
time on other things that we couldn’t as easily get.
A lot of work went into Vertica. We're not going to reproduce it very easily. The open-source
tools that are out there don’t match it yet. They may eventually, but they don’t now.

Getting it right
Gardner: When we think about the paybacks or determining return on investment (ROI) in a
business setting, there’s a fairly simple straightforward formula. For you, how do you know
you’ve got this right? What is it when you see certain, what we might refer to in the business
world as service-level agreements (SLAs) or key performance indicators (KPIs)? What are you
looking for when you know that you’ve got it right and when you’re getting the job done, based
all of its requirements and from all of these different constituencies?
Bloom: There’s a set of different things. The thing I am looking for first is whether the scientists
who we work with most closely, who will use this first, will be able to frame the questions they
want to ask in terms of the interface and infrastructure we’ve provided.
I want to know that we can answer the scientific questions that people have with the data we
have and that we’ve made it accessible in the right way. That we’ve integrated, connected and
aggregated the data in the right ways, so they can find what they are looking for. There's no easy
metric for that. There’s going to be a lot of beta testing.
The second thing is, are we are hitting the performance standards we want? How much data can I
load how fast? How much data can I retrieve from a query? Those statisticians who don’t want to
use relational databases, still want to pull out all those columns and they want to do their
sophisticated analysis outside the database.
Eventually, I may convince them that they can leave the data in the database and run their R-
scripts there, but right now they want to pull it out. I need to know that I can pull it out fast for
them, and that they're not going to object that this is organized so they can get their data out.
Gardner: Let's step back to the big picture of what we can accomplish in a health-level payback.
When you’ve got the data managed, when you’ve got the input and output at a speed that’s
acceptable, when you’re able to manage all these different level studies, what sort of paybacks do
we get in terms of people’s health? How do we know we are succeeding when it comes to
disease, treatment, and understanding more about people and their health?
Bloom: The place where this database is going to be the most useful, not by any means the only
way it will be used, is in our investigations of common and complex diseases, and how we find
the causes of them and how we can get from causes to treatments.
I'm talking about looking at diseases like Alzheimer’s, asthma, diabetes, Parkinson’s, and ALS,
which is not so common, but certainly falls in the complex disease category. These are diseases
that are caused by some combinations of genomic variance, not by a single gene gone wrong.
There are a lot of complex questions we need to ask in finding those. It takes a lot of patience
and a lot of genomes, to answer those questions.

The payoff is that if we can use this data to collect enough information about enough diseases
that we can ask the questions that say it looks like this genomic variant is correlated with this
disease, how many people in your database have this variant and of those how many actually
have the disease, and of the ones who have the disease, how many have this variant. I need to ask
both those questions, because a lot of these variants confer risk, but they don’t absolutely give
you the disease.
If I am going to find the answers, I need to be able to ask those questions and those are the things
that are really hard to do with the raw data in files. If I can do just that, think about the impact on
all of us? If we can find the molecular causes of Alzheimer’s that could lead to treatments or
prevention and all of those other diseases as well.
Gardner: It’s a very compelling and interesting big data use case, one of the best I’ve heard.
I am afraid we’ll have to leave it there. We've been examining how the New York Genome
Center manages and analyzes vast data outputs to speedily correlate for time sensitive reporting,
and we’ve learned how the drive to better diagnose diseases and develop more effective
treatments is aided by swift, cost efficient, and accessible big data analytics infrastructure.
Start Your
HPE Vertica
Community Edition Trial Now
So, join me in thanking our guest. We’ve been here with Toby Bloom, Deputy Scientific Director
for Informatics at the New York Genome Center. Thank you so much, Toby.
Bloom: Thank you, and thanks for inviting me.
Gardner: Thank you also to our audience for joining us for this big data innovation case study
discussion.
I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host for this ongoing series of
HP sponsored discussions. Thanks again for listening, and come back next time.
Listen to the podcast. Find it on iTunes. Get the mobile app. Sponsor: Hewlett
Packard Enterprise.
Transcript of a sponsored discussion on how the drive to better diagnose diseases and develop
more effective treatments is aided by swift, cost efficient, and accessible big data analytics
infrastructure. Copyright Interarbor Solutions, LLC, 2005-2016. All rights reserved.
You may also be interested in:
• Redmonk analysts on best navigating the tricky path to DevOps adoption

• DevOps by design--A practical guide to effectively ushering DevOps into any
organization
• Need for Fast Analytics in Healthcare Spurs Sogeti Converged Solutions Partnership
Model
• HPE's composable infrastructure sets stage for hybrid market brokering role
• Nottingham Trent University Elevates Big Data's role to Improving Student Retention in
Higher Education
• Forrester analyst Kurt Bittner on the inevitability of DevOps
• Agile on fire: IT enters the new era of 'continuous' everything
• Big data enables top user experiences and extreme personalization for Intuit TurboTax
• Feedback loops: The confluence of DevOps and big data
• IoT brings on development demands that DevOps manages best, say experts
• Big data generates new insights into what’s happening in the world's tropical ecosystems
• DevOps and security, a match made in heaven
• How Sprint employs orchestration and automation to bring IT into DevOps readiness