The Next Big Thing in Big Data

Because I have a lot to cover there won’t be time for questions at the end. And
I’m guessing some of the question won’t have simple answers. So you can go
to my blog jamesdixon.wordpress.com where each of the sections is a
separate post that you can comment on or ask questions about.

First let’s look at the data explosion that everyone is talking about at the
moment.

This is a quote from a paper about the importance of data compression
because of data explosion. It seems reasonable. Store information as
efficiently as possible so that the effects of the explosion are manageable.
[TRANSITION]
This was written in 1969.
So the data explosion is not a new phenomenon. It has been going on since
the mid 60’s.
http://www.sciencedaily.com/releases/2013/05/130522085217.htm

This is another quote, much more recent, that you might see online. This says
that the amount of data being created and stored is multiplying by a factor of 10
every two years. I have not found any numerical data to back this up so I will
drill into this in a few minutes.

So consider this graph of data quantities. It looks like it might qualifies as a
data explosion. But this is actually just the underlying trend of data growth with
no explosion happening.
This graph is just shows hard drive sizes for home computers. Starting with the
first PCs with 10MB drives in 1983, and going up to a 512GB drive today.
Some of you might recognize that this exponential growth is the storage
equivalent of Moore’s law, which states that the computing power doubles
every 2 years. And we can see from these charts that hard drives have
followed along at the same rate.
This exponential growth in storage combines with a second principle.

http://www.sciencedaily.com/releases/2013/05/130522085217.htm
This statement is not just an ironic observation.
This effect is due to the fact that the amount of data stored is also affected by
Moore’s law. With twice the computing power, you can process images that are
twice as big. You can run applications with twice the logic. You can watch
movies with twice the resolution. You can play games that are twice as
detailed. All of these things require twice the space to store them.
Today an HD movie can be 3 or 4 gigabytes. In 2001 that was your entire hard
drive.

With processing power doubling at the same rate that storage is increasing
what does this say about any gap between the data explosion and the CPU
power required to process it?

This is the growth in processing power

If we divide the amount of data by the amount of processing power we get a
constant. We get a straight line. If this holds to be true then we will never drown
in our own data.

Can we really call it an explosion, if it is just a natural trend? We don’t talk
about the explosion of processing power – it’s just Moore’s law. Is there a new
explosion that is over and above the underlying trend. If so how big is it and will
it continue? We are going to find the answers to all of these questions. Before
we do there are some things to understand.

Firstly there is a point, for any one kind of data, where the explosion stops or
slows down. It is the point at which the data reaches its natural maximum
granularity, and beyond which there is little practical value to increasing the
granularity. I’m going to demonstrate this natural maximum using some well
known data types.

Let’s start with color. Back in the early 80s we went from black and white
computers to 16 color computers. The 16 color palette was a milestone
because each color needed to have a name, and most computer programmers
at the time couldn’t name that many colors. So we had to learn teal, and
fushcia, and cyan and magenta.
Then 256 colors arrived a few years later. Which was great because it was too
many colors to name, so we didn’t have to.
Then 4,000 colors. And within the decade we were up to 24-bit color with 16
million colors. Since then color growth has slowed down. 30-bit color came 10
years later, followed by 48-bit color a decade ago, with its 280 trillion colors.
But in reality most image and video editing software, and most images and

Because we see a similar thing with video resolutions. They have increased,
but not exponentially. The current standard is 4K, which has 4 times the
resolution of 1080p. With 20/20 vision 4K images exceed the power of the
human eyeball when you view them from a typical distance. The “retina”
displays on Apple products are called that because they have a resolution
designed to match the power of human vision. So images and video are just
reaching their natural maximum but these files will continue to grow in size as
we gradually reduce the compression and increase the quality of the content.

In terms of ability, the human hearing system lies in between 16-bit sound and
24-bit sound. So again we have hit the natural limit of this data type.
If you still don’t believe in the natural granularity have I one further example.

Dates. In the 60’s and 70’s we stored dates in COBOL as 6 digits. This gave
rise to the Y2K issue.
We managed to avoid that apocalypse. With 32-bit dates we extended the date
range by 38 years. But since the creation of 64bit systems and 64 bit dates, the
next crisis for dates is? Everyone should have this in their diary. It’s a Sunday
afternoon. December 4th. But what year? Anyone? It’s the year 292 billion blah
blah blah.

So this is the graph showing the natural granularity of dates for the next 290
billion years.
[TRANSITION]
For reference the green line shows the current age of the universe, which is 14
billions years.

So now that we understand that different data types have a natural maximum
granularity, how does it relate to big data and the data explosion?

Look at the example of a utility company that used to record your power
consumption once and month and now does it every 10 seconds. Your
household applicances, the dishwasher, fridge, oven, heating and air
conditioning, TVs, computers don’t turn on an off that often. The microwave
has the shortest duration, but usually 20 seconds is the shortest time it is on
for.
[TRANSITION]
So this seems like a reasonable natural maximum

Now let’s take a cardiologist who, instead of seeing a patient once a month to
record some data, now can get data recorded once a second, 24 hours a day.
Your heart rate, core temperature, and blood pressure don’t change on a sub-
second interval.
[TRANSITION]
So again this seems like a reasonable natural maximum

As companies create a production big data system the amount of data stored
will increase dramatically until they have enough history stored – anywhere
from 2 to 10 years of data. Then the growth will reduce again. So the amount of
data will explode, or pop, over a period of a few years.

If this is your data before it pops
[TRANSITION]
Then this is your data after it pops

There are millions of companies in the world. If you only talk to the top 1000
companies in the USA you only get a very small view of the whole picture.

This brings us back to this claim, which aligns with the hype. How can we really
asses the growth in data?

My thought is that if the data explosion is really going at a rate of 10x every two
years, then HP, Dell, Cisco, and IBM must be doing really well, as these
manufacturers account for 97% of the blade server market in North America.
And Seagate, and Sandisk, and Fujitsu, and Hitachi must be doing well really
well too, as they make the storage. And Intel and AMD must be doing really
well because they make the processors.
Let’s look at HP who has 43% of worldwide market in blade servers.

From graphs of stock prices we can see that IBM, Cisco, Intel, EMC, and HP
don’t have growth rates that substantiate a data explosion.

When we look at memory and drive manufacturers the best of all of these is
Seagate and Micron with about a 200-300% growth over 5 years. That is a
multiplier of about a 1.7 year over year.

If we apply that multiplier of 1.7 to the underlying data growth trend we see that
the effect is noticeable but not really that significant. And that represents the
maximum growth of any vendor, so the actual growth will be less than this.

When we look at the computing industry from a high level we see a shift in
values, from hardware in the 60’s and 70’s with IBM as the king. To software
with Microsoft, then to tech products and solutions from companies like Google
and Apple and finally to products that are based purely on data like Facebook
and LinkedIn.
Over the same time periods we have seen statistics [TRANSITION] be
augmented with machine learning [TRANSITION] and more recently with deep
learning [TRANSITION]
The emergence of deep learning is interesting, because it provides
unsupervised or semi-supervised data manipulation for creating predictive
models. It is interesting because

It’s like the difference between mining for gold when you can just hammer
lumps of it out of the ground, and panning for tiny gold flakes in a huge pile of
sand and stones

The number of data scientists is not increasing at the same rate as the amount
of data and the number of data analysis projects is. We are not doubling the
number of data scientists every two years. This is why Deep Learning is a big
topic at the moment because it automates part of the data science process.
The problem is that the tools and techniques are very complicated.

For an example of complexity here is a classical problem know as the German
Tank Problem

In the second world war, leading up to the D-Day invasions the allies wanted to
know how many tanks the Germans were making

So statistics was applied to the serial numbers found on captured and
destroyed tanks.

As you can see the formulas are not simple.

And this problem deals with a very small data set.

The results were very accurate.
[TRANSITION]
When intelligence reports estimated that 1,400 tanks were being produced per
month,
[TRANSITION]
the statistics estimated 273.
[TRANSITION]
The actual figure was later found to be 274.

This next example is one of the greatest early works in the field of operation
research. This is interesting for several reasons. Firstly because, with the
creation of Storm and Spark Streaming and other real-time technologies we
are seeing a dramatic increase in the number of real-time systems that include
advanced analytics, machine learning, and model scoring. But this field is not
new. The other reason this is interesting is that it shows that correctly
interpreting the analysis is not always obvious and is more important than
crunching the data.

In an effort to make bombers more effective each plane returning from a
mission was examined for bullet holes and a map showing the density of bullet
holes over the planes was generated.
Tally ho chaps, said the bomber command commanders, slap some lovely
armor on these beauties where-ever you see the bullet holes
[TRANSITION]
Hold on a minute, said one bloke. I do not believe you want to do that.
Well, who are you said the bomber command commanders?
My name is Abram Wald. I am hungarian statistician who sounds a lot like
Michael Caine for some reason.
Wald’s reasoning was that they should put the armor where there are no bullet
holes, because that’s where the planes that don’t make it back must be getting
hit. Which happened to be places like the cockpit and the engines.

I deliberately chose two examples from 70 years ago to show that the problems
of analysis and interpretation are not new, and they are not easy. In 70 years
we have managed to make tools more capable but not much easier. But this
has to change.

So these are my conclusions on data science. We have more and more data,
but not enough human power to handle it, so something has to change.

Google trends shows us that interest in Hadoop is not dropping off.

And that R now has as much interest as SAS and SPSS combined.

Up until recently there was more interest in MapReduce than Spark and so
today we see mainly MapReduce in production. But as we can see from the
chart this is likely to change soon.

The job market shows us similar data with the core Hadoop technologies
currently providing more than 3 /4 of the job opportunities

And also that Java, the language of choice for big data technologies, has the
largest slice of the open jobs positions

One issue that is not really solved well today is SQL on Big Data.

On the job market HBase is the most sought after skill set. But you can see
that Phoenix, which is the SQL interface for HBase, is not represented in terms
of jobs. This chart also shows that the many proprietary big data SQL solutions
are not sought after skills at the moment. We don’t have a good solution for
SQL on big data yet.

Today aspects of an application that relate to the value of the data are typically
a version 2 afterthought for application developers.
[TRANSITION]
This affects the design of both applications, and data analysis projects.

For a software applications, the value of the data is not factored in, the natural
granularity is not considered, and the data analysis is not part of the
architecture. So we see architectures like this. With a database and business
logic and a web user interface.

The data analysis has to be built as a separate system, which is created and
integrated after the fact.
At a high-level, it will be something like this for a big data project, given the the
charts and trends we saw earlier. We see commonly see Hadoop, MapReduce,
Hbase, and R.

So here are the summary points for today’s technology stack.

Now let’s look into the future a little

If data is more valuable than software,
[TRANSITION]
we should design the software to maximize the value of the data,
and not the other way around.

We should design applications with the purpose of storing and getting value
from the natural maximum granularity

We should provide access to the granular data for the purpose of analysis and
operational support

If data is where the value is, then the use and treatment of the data should be
factored into an application from the start.
[TRANSITION]
It should be a priority of version 1.

[TRANSITION]
Valuing the data more than the software is a new requirement.
[TRANSITION]
Which demands new applications
[TRANSITION]
Which need new architectures

To illustrate this lets take the example of Blockbuster as an old architecture.
Hollywood studios would create content that was packaged and loaded in
batches to Blockbuster stores. The consumer would then jump in their car
every time little Debbie wanted to watch the Little Mermaid. Notice that in this
architecture there are a number of physical barriers between the parts.

Today we can still watch a Hollywood movie on our TV just as the Blockbuster
model enabled. But we have more sources of content, and we have more
devices we can use. And the architecture for this is very different in the middle
layers.

Consider implementing YouTube using the Blockbuster architecture. You take a
cat video on your camera or phone.
[TRANSITION]
Then you spend months burning 10,000 DVDs.
[TRANSITION]
Next you go to Fedex and spend $25,000 to get your DVDs to the Blockbuster
stores.
[TRANSITION]
Each blockbuster store will need 950 miles of shelving to store 120 million
videos and will have to add shelving at the rate of 1 foot per second to handle
the incoming videos.
As you can see it is economically not viable, and physically impossible to
implement YouTube using the old architecture

Consider an Internet of Things architecture where you have millions of devices
that have their own state and communicate with each other and with the central
system in real time. These systems need to perform complex event processing,
and analysis of state, and use predictive and prescriptive analytics to avoid
failures. You cannot bolt all of this analysis to the outside of the system as an
after-thought, it has to be designed-in and it has to be embedded.

SQL on Big Data is a separate topic for the future. This problem will get solved,
it is only a matter of time before we have a robust and full-featured scale-out
relational database for Big Data.
When this happens it will have a negative effect on the current database
vendors.
It will also affect the niche big data vendors whose main advantage is query
performance.
But it will help both the traditional analytic vendors and the open source
ecosystem.

Overall I think scalable technology is more interesting and more powerful that
“big” technology that cannot scale down. Scalable technology allows you to
start small and to grow without having to re-write, re-design or re-tool when you
don’t have to.

So this is my prediction of future software architectures that combine the
application and the analysis into one. By recognizing the value of the data we
build the big data technologies into the stack. We don’t have them as a
separate architecture that is built afterwards.

So here are the summary points for tomorrow’s technology stack.

Let’s look at the big data use cases and consider why they will change in the
future

In the database world if you took 50 database administrators and described a
data problem to them they would probably come up with a small number of
architectures and schemas between them. Probably only 3 or 4 when you
discount the minor differences. This happens because, as a community, we
understand how to solve problems using this kind of technology. We been
doing it for a while, there are lots of examples and teachings and papers, and
we have come to a consensus.
In the big data world we have not got to that point yet. Today if you took 50 big
data engineers and described a problem you would get a large number of
potential solutions back. Maybe as many as 50. We don’t collectively have
enough experience of trying similar things with different architectures and
technologies. But the emergence of these use cases helps a lot, because now
we can categorize different solutions together, even though the actual problem
might be different. Once we can do that we can compare the solutions and get
a better understanding of what works well and what the best practices should
be.

There is a set of problems that can be solved by SQL on Big Data as we talked
about earlier

There is a second set of problems that can be solved using a data lake
approach. These include agile analytics, and rewinding an application or device
state and replying events.

A third set of solutions exist around processing streams of data in real-time

Obviously if we really value data, we should value big data as well.

And if we value big data, then it should be built into the system from the start.
So the off-to-the-side big data projects should not exist, with the exception of a
data warehouse solution.

Some of the big data use cases that are emerging today only exist because we
are not building big data into the applications. Once we do that we will see
some of the big data use cases change.

So to conclude this section, while big data use cases are important, the
architecture stack needs to change, and with big data built in, we will see some
of the big data use cases changing in the future.

Big data would not exist without open source. The reason these big data
projects were created was because of a lack of existing products that were
scalable and cost-effective.

Many of these big data projects were created and donated by fourth generation
software companies – the companies who value data highly

Large enterprises understand the advantage of big data solutions and are
adopting them

Usage of Big Data by large enterprises fuels the news and excites the market
analysts and commentators, because this is who they talk to the most. The fact
that open source is not the main story is good because it makes acceptance of
open source an assumption, and not a point of discussion or contention.

The widespread interest in big data technologies fuels adoption and
contributions

Which benefits the open source ecosystem, So we have a feedback loop
where open source and the big data technologies both benefit

According to the most recent Black Duck “Future of Open Source” survey open
source is now used to build products and service offerings at 78% of
companies.
All of these statistics are trending in the favor of open source adoption

The Apache Foundation, the organization that stewards Hadoop, Hive, Hbase,
Cassandra, Spark and Storm, currently has over 160 projects. These are just
the Big Data projects.

This is the full list that includes the Apache HTTP server which has a 60%
market share of the web server market. Hadoop and Spark and Cassandra are
hugely popular in the Big Data space. We can expect more of these
technologies to become the standard or the default solution in their space.

Spark – in-memory general purpose large-scale processing engine
Kafka – cluster-based central data backbone
Samza – stream processing system
Mesos – cluster abstraction

I hope you enjoyed this talk. Whether or agree or disagree with these ideas or
if you have questions you can comment on my blog at any time. Thank you all
for joining.

The Next Big Thing in Big Data

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie The Next Big Thing in Big Data

Ähnlich wie The Next Big Thing in Big Data (20)

Mehr von Pentaho

Mehr von Pentaho (18)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

The Next Big Thing in Big Data