Why is big data sooo fashionable with big and small companies from different industries? What has suddenly changed?
But we have seen constant growth for a long time. And simple growth would only explain some kinds of companies starting with big data (probably big ones) and then slow adoption. Databases started with big companies and took 20 years or more to reach everywhere because the need exceeded cost at different times for different companies. The internet, on the other hand, largely happened to everybody at the same time so it changed things in nearly all industries at all scales nearly simultaneously.
Why is big data exploding right now and why is it exploding at all?
The different kinds of scaling laws have different shape and I think that shape is the key.
The value of analytics always increases with more data, but the rate of increase drops dramatically after an initial quick increase.
The value of analytics always increases with more data, but the rate of increase drops dramatically after an initial quick increase.
The value of analytics always increases with more data, but the rate of increase drops dramatically after an initial quick increase.
In classical analytics, the cost of doing analytics increases sharply.
The result is a net value that has a sharp optimum in the area where value is increasing rapidly and cost is not yet increasing so rapidly.
New techniques such as Hadoop result in linear scaling of cost. This is a change in shape and it causes a qualitative change in the way that costs trade off against value to give net value. As technology improves, the slope of this cost line is also changing rapidly over time.
This next sequence shows how the net value changes with different slope linear cost models.
Notice how the best net value has jumped up significantly
And as the line approaches horizontal, the highest net value occurs at dramatically larger data scale.
Laugh on tech term in American English = garbage 10:38