33. function
data data
data data
data data
data data
data data
34. function
data data
data data
data data
data data
data data
ship code not data
35. function
function
data
function function
data data
function function
data data data data
data data
function function
data data
data data
data data
data data function function
data data
function
data
ship code not data
90 slides - coffee\n\nBig Data guy - Data Scientist?\n\nScaling Data helps our customers tackle this new Big Data space - their whole stack\n
if you write applications that are JVM-based and you’re not using Metrics, you are doing it wrong\n\ninstrument your running production code to get real intelligence on what’s going on AS your running production code creates business value\n
At scaling data, people give us money for crunching data.\n
the reason they pay us so much money is that we crunch data that generates business value.\n
I thought this was going to be about big data\n
topline\n
recommendations for other complimentary products, driving overall spend higher\n\ncustomer classification and scoring - offer good customers deals for repeat business\n\ntransactional retargeting - abandoned shopping carts are mined, and personalized ads are returned to that specific user\n
cell tower data used to track where people go for lunch - identify a new restaurant site\n\nwhat roads are used so we can target billboards - demand higher prices\n\nmunicipal planning\n
cost cutting\n
pattern recognition in the power signature can point to imminent failure for expensive equipment\n
imagine a diagnosis that was cured with 17 procedures at immense cost\n\nsame diagnosis was cured with 5 procedures elsewhere\n\nanalyzing patient histories across the country / world can get us here\n\n
because we like more money... \n
\n
\n
\n
\n
We have even more types of data,\nbecoming ever more complex,\ndistributed across multiple existences,\nand we are left with the task of parsing out terabytes of noise to get to a megabyte of signal.\n
\n
Ever more data to try to find the business value\n\nCurrent tools are straining under the load, (banks) my talk last year\n\nThere is significant pain while using these big data tools - Why are they so hot now?\n\ngetting better\n
put it on disk in a database\n
SAN\n
even with the SAN... so you get a bigger machine\n
Oracle loves you for this!\n\n37signals approach - basecamp 1 server\n
\n
EMC loves you for this!\n\n\n
\n
IBM, HP, Sun loves (or loved) you for this!\n\nmore processors, more memory, more disk\n\n
\n
mounting costs are not good for...\n
the new approach, starting about 5 years ago\n\nNoSQL?\n\nNewSQL?\n
\n
\n
\n
so you’re sold on ‘scale out’\n
if you want your ops co-workers to be outside of their happy space, this is the ticket\n
lots of commodity hardware boxes ... racks\n
haproxy is a good one\n
things will break - fault tolerance\n\ndistribution of data - rebalancing\n\ntask coordination - leader election / masterless\n
reduce ops headache - Chef, Puppet\n
I still have the pain... I want to go forward with this\n
Cambrian explosion 530 million years ago\n\nappearance of most major animal phyla\n\ndiversification of organisms as earth warms, forms different climates\n
small records/files\n\nfixed schema, semi-structured, totally unstructured\n\ncolumn store, graph store\n
how will you ask for the data?\n\nkey lookup\n\ntable scan otherwise\n\nsecondary indices for oft-queried fields? mostly roll-your-own\n\n
per-request speed - fast = column db\n\namount of requests - availability of reads/writes under load becomes important\n
cassandra - read/write speed impressive\n\ndynamo-based clusters\n\nvery capable data stores\n
\n
hadoop rules the batch world for massive data sets\n
\n
probably 40-50 satellite projects that are non-core hadoop\n
distributions - should be matched to your use-case\n
\n
data --> business value\n
logging only, from Facebook\n\nkind of old and busted\n\nbut still on every Facebook server (or was at one time), so battle-tested\n
near-realtime: minutes\n\nreliability: getting better with recent releases\n\nmgmt: complicated\n\nsupport: apache project\n
a more general data ingest tool, although it started with log files\n\nnear-realtime: seconds\n\nreliability: best effort, store+retry on failure, and end-to-end mode \nthat uses acks and a write ahead log.\n\nmgmt: master or masters, then smooth from there\n\nsupport: cloudera\n
if you have a realtime component, use Storm\n\nit’s already distributed, reliable, easily manageable.\n
big files\n\nrecent performance improvements\n\nships with hadoop\n
unique for small files\n\nperformance over hdfs\n\nsnapshotting\n
low-latency column store\n\nfast key-based access\n\nalso have MR to do in batch/background\n
time series schema for hbase\n\nStumbleupon\n
a framework for processing in parallel on large clusters\n\nmap - nodes process local data\n\nreduce - reduces the ‘map output’ in some way (sum, count, etc)\n\n(shuffle & sort are in between M & R)\n
high-level language built on top of MR\n\noften favored for data movement, but can be used for querying / processing too\n
\n
\n
\n
high-level language built on top of MR\n\nstriving for SQL-like language\n
\n
high-level language built on top of MR\n\nmultiple MR jobs linked together\n\ncomplex query workflows\n
querying DSL written in Clojure\n
Excel-like frontend tool on top of Hadoop\n\nspreadsheet-like interface targets business users\n\njoins, data ingest too\n\n
released with Hadoop 0.23\n\nsplit JobTracker into:\n - ResourceManager (RM)\n - ApplicationMaster (AM), which does job scheduling/monitoring\n\nyou can run different applications now (next slide)\n
\n
one of the highest levels of ‘gaining insight’\n\nRecommendation\nClassification\nClustering / Segmentation\nPredictive Analytics\nSimilarity\n
loose federation of machine learning algorithms that run on hadoop\n\nHadoop not best system for some of these, although MRv2 is now here\n\nsome algos are better than others - you have been warned\n
output targets of Hadoop jobs\n
I’m not a hater!\n\nGreat tool for 40 years\n\n
mongo, redis\n
back into the cluster for use in another MR job\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
sexy, complicated algorithms are very insightful\n\nBUT more data and a shittier / more basic algorithm wins\n\ndata can overcome “known truths” and organizational inertia\n\n
for your organization, start small\n\ndon’t bet the farm... maybe 10-15% of your analytics budget\n\nskunkworks projects, hackers, etc.\n