1. Hands-on Classification

2. Preliminaries
• Code is available from github:
– git@github.com:tdunning/Chapter-16.git
• EC2 instances available
• Thumb drives also available
• Email to ted.dunning@gmail.com
• Twitter @ted_dunning

3. A Quick Review
• What is classification?
– goes-ins: predictors
– goes-outs: target variable
• What is classifiable data?
– continuous, categorical, word-like, text-like
– uniform schema
• How do we convert from classifiable data to
feature vector?

4. Data Flow
Not quite so
simple

5. Classifiable Data
• Continuous
– A number that represents a quantity, not an id
– Blood pressure, stock price, latitude, mass
• Categorical
– One of a known, small set (color, shape)
• Word-like
– One of a possibly unknown, possibly large set
• Text-like
– Many word-like things, usually unordered

6. But that isn’t quite there
• Learning algorithms need feature vectors
– Have to convert from data to vector
• Can assign one location per feature
– or category
– or word
• Can assign one or more locations with hashing
– scary
– but safe on average

7. Data Flow

9. Classifiable Data Vectors

12. Hashed Encoding

13. What about collisions?

14. Let’s write some code
(cue relaxing background music)

15. Generating new features
• Sometimes the existing features are difficult to
use
• Restating the geometry using new reference
points may help
• Automatic reference points using k-means can
be better than manual references

16. K-means using target

17. K-means features

18. More code!
(cue relaxing background music)

19. Integration Issues
• Feature extraction is ideal for map-reduce
– Side data adds some complexity
• Clustering works great with map-reduce
– Cluster centroids to HDFS
• Model training works better sequentially
– Need centroids in normal files
• Model deployment shouldn’t depend on HDFS

20. Average
models
Parallel Stochastic Gradient Descent
Train
sub
model
Model
I
n
p
u
t

21. Update
model
Variational Dirichlet Assignment
Gather
sufficient
statistics
Model
I
n
p
u
t

22. Old tricks, new dogs
• Mapper
– Assign point to cluster
– Emit cluster id, (1, point)
• Combiner and reducer
– Sum counts, weighted sum of points
– Emit cluster id, (n, sum/n)
• Output to HDFS
Read from
HDFS to local disk
by distributed cache
Written by
map-reduce
Read from local disk
from distributed cache

23. Old tricks, new dogs
• Mapper
– Assign point to cluster
– Emit cluster id, 1, point
• Combiner and reducer
– Sum counts, weighted sum of points
– Emit cluster id, n, sum/n
• Output to HDFS
MapR FS
Read
from
NFS
Written by
map-reduce

24. Modeling architecture
Feature
extraction
and
down
sampling
I
n
p
u
t
Side-data
Data
join
Sequential
SGD
Learning
Map-reduce
Now via NFS