How to get the best of both worlds : Big Data and Data Science? Run Deep Learning on Spark easily with BigDL library! Slides of my short conference, introduction to BigDL, for Christmas JUG event in Montpellier

1. Deep Learning
with Spark
Anastasia Lieva
Fuzzy Humanist, Data-Scientist
@lievAnastazia

2. Spark is a new Hero

3. Deep Learning is a new Hero

4. BigDL is a new epic story

5. BigDL
High-level deep learning library

6. BigDL
High-level deep learning library

7. BigDL
High-level deep learning library
Intel MKL
Scale-out w/ Spark

8. BigDL
Intel MKL

9. BigDL : Deep Learning on Spark

10. BigDL : Deep Learning on Spark

11. API:
Scala and Python

12. API:
Scala and Python
BUT

13. API:
Scala and Python
BUT
the disadvantage of all Python APIs
is

14. API:
Scala and Python
BUT
the disadvantage of all Python APIs
is
that they are written in Python

15. API:
Scala a̶̶̶n̶̶̶d̶̶̶ ̶̶̶P̶̶̶y̶̶̶t̶̶̶h̶̶̶o̶̶̶n̶̶̶

16. val conf = Engine.createSparkConf()
.setAppName("DeepLearningOnSpark")
.setMaster("local[3]")
val sparkSession = SparkSession.builder()
.config(conf).getOrCreate()
val sqlContext = sparkSession.sqlContext
val sparkContext = sparkSession.sparkContext
Engine.init
The same configs as Spark

17. val conf = Engine.createSparkConf()
.setAppName("DeepLearningOnSpark")
.setMaster("local[3]")
val sparkSession = SparkSession.builder()
.config(conf).getOrCreate()
val sqlContext = sparkSession.sqlContext
val sparkContext = sparkSession.sparkContext
Engine.init
The same configs as Spark

18. L
A
Y
E
R
1
L
A
Y
E
R
2
L
A
Y
E
R
3
L
A
Y
E
R
4
L
A
Y
E
R
5
Input
Data
Model Architecture

19. Tensor
https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/linear_algebra.html
DATA

20. Tensor
Sparse TensorTable
Sample
DATA

21. Tensor
Sparse TensorTable
Sample
Lua / Torch Tables
(Tensor of Features, Tensor of Targets)
Tensor(indices, values, shape)
DATA

22. Tensor
Sparse TensorTable
Sample
Mini-batch
Batch of Samples
DATA

23. DATA
Tensor
Sparse TensorTable
Sample
Mini-batch DataSet
For advanced applications only

24. L
A
Y
E
R
1
L
A
Y
E
R
2
L
A
Y
E
R
3
L
A
Y
E
R
4
L
A
Y
E
R
5
Input
Data
Model Architecture

25. More than 100 layers !
Embedding
Pooling
Convolution
Normalization
Reccurent
DropOut
Sparse
… and others
Layers

26. L
A
Y
E
R
1
L
A
Y
E
R
2
L
A
Y
E
R
3
L
A
Y
E
R
4
L
A
Y
E
R
5
Input
Data
Expected
Learning by Backpropagation

27. L
A
Y
E
R
1
L
A
Y
E
R
2
L
A
Y
E
R
3
L
A
Y
E
R
4
L
A
Y
E
R
5
Input
Data
Prediction
Learning by Backpropagation

28. L
A
Y
E
R
1
L
A
Y
E
R
2
L
A
Y
E
R
3
L
A
Y
E
R
4
L
A
Y
E
R
5
Input
Data
Prediction
Ground truth
Error
Learning by Backpropagation

29. L
A
Y
E
R
1
L
A
Y
E
R
2
L
A
Y
E
R
3
L
A
Y
E
R
4
L
A
Y
E
R
5
Input
Data
Prediction
Ground truth
Error
Update weights in every layer w/ an optimization algorithm
Learning by Backpropagation

30. L
A
Y
E
R
1
L
A
Y
E
R
2
L
A
Y
E
R
3
L
A
Y
E
R
4
L
A
Y
E
R
5
Input
Data
Prediction
Ground truth
Error
Update weights in every layer w/ an optimization algorithm
Retry prediction with updated weights
Learning by Backpropagation

31. Losses
More than 30 criterions :
mean squared error,
binary cross entropy,
negative log likelihood criterion,
KL-divergence of the Gaussian distribution...

32. Losses
More than 30 criterions :
mean squared error,
binary cross entropy,
negative log likelihood criterion,
KL-divergence of the Gaussian distribution...
Optimization algorithms
Most popular gradient descent algorithms :
SGD, Adam, Adagrad, Adadelta, AdaMax

33. Let’s predict something!

34. Let’s predict something!
X X

35. Let’s predict something!
X X
Good BadMore Or Less

36. RegexTokenizer()
Word2Vec()
SpakMLlib
Preprocess unstructured data

37. RegexTokenizer()
Word2Vec()
Tensor[Vector]
Sample(featureTensor, label)
SpakMLlib
BigDL
Preprocess unstructured data

38. http://intellabs.github.io/RiverTrail/tutorial/
Convolutional Neural Network

39. Convolutional Neural Network

40. Convolutional Neural Network
Bonjour, on recrute à Montpellier (#systeme, reseau , #Devops, #Linux ).
n'hésitez pas à postuler et à diffuser, Merci beaucoup .
PS nous ne sommes pas une SSII

41. Convolutional Neural Network
Bonjour, on recrute à Montpellier (#systeme, reseau , #Devops, #Linux ).
n'hésitez pas à postuler et à diffuser, Merci beaucoup .
PS nous ne sommes pas une SSII
Montpellier
#systeme, reseau , #Devops, #Linux
pas une SSII

42. Convolutional Neural Network
Bonjour, on recrute à Montpellier (#systeme, reseau , #Devops, #Linux ).
n'hésitez pas à postuler et à diffuser, Merci beaucoup .
PS nous ne sommes pas une SSII
Montpellier
#systeme, reseau , #Devops, #Linux
pas une SSII
$$$$$ ?

43. Convolutional Neural Network
Bonjour, on recrute à Montpellier (#systeme, reseau , #Devops, #Linux ).
n'hésitez pas à postuler et à diffuser, Merci beaucoup .
PS nous ne sommes pas une SSII
Montpellier
#systeme, reseau , #Devops, #Linux
pas une SSII
$$$$$ ?
Bad

44. T
E
M
P
O
R
A
L
Conv
R
E
L
U
T
E
M
P
O
R
A
L
MaxP
ool
L
I
N
E
A
R
D
R
O
P
O
U
T
R
E
L
U
L
I
N
E
A
R
L
O
G
S
O
F
T
M
A
X
Model Architecture

45. val model = Sequential[Double]()
.add(TemporalConvolution(inputSize, outputSizeTempConv, kernelSize))
.add(ReLU())
.add(TemporalMaxPooling(outputSizeMaxPooling)
.add(Linear(inputSizeLinearLayer, outputSizeLinearLayer))
.add(Dropout(0.1))
.add(ReLU())
.add(Linear(inputSizeLinearLayer2, outputSizeLinearLayer2))
.add(LogSoftMax())
Model Architecture
In BigDL

46. val criterion = new ClassNLLCriterion[Double]
val optimizer = Optimizer(model, trainData, criterion, batchSize)
optimizer
.setOptimMethod(
new Adagrad(learningRate, learningRateDecay))
.optimize()
Training model
In BigDL

47. val criterion = new ClassNLLCriterion[Double]
val optimizer = Optimizer(model, trainData, criterion, batchSize)
optimizer
.setOptimMethod(
new Adagrad(learningRate, learningRateDecay))
.optimize()
Training model
In BigDL

48. val optimizer = Optimizer.apply(model, trainData, criterion, 6)
val logdir = "mylogdir"
val appName = "job-offers-filter"
val trainSummary = TrainSummary(logdir, appName)
val validationSummary = ValidationSummary(logdir, appName)
optimizer.setTrainSummary(trainSummary)
optimizer.setValidationSummary(validationSummary)
optimizer
.setOptimMethod(
new Adagrad(learningRate = 0.01, learningRateDecay = 0.0002))
.optimize()
Config for tensorboard

49. BigDL & Tensorboard

50. BigDL & Tensorboard

51. Spark Pipelines Integration

52. Preprocess unstructured data
RegexTokenizer()
Word2Vec()
Dataframe
.select(“features”, “label”)
SpakMLlib

53. val model = Sequential[Double]()
.add(TemporalConvolution(100, 20, 5))
.add(ReLU())
.add(TemporalMaxPooling(96))
.add(Linear(20, 100))
.add(Dropout(0.1))
.add(ReLU())
.add(Linear(100, 3))
.add(LogSoftMax())
val criterion = new ClassNLLCriterion[Double]
Spark Integration

54. Spark Integration
val estimator = new DLEstimator(model, criterion, featureSize, labelSize)
.setLearningRate(0.01)
.setBatchSize(6)
val trainedModel = estimator.fit(trainDataframe)
val predictions = trainedModel.transform(testDataframe)

55. Spark Integration
val estimator = new DLEstimator(model, criterion, featureSize, labelSize)
.setLearningRate(0.01)
.setBatchSize(6)
val trainedModel = estimator.fit(trainDataframe)
val predictions = trainedModel.transform(testDataframe)
Estimator
Transformer

56. Interoperability
Your
model
BigDL Torch
Tensor
flowCaffe
Keras

58. Post more job offers on comm-montpellier.slack !
https://bit.ly/comm-mtp
offres qualifiées correctement tant sur le domaine,
les technos que la fourchette salariale. Ou à minima
avec un pitch marrant ;)