Slides from a talk I will give in early 2016 at the Luxembourg Data Science Meetup. Aim is to give an introduction to Apache Spark, from a Machine Learning experts point of view. Based on various other tutorials out there. This will be aimed at non-specialists.

1. Introduction toIntroduction to
Peadar Coyle @springcoil
Luxembourg - Early 2016

2. Aims of this talkAims of this talk
Explain what Spark is.Explain what Spark is.
I'm more a data scientist than an engineer...

3. Who am I?Who am I?
Math and Data nerd
Interested in machine learning and data processing
Speaker at PyData/ PyCons throughout Europe

4. 'Big data' so far'Big data' so far

5. Why care?Why care?
big data analytics in memory
Resilient Distributed Datasets (RDD)
Flexible programming models
complements Hadoop
better performance than Hadoop
https://github.com/springcoil/scalable_ml

6. Who uses it?Who uses it?
Current tech not future tech!Current tech not future tech!

7. Supported LanguagesSupported Languages

8. val data = Array(1, 2, 3, 4, 5)
val distData = sc.parallelize(data)
sc is 'Spark Context
Here is one RDD
CodeCode

9. https://github.com/springcoil/scalable_ml/
package scalable_ml
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.DenseVector
import org.apache.spark.rdd.RDD
import breeze.linalg.{DenseVector => BDV}
import breeze.linalg.{DenseMatrix => BDM}
class LeastSquaresRegression {
def fit(dataset: RDD[LabeledPoint]): DenseVector = {
val features = dataset.map {
_.features
}
val covarianceMatrix: BDM[Double] = features.map { v =>
val x = BDM(v.toArray)
x.t * x
}.reduce(_ + _)
val featuresTimesLabels: BDV[Double] = dataset.map { xy =>
BDV(xy.features.toArray) * xy.label
}.reduce(_ + _)
val weight = covarianceMatrix featuresTimesLabels
new DenseVector(weight.data)
}
}

10. Resilient DistributedResilient Distributed
Datasets (RDD)Datasets (RDD)
Process in parallel
Actions on RDDs = transformations and actions
persistance: Memory, Disk, Memory and Disk

11. Spark EcosystemSpark Ecosystem
Spark streaming
Spark SQL - Really the creation of a data frame
More stuff will come soon...
IBM and others heavily investing in this.

12. Any questions?Any questions?