1. Big Data Processing With
Scala and Spark
Slide 1 www.edureka.Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions co/apache-spark-scala-training

2. Objectives of this Session
What is Big Data?
What is Spark?
Why Spark?
Spark Ecosystem
A note about Scala
Why Scala?
Hello Spark!
For Queries during the session and class recording:
Post on Twitter @edurekaIN: #askEdureka
Post on Facebook /edurekaIN
Slide 2 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

3. Big Data
 Lots of Data (Terabytes or Petabytes)
 Big data is the term for a collection of data sets
so large and complex that it becomes difficult to
process using on-hand database management
tools or traditional data processing applications
 The challenges include capture, curation,
storage, search, sharing, transfer, analysis, and
visualization
cloud
tools
statistics
No SQL
Big Data
compression
support
database
storage
analyze
information
mobile
processing
terabytes
Slide 3 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

4. What is Spark?
 Apache Spark is a general-purpose cluster in-memory computing system
 Provides high-level APIs in Java, Scala and Python, and an optimized engine that supports general execution graphs
 Provides various high level tools like Spark SQL for structured data processing, Mlib for Machine Learning and more..
High Level
APIs
High Level
Tools
More…
Slide 4 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

5. Why Spark?
via YARN
Cluster Manager
 The Spark framework can be deployed through
Apache Mesos, Apache Hadoop via Yarn, or
Spark’s own cluster manager.
Deployment
Slide 5 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

6. Why Spark?
 Spark framework is polyglot – Can be programmed
in several programming languages (Currently
Scala, Java and Python supported).
Polyglot Scala
Slide 6 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

7. Why Spark?
A fully Apache Hive compatible data
warehousing system that can run 100x
faster than Hive.
100x faster than for certain applications.
Slide 7 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

8. Why Spark?
 Provides powerful caching and disk persistence capabilities
 Interactive Data Analysis
 Faster Batch
 Iterative Algorithms
 Real-Time Stream Processing
 Faster Decision-Making
Slide 8 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

9. Spark Community is Super Active!
Slide 9 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

10. Spark Ecosystem
MLLib
(Machine
learning)
Spark Core Engine
Aplha/Pre-alpha
BlindDB
(Approximate
SQL)
Shark
(SQL)
Spark
Streaming
(Streaming)
GraphX
(Graph
Computation)
SparkR
(R on Spark)
Slide 10 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

11. Spark Ecosystem (Contd.)
An approximate
query engine. To
run over Core
Spark Engine.
Used for structured
data. Can run
unmodified hive
queries on existing
Hadoop
deployment.
MLLib
(Machine
learning)
Enables analytical
and interactive
apps for live
streaming data.
Spark Core Engine
Aplha/Pre-alpha
BlindDB
(Approximate
SQL)
Shark
(SQL)
Spark
Streaming
(Streaming)
Graph Computation
engine.
(Similar to Giraph)
GraphX
(Graph
Computation)
Package for R language
to enable R-users to
leverage Spark power
from R shell.
SparkR
(R on Spark)
Machine learning library being built on top of Spark. Provision for support to many
machine learning algorithms with speeds upto 100 times faster than Map-Reduce.
Slide 11 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

12. A Note on Scala
 Scala is a general-purpose programming language designed
to express common programming patterns in a concise,
elegant, and type-safe way
 Scala supports both Object Oriented Programming and
Functional Programming
 Scala is very much in fabric of present and Future Big Data
frameworks like Scalding, Spark, Akka
» All examples of Spark in class will be
covered in Scala
» Scala would be covered before Spark
coverage as part of course!
Slide 12 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

13. Why Scala?
 Scala is a pure object-oriented language. Conceptually, every value is an object and every operation is a
method-call. The language supports advanced component architectures through classes and traits
 Scala is also a functional language. Supports functions, immutable data structures and preference for
immutability over mutation
 Seamlessly integrated with Java
 Being used heavily for future Big data and we developments frameworks like Spark, Akka, Scalding, Play etc
Slide 13 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/apache-spark-scala-training

14. Hello Spark!
Hello Spark!
Slide 14 www.Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions edureka.co/apache-spark-scala-training

15. Questions?
Buy Spark Course at : www.edureka.co
Slide 15 www.edureka.Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions co/apache-spark-scala-training