With the massive amount of increase in big data technologies today, it is becoming very important to use the right tool for every process. The process can be anything like Data ingestion, Data processing, Data retrieval, Data Storage, etc. Today we are going to focus on one of those popular big data technologies i.e., Apache Spark. As we all know the Spark API like RDD and data frames and these are best suited with big data but some of the things that we do in Spark which makes it slow or inefficient. Somethings that we misconfigured or use in a wrong way, we'll be talking about those in today's session. After that, we are going to unravel the cases where choosing Spark as your Big Data technology would be the best choice you could make. Also, we are going to consider the cases where Spark would be a disastrous choice for your project and should be avoided

1. Ramandeep Kaur
Anuj Saxena
How Spark may
help you with
your code?

2. Agenda
➢ What is Spark and its Architecture
➢ Spark- Yay/Nay?
➢ Case study

3. What is Apache Spark?
open-source
distributed
general-purpose
cluster-computing framework

4. Apache Spark Architecture

5. Big Data Architecture

7. Lorem Ipsum comes from section Contrary to
popular belief, Lorem Ipsum is not simply random
text.

11. DAG

20. ETL of a large amount
of data

21. Analyzing a large set of
data files

22. Machine Learning &
Data Science to a large
dataset

23. Connecting
BI/Visualization tools

25. Frequent Inserts

26. Frequent inserts

27. Frequent inserts

28. Random Access

29. Random Access

30. Random Access

31. External Reporting
w/load

33. External Reporting
w/load

34. Searching Content
w/load

35. Searching Content
w/load

37. Record count 10 100 1000 10000 100000
Spark 1410 millis 1997 millis 1839 millis 1977 millis 3063 millis
Scala 42 millis 108 millis 623 millis 2882 millis 15375 millis

45. Thank You!