SlideShare ist ein Scribd-Unternehmen logo
1 von 39
Downloaden Sie, um offline zu lesen
Functional programming in
Scala
https://github.com/ganeshayadiyala/functional-programming-in-scala
● Ganesha Yadiyala
● Big data consultant at
datamantra.io
● Consult in spark and scala
● ganeshayadiyala@gmail.com
Agenda
● Programming paradigms
● Functional Programming
● Advantages of FP
● History of Scala
● Basics of scala
● Functional Aspect of Scala
Programming Paradigms
Programming Paradigms
● Programming Paradigm is a approach to programming
● It will be based on set of principles or theory
● Different ways of thinking.
● Different types
○ Imperative
○ Functional
Imperative Programming
Imperative programming is a programming paradigm that uses
statements that change a program's state.
Example :
● C#
● C++
● Java
Functional Programming
A programming style that models computation as the evaluation of
expression.
Example:
● Haskell
● Erlang
● Lisp
● Clojure
Expression vs Statement
Expression : evaluated to produce a value without changing state.
Statement : executed to update state.
Example:
Expression -> val isEven = if(x%2 == 0) true else false
Example :
Statement -> boolean isEven = true
if(x%2 == 0)
isEven = true
else
isEven = false
Functional Programming
History of Functional Programming
● Mathematical abstraction Lambda Calculus was the foundation
for functional Programming
● Lambda calculus was developed in 1930s to be used for
functional definition, functional application and recursion.
● It states that any computation can be achieved by sending the
input to some black box, which produces its result.
● This black box is lambda functions, and in FP it’s functions.
● LISP was the first functional programming language. It was
defined by McCarthy in 1958
Imperative vs Functional Programming
Characteristics Imperative Functional
State changes Important Non-existent
Order of execution Important Low importance
Primary flow control
Loops, conditions and method
(function) calls
Function calls
Primary manipulation unit
Instances of structures or
classes
Functions
Characteristics of FP
● Immutable data
● Lazy evaluation
● No side effects
● Referential transparency
● Functions are first class citizens
Immutable data
Mutable Immutable
var collection = [1,2,3,4]
for ( i = 0; i<collection.length; i++) {
collection[i]+=1;
}
Uses loop for updating
collection is updated in place
val collection = [1,2,3,4]
val newCollection = collection.map(value
=> value +1)
Uses transformation for change
Creates a new copy of collection. Leaves
collection intact
Advantages of FP
● Better modularity
● Shorter code
● Increased developer productivity
● Less number of bugs
● Perfect for distributed computing because of immutability
● Easier debugging
Limitations of FP
● Real world programs have to do side effects
● Difficult to understand for the beginner
● Very difficult to debug the nature of lazy evaluation
● Memory intensive
History of Scala
Scala History
● Martin Odersky is the creator
● Curious about compilers from beginning
● Created first compiler in 1995 - Pizza
○ Generics,Function pointers, Case class and pattern matching
● Led to javac compiler and generics eventually
○ Java is a strict language and hard to implement these new ideas
Scala History cont..
● 1999 - Started working on a language
○ Combines Object orientation with functional programming
○ Created Funnel language
○ It was not a user friendly language
○ New languages with less libraries are hard to adopt
● In 2001, design for Scala started, released first in 2003
● Scala is not an extension of Java
● Scala translates into JAVA byte code
● One implementation of .Net compatible version, now out dated
Scala Philosophy
● Growable language
○ Embed new DSL/add new types
● Scalable language
○ Same concepts in small and large applications
● Deep rather than broad
○ Focus on abstraction and composition
Why scala
● Hybrid programming language
● Statically typed
● Runs on the jvm
● Can execute java code
Basics of Scala
Everything is an Object
● Supports object oriented programming
● Including primitive types, everything in scala is an object
Example:
● val x = 10
● x.toString()
Statically Typed Language
● var message = “hello”
● message = “bye”
● message = 100 //compilation error
● def abs(x:Int) : Int = { ??? }
● abs(“100”) //compilation error
Type Inference
● val employeeName : String = “Joy”
● val department = “HR” //Type inferred as String
● val age = 35 //Type inferred as Int
Collections
● List,Set,Map etc..
● Mutable
● Immutable
● Functional API
Example:
val integerList = List(1,2,3)
Functional Aspect of Scala
Operators are also a Function
For example
● val sum = 2 + 3
Is same as,
● val sum = 2.+(3) //because ‘+’ operator here is a function
Functions in Scala
● Higher order function
● Anonymous function
● Currying
● Tail recursion
Defining a Function in Scala
● Defining a function
In Scala -
def sum(x:Int,y:Int) : Int = x+y
In Java -
public int sum(int x,int y){
return x+y;
}
Assigning function to variable
Example :
def add = (x:Int,y:Int) => x+y
add(2,3) //should return 5
Higher Order Functions
Function that takes functions as a parameters or return function as
a result is called higher order functions
Example : com.ganesh.functions.HigherOrderFunctions.scala
Anonymous Function
Scala provides a relatively lightweight syntax for defining
anonymous functions.
Example:
(x: Int) => x + 1
Shorthand for above function is,
new Function1[Int, Int] {
def apply(x: Int): Int = x + 1
}
Currying
Currying allows to turn a function that expects two arguments into a
function that expects only one, and that function returns a function
that expects the second argument. Creating basically a chain of
functions.
Example : com.ganesh.functions.Currying.scala
Implicit Parameters
The final parameter list on a method can be marked implicit, which
means the values will be taken from the context in which they are
called.
● It means that if no value is supplied when called, the compiler
will look for an implicit value and pass it in for you.
● Implicits are useful in defining default implementations
available. When you need a custom implementation, you can
pass one in explicitly.
Example : com.ganesh.implicits.Implicits.scala
Tail Recursion
● Tail recursion is a special kind of recursion where the recursive
call is the very last thing in the function.
● tail-recursive function will only use a single stack frame as
opposed to hundreds or even thousands in case of normal
recursive function.
● Tail recursion is much faster because it executes using single
stack frame
Example : com.ganesh.functions.TailRecursion.scala
Transformation on Collections
We can perform different transformations on collection using
collection API.
Example:
com.ganesh.transformations.SimpleTransformations.scala
com.ganesh.transformations.AdvancedTransformations.scala
Error Handling
Scala does not have checked exceptions like Java, so you can't do
something like this to force a programmer to deal with an
exception.
Example :
com.ganesh.exceptionhandling.ExceptionThrowing.scala
com.ganesh.exceptionhandling.UsingOptions.scala
References
● https://www.coursera.org/learn/progfun1/ - Scala course by
Martin Odersky
● http://www.dipmat.unict.it/~barba/PROG-LANG/PROGRAMMI-
TESTI/READING-MATERIAL/ShortIntroFPprog-lang.htm -
Functional programming and lambda calculus
● Functional Programming in Scala

Weitere ähnliche Inhalte

Was ist angesagt?

Getting Started with Infrastructure as Code
Getting Started with Infrastructure as CodeGetting Started with Infrastructure as Code
Getting Started with Infrastructure as CodeWinWire Technologies Inc
 
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
 Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F... Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...Databricks
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache SparkRahul Jain
 
Apache Spark Fundamentals
Apache Spark FundamentalsApache Spark Fundamentals
Apache Spark FundamentalsZahra Eskandari
 
Elasticsearch for beginners
Elasticsearch for beginnersElasticsearch for beginners
Elasticsearch for beginnersNeil Baker
 
Understanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIsUnderstanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIsDatabricks
 
Introduction to Ansible
Introduction to AnsibleIntroduction to Ansible
Introduction to AnsibleKnoldus Inc.
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explainedconfluent
 
How to Actually Tune Your Spark Jobs So They Work
How to Actually Tune Your Spark Jobs So They WorkHow to Actually Tune Your Spark Jobs So They Work
How to Actually Tune Your Spark Jobs So They WorkIlya Ganelin
 
Understanding Data Partitioning and Replication in Apache Cassandra
Understanding Data Partitioning and Replication in Apache CassandraUnderstanding Data Partitioning and Replication in Apache Cassandra
Understanding Data Partitioning and Replication in Apache CassandraDataStax
 
Spark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student SlidesSpark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student SlidesDatabricks
 
Spring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise PlatformSpring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise PlatformVMware Tanzu
 
Introduction to Node.js
Introduction to Node.jsIntroduction to Node.js
Introduction to Node.jsVikash Singh
 

Was ist angesagt? (20)

Getting Started with Infrastructure as Code
Getting Started with Infrastructure as CodeGetting Started with Infrastructure as Code
Getting Started with Infrastructure as Code
 
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
 Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F... Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
 
Dart programming language
Dart programming languageDart programming language
Dart programming language
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
 
Apache Spark Fundamentals
Apache Spark FundamentalsApache Spark Fundamentals
Apache Spark Fundamentals
 
Elasticsearch for beginners
Elasticsearch for beginnersElasticsearch for beginners
Elasticsearch for beginners
 
DevOps with Kubernetes
DevOps with KubernetesDevOps with Kubernetes
DevOps with Kubernetes
 
Understanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIsUnderstanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIs
 
Introduction to Ansible
Introduction to AnsibleIntroduction to Ansible
Introduction to Ansible
 
Ansible - Introduction
Ansible - IntroductionAnsible - Introduction
Ansible - Introduction
 
Cassandra Database
Cassandra DatabaseCassandra Database
Cassandra Database
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
 
How to Actually Tune Your Spark Jobs So They Work
How to Actually Tune Your Spark Jobs So They WorkHow to Actually Tune Your Spark Jobs So They Work
How to Actually Tune Your Spark Jobs So They Work
 
Introduction to AWS Glue
Introduction to AWS Glue Introduction to AWS Glue
Introduction to AWS Glue
 
Understanding Data Partitioning and Replication in Apache Cassandra
Understanding Data Partitioning and Replication in Apache CassandraUnderstanding Data Partitioning and Replication in Apache Cassandra
Understanding Data Partitioning and Replication in Apache Cassandra
 
Spark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student SlidesSpark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student Slides
 
Spring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise PlatformSpring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise Platform
 
Yaml
YamlYaml
Yaml
 
Terraform
TerraformTerraform
Terraform
 
Introduction to Node.js
Introduction to Node.jsIntroduction to Node.js
Introduction to Node.js
 

Andere mochten auch

Interactive Data Analysis in Spark Streaming
Interactive Data Analysis in Spark StreamingInteractive Data Analysis in Spark Streaming
Interactive Data Analysis in Spark Streamingdatamantra
 
Real time ETL processing using Spark streaming
Real time ETL processing using Spark streamingReal time ETL processing using Spark streaming
Real time ETL processing using Spark streamingdatamantra
 
Introduction to Spark 2.0 Dataset API
Introduction to Spark 2.0 Dataset APIIntroduction to Spark 2.0 Dataset API
Introduction to Spark 2.0 Dataset APIdatamantra
 
Productionalizing a spark application
Productionalizing a spark applicationProductionalizing a spark application
Productionalizing a spark applicationdatamantra
 
Introduction to spark 2.0
Introduction to spark 2.0Introduction to spark 2.0
Introduction to spark 2.0datamantra
 
Building end to end streaming application on Spark
Building end to end streaming application on SparkBuilding end to end streaming application on Spark
Building end to end streaming application on Sparkdatamantra
 
Anatomy of spark catalyst
Anatomy of spark catalystAnatomy of spark catalyst
Anatomy of spark catalystdatamantra
 
Anatomy of Data Frame API : A deep dive into Spark Data Frame API
Anatomy of Data Frame API :  A deep dive into Spark Data Frame APIAnatomy of Data Frame API :  A deep dive into Spark Data Frame API
Anatomy of Data Frame API : A deep dive into Spark Data Frame APIdatamantra
 
Introduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsIntroduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsdatamantra
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured Streamingdatamantra
 
Interactive workflow management using Azkaban
Interactive workflow management using AzkabanInteractive workflow management using Azkaban
Interactive workflow management using Azkabandatamantra
 
Anatomy of Data Source API : A deep dive into Spark Data source API
Anatomy of Data Source API : A deep dive into Spark Data source APIAnatomy of Data Source API : A deep dive into Spark Data source API
Anatomy of Data Source API : A deep dive into Spark Data source APIdatamantra
 
Building scalable rest service using Akka HTTP
Building scalable rest service using Akka HTTPBuilding scalable rest service using Akka HTTP
Building scalable rest service using Akka HTTPdatamantra
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Sparkdatamantra
 
Functional programming in Scala
Functional programming in ScalaFunctional programming in Scala
Functional programming in ScalaDamian Jureczko
 
Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark InternalsPietro Michiardi
 
Platform for Data Scientists
Platform for Data ScientistsPlatform for Data Scientists
Platform for Data Scientistsdatamantra
 
Telco analytics at scale
Telco analytics at scaleTelco analytics at scale
Telco analytics at scaledatamantra
 
Introduction to Spark Streaming
Introduction to Spark StreamingIntroduction to Spark Streaming
Introduction to Spark Streamingdatamantra
 
Evolution of apache spark
Evolution of apache sparkEvolution of apache spark
Evolution of apache sparkdatamantra
 

Andere mochten auch (20)

Interactive Data Analysis in Spark Streaming
Interactive Data Analysis in Spark StreamingInteractive Data Analysis in Spark Streaming
Interactive Data Analysis in Spark Streaming
 
Real time ETL processing using Spark streaming
Real time ETL processing using Spark streamingReal time ETL processing using Spark streaming
Real time ETL processing using Spark streaming
 
Introduction to Spark 2.0 Dataset API
Introduction to Spark 2.0 Dataset APIIntroduction to Spark 2.0 Dataset API
Introduction to Spark 2.0 Dataset API
 
Productionalizing a spark application
Productionalizing a spark applicationProductionalizing a spark application
Productionalizing a spark application
 
Introduction to spark 2.0
Introduction to spark 2.0Introduction to spark 2.0
Introduction to spark 2.0
 
Building end to end streaming application on Spark
Building end to end streaming application on SparkBuilding end to end streaming application on Spark
Building end to end streaming application on Spark
 
Anatomy of spark catalyst
Anatomy of spark catalystAnatomy of spark catalyst
Anatomy of spark catalyst
 
Anatomy of Data Frame API : A deep dive into Spark Data Frame API
Anatomy of Data Frame API :  A deep dive into Spark Data Frame APIAnatomy of Data Frame API :  A deep dive into Spark Data Frame API
Anatomy of Data Frame API : A deep dive into Spark Data Frame API
 
Introduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsIntroduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actors
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured Streaming
 
Interactive workflow management using Azkaban
Interactive workflow management using AzkabanInteractive workflow management using Azkaban
Interactive workflow management using Azkaban
 
Anatomy of Data Source API : A deep dive into Spark Data source API
Anatomy of Data Source API : A deep dive into Spark Data source APIAnatomy of Data Source API : A deep dive into Spark Data source API
Anatomy of Data Source API : A deep dive into Spark Data source API
 
Building scalable rest service using Akka HTTP
Building scalable rest service using Akka HTTPBuilding scalable rest service using Akka HTTP
Building scalable rest service using Akka HTTP
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
 
Functional programming in Scala
Functional programming in ScalaFunctional programming in Scala
Functional programming in Scala
 
Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark Internals
 
Platform for Data Scientists
Platform for Data ScientistsPlatform for Data Scientists
Platform for Data Scientists
 
Telco analytics at scale
Telco analytics at scaleTelco analytics at scale
Telco analytics at scale
 
Introduction to Spark Streaming
Introduction to Spark StreamingIntroduction to Spark Streaming
Introduction to Spark Streaming
 
Evolution of apache spark
Evolution of apache sparkEvolution of apache spark
Evolution of apache spark
 

Ähnlich wie Functional programming in Scala

Understanding Implicits in Scala
Understanding Implicits in ScalaUnderstanding Implicits in Scala
Understanding Implicits in Scaladatamantra
 
Functional Programming in Ruby
Functional Programming in RubyFunctional Programming in Ruby
Functional Programming in RubyAlex Teut
 
Functional Programming in JavaScript & ESNext
Functional Programming in JavaScript & ESNextFunctional Programming in JavaScript & ESNext
Functional Programming in JavaScript & ESNextUnfold UI
 
Python functional programming
Python functional programmingPython functional programming
Python functional programmingGeison Goes
 
Ruby Functional Programming
Ruby Functional ProgrammingRuby Functional Programming
Ruby Functional ProgrammingGeison Goes
 
Introduction to functional programming
Introduction to functional programmingIntroduction to functional programming
Introduction to functional programmingKonrad Szydlo
 
Functional Programming.pptx
Functional Programming.pptxFunctional Programming.pptx
Functional Programming.pptxKarthickT28
 
(3) cpp procedural programming
(3) cpp procedural programming(3) cpp procedural programming
(3) cpp procedural programmingNico Ludwig
 
An Introduction to Scala - Blending OO and Functional Paradigms
An Introduction to Scala - Blending OO and Functional ParadigmsAn Introduction to Scala - Blending OO and Functional Paradigms
An Introduction to Scala - Blending OO and Functional ParadigmsMiles Sabin
 
An introduction to Object Oriented JavaScript
An introduction to Object Oriented JavaScriptAn introduction to Object Oriented JavaScript
An introduction to Object Oriented JavaScriptTO THE NEW | Technology
 
Scala Talk at FOSDEM 2009
Scala Talk at FOSDEM 2009Scala Talk at FOSDEM 2009
Scala Talk at FOSDEM 2009Martin Odersky
 
Develop realtime web with Scala and Xitrum
Develop realtime web with Scala and XitrumDevelop realtime web with Scala and Xitrum
Develop realtime web with Scala and XitrumNgoc Dao
 
PL Lecture 01 - preliminaries
PL Lecture 01 - preliminariesPL Lecture 01 - preliminaries
PL Lecture 01 - preliminariesSchwannden Kuo
 
Reactive Software Systems
Reactive Software SystemsReactive Software Systems
Reactive Software SystemsBehrad Zari
 
Introduction of Object Oriented JavaScript
Introduction of Object Oriented JavaScriptIntroduction of Object Oriented JavaScript
Introduction of Object Oriented JavaScriptNexThoughts Technologies
 

Ähnlich wie Functional programming in Scala (20)

Understanding Implicits in Scala
Understanding Implicits in ScalaUnderstanding Implicits in Scala
Understanding Implicits in Scala
 
Functional Programming in Ruby
Functional Programming in RubyFunctional Programming in Ruby
Functional Programming in Ruby
 
Functional Programming in JavaScript & ESNext
Functional Programming in JavaScript & ESNextFunctional Programming in JavaScript & ESNext
Functional Programming in JavaScript & ESNext
 
Python functional programming
Python functional programmingPython functional programming
Python functional programming
 
Yes scala can!
Yes scala can!Yes scala can!
Yes scala can!
 
Ruby Functional Programming
Ruby Functional ProgrammingRuby Functional Programming
Ruby Functional Programming
 
Introduction to functional programming
Introduction to functional programmingIntroduction to functional programming
Introduction to functional programming
 
Functional Programming.pptx
Functional Programming.pptxFunctional Programming.pptx
Functional Programming.pptx
 
Java 8
Java 8Java 8
Java 8
 
(3) cpp procedural programming
(3) cpp procedural programming(3) cpp procedural programming
(3) cpp procedural programming
 
An Introduction to Scala - Blending OO and Functional Paradigms
An Introduction to Scala - Blending OO and Functional ParadigmsAn Introduction to Scala - Blending OO and Functional Paradigms
An Introduction to Scala - Blending OO and Functional Paradigms
 
An introduction to Object Oriented JavaScript
An introduction to Object Oriented JavaScriptAn introduction to Object Oriented JavaScript
An introduction to Object Oriented JavaScript
 
Scala-Ls1
Scala-Ls1Scala-Ls1
Scala-Ls1
 
Scala Talk at FOSDEM 2009
Scala Talk at FOSDEM 2009Scala Talk at FOSDEM 2009
Scala Talk at FOSDEM 2009
 
Develop realtime web with Scala and Xitrum
Develop realtime web with Scala and XitrumDevelop realtime web with Scala and Xitrum
Develop realtime web with Scala and Xitrum
 
Dart workshop
Dart workshopDart workshop
Dart workshop
 
PL Lecture 01 - preliminaries
PL Lecture 01 - preliminariesPL Lecture 01 - preliminaries
PL Lecture 01 - preliminaries
 
Reactive Software Systems
Reactive Software SystemsReactive Software Systems
Reactive Software Systems
 
Introduction of Object Oriented JavaScript
Introduction of Object Oriented JavaScriptIntroduction of Object Oriented JavaScript
Introduction of Object Oriented JavaScript
 
Scala functions
Scala functionsScala functions
Scala functions
 

Mehr von datamantra

Multi Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and TelliusMulti Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and Telliusdatamantra
 
State management in Structured Streaming
State management in Structured StreamingState management in Structured Streaming
State management in Structured Streamingdatamantra
 
Spark on Kubernetes
Spark on KubernetesSpark on Kubernetes
Spark on Kubernetesdatamantra
 
Understanding transactional writes in datasource v2
Understanding transactional writes in  datasource v2Understanding transactional writes in  datasource v2
Understanding transactional writes in datasource v2datamantra
 
Introduction to Datasource V2 API
Introduction to Datasource V2 APIIntroduction to Datasource V2 API
Introduction to Datasource V2 APIdatamantra
 
Exploratory Data Analysis in Spark
Exploratory Data Analysis in SparkExploratory Data Analysis in Spark
Exploratory Data Analysis in Sparkdatamantra
 
Core Services behind Spark Job Execution
Core Services behind Spark Job ExecutionCore Services behind Spark Job Execution
Core Services behind Spark Job Executiondatamantra
 
Optimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsOptimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsdatamantra
 
Structured Streaming with Kafka
Structured Streaming with KafkaStructured Streaming with Kafka
Structured Streaming with Kafkadatamantra
 
Understanding time in structured streaming
Understanding time in structured streamingUnderstanding time in structured streaming
Understanding time in structured streamingdatamantra
 
Spark stack for Model life-cycle management
Spark stack for Model life-cycle managementSpark stack for Model life-cycle management
Spark stack for Model life-cycle managementdatamantra
 
Productionalizing Spark ML
Productionalizing Spark MLProductionalizing Spark ML
Productionalizing Spark MLdatamantra
 
Introduction to Structured streaming
Introduction to Structured streamingIntroduction to Structured streaming
Introduction to Structured streamingdatamantra
 
Building real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark StreamingBuilding real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark Streamingdatamantra
 
Testing Spark and Scala
Testing Spark and ScalaTesting Spark and Scala
Testing Spark and Scaladatamantra
 
Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2datamantra
 
Migrating to spark 2.0
Migrating to spark 2.0Migrating to spark 2.0
Migrating to spark 2.0datamantra
 
Scalable Spark deployment using Kubernetes
Scalable Spark deployment using KubernetesScalable Spark deployment using Kubernetes
Scalable Spark deployment using Kubernetesdatamantra
 
Anatomy of Spark SQL Catalyst - Part 2
Anatomy of Spark SQL Catalyst - Part 2Anatomy of Spark SQL Catalyst - Part 2
Anatomy of Spark SQL Catalyst - Part 2datamantra
 

Mehr von datamantra (19)

Multi Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and TelliusMulti Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and Tellius
 
State management in Structured Streaming
State management in Structured StreamingState management in Structured Streaming
State management in Structured Streaming
 
Spark on Kubernetes
Spark on KubernetesSpark on Kubernetes
Spark on Kubernetes
 
Understanding transactional writes in datasource v2
Understanding transactional writes in  datasource v2Understanding transactional writes in  datasource v2
Understanding transactional writes in datasource v2
 
Introduction to Datasource V2 API
Introduction to Datasource V2 APIIntroduction to Datasource V2 API
Introduction to Datasource V2 API
 
Exploratory Data Analysis in Spark
Exploratory Data Analysis in SparkExploratory Data Analysis in Spark
Exploratory Data Analysis in Spark
 
Core Services behind Spark Job Execution
Core Services behind Spark Job ExecutionCore Services behind Spark Job Execution
Core Services behind Spark Job Execution
 
Optimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsOptimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloads
 
Structured Streaming with Kafka
Structured Streaming with KafkaStructured Streaming with Kafka
Structured Streaming with Kafka
 
Understanding time in structured streaming
Understanding time in structured streamingUnderstanding time in structured streaming
Understanding time in structured streaming
 
Spark stack for Model life-cycle management
Spark stack for Model life-cycle managementSpark stack for Model life-cycle management
Spark stack for Model life-cycle management
 
Productionalizing Spark ML
Productionalizing Spark MLProductionalizing Spark ML
Productionalizing Spark ML
 
Introduction to Structured streaming
Introduction to Structured streamingIntroduction to Structured streaming
Introduction to Structured streaming
 
Building real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark StreamingBuilding real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark Streaming
 
Testing Spark and Scala
Testing Spark and ScalaTesting Spark and Scala
Testing Spark and Scala
 
Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2
 
Migrating to spark 2.0
Migrating to spark 2.0Migrating to spark 2.0
Migrating to spark 2.0
 
Scalable Spark deployment using Kubernetes
Scalable Spark deployment using KubernetesScalable Spark deployment using Kubernetes
Scalable Spark deployment using Kubernetes
 
Anatomy of Spark SQL Catalyst - Part 2
Anatomy of Spark SQL Catalyst - Part 2Anatomy of Spark SQL Catalyst - Part 2
Anatomy of Spark SQL Catalyst - Part 2
 

Kürzlich hochgeladen

April 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's AnalysisApril 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's Analysismanisha194592
 
Halmar dropshipping via API with DroFx
Halmar  dropshipping  via API with DroFxHalmar  dropshipping  via API with DroFx
Halmar dropshipping via API with DroFxolyaivanovalion
 
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfSchema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfLars Albertsson
 
Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...shambhavirathore45
 
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Valters Lauzums
 
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call me @ 9892124323  Cheap Rate Call Girls in Vashi with Real Photo 100% SecureCall me @ 9892124323  Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% SecurePooja Nehwal
 
VidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptxVidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptxolyaivanovalion
 
Mature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptxMature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptxolyaivanovalion
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfMarinCaroMartnezBerg
 
Log Analysis using OSSEC sasoasasasas.pptx
Log Analysis using OSSEC sasoasasasas.pptxLog Analysis using OSSEC sasoasasasas.pptx
Log Analysis using OSSEC sasoasasasas.pptxJohnnyPlasten
 
Week-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interactionWeek-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interactionfulawalesam
 
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al BarshaAl Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al BarshaAroojKhan71
 
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfMarket Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfRachmat Ramadhan H
 
Generative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusGenerative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusTimothy Spann
 
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...amitlee9823
 
100-Concepts-of-AI by Anupama Kate .pptx
100-Concepts-of-AI by Anupama Kate .pptx100-Concepts-of-AI by Anupama Kate .pptx
100-Concepts-of-AI by Anupama Kate .pptxAnupama Kate
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...amitlee9823
 

Kürzlich hochgeladen (20)

April 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's AnalysisApril 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's Analysis
 
Halmar dropshipping via API with DroFx
Halmar  dropshipping  via API with DroFxHalmar  dropshipping  via API with DroFx
Halmar dropshipping via API with DroFx
 
Schema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdfSchema on read is obsolete. Welcome metaprogramming..pdf
Schema on read is obsolete. Welcome metaprogramming..pdf
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
 
Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...Determinants of health, dimensions of health, positive health and spectrum of...
Determinants of health, dimensions of health, positive health and spectrum of...
 
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
 
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call me @ 9892124323  Cheap Rate Call Girls in Vashi with Real Photo 100% SecureCall me @ 9892124323  Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
 
VidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptxVidaXL dropshipping via API with DroFx.pptx
VidaXL dropshipping via API with DroFx.pptx
 
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
Mature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptxMature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptx
 
FESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdfFESE Capital Markets Fact Sheet 2024 Q1.pdf
FESE Capital Markets Fact Sheet 2024 Q1.pdf
 
Log Analysis using OSSEC sasoasasasas.pptx
Log Analysis using OSSEC sasoasasasas.pptxLog Analysis using OSSEC sasoasasasas.pptx
Log Analysis using OSSEC sasoasasasas.pptx
 
Week-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interactionWeek-01-2.ppt BBB human Computer interaction
Week-01-2.ppt BBB human Computer interaction
 
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al BarshaAl Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
 
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfMarket Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdf
 
Generative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusGenerative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and Milvus
 
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
 
100-Concepts-of-AI by Anupama Kate .pptx
100-Concepts-of-AI by Anupama Kate .pptx100-Concepts-of-AI by Anupama Kate .pptx
100-Concepts-of-AI by Anupama Kate .pptx
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
 

Functional programming in Scala

  • 2. ● Ganesha Yadiyala ● Big data consultant at datamantra.io ● Consult in spark and scala ● ganeshayadiyala@gmail.com
  • 3. Agenda ● Programming paradigms ● Functional Programming ● Advantages of FP ● History of Scala ● Basics of scala ● Functional Aspect of Scala
  • 5. Programming Paradigms ● Programming Paradigm is a approach to programming ● It will be based on set of principles or theory ● Different ways of thinking. ● Different types ○ Imperative ○ Functional
  • 6. Imperative Programming Imperative programming is a programming paradigm that uses statements that change a program's state. Example : ● C# ● C++ ● Java
  • 7. Functional Programming A programming style that models computation as the evaluation of expression. Example: ● Haskell ● Erlang ● Lisp ● Clojure
  • 8. Expression vs Statement Expression : evaluated to produce a value without changing state. Statement : executed to update state. Example: Expression -> val isEven = if(x%2 == 0) true else false
  • 9. Example : Statement -> boolean isEven = true if(x%2 == 0) isEven = true else isEven = false
  • 11. History of Functional Programming ● Mathematical abstraction Lambda Calculus was the foundation for functional Programming ● Lambda calculus was developed in 1930s to be used for functional definition, functional application and recursion. ● It states that any computation can be achieved by sending the input to some black box, which produces its result. ● This black box is lambda functions, and in FP it’s functions. ● LISP was the first functional programming language. It was defined by McCarthy in 1958
  • 12. Imperative vs Functional Programming Characteristics Imperative Functional State changes Important Non-existent Order of execution Important Low importance Primary flow control Loops, conditions and method (function) calls Function calls Primary manipulation unit Instances of structures or classes Functions
  • 13. Characteristics of FP ● Immutable data ● Lazy evaluation ● No side effects ● Referential transparency ● Functions are first class citizens
  • 14. Immutable data Mutable Immutable var collection = [1,2,3,4] for ( i = 0; i<collection.length; i++) { collection[i]+=1; } Uses loop for updating collection is updated in place val collection = [1,2,3,4] val newCollection = collection.map(value => value +1) Uses transformation for change Creates a new copy of collection. Leaves collection intact
  • 15. Advantages of FP ● Better modularity ● Shorter code ● Increased developer productivity ● Less number of bugs ● Perfect for distributed computing because of immutability ● Easier debugging
  • 16. Limitations of FP ● Real world programs have to do side effects ● Difficult to understand for the beginner ● Very difficult to debug the nature of lazy evaluation ● Memory intensive
  • 18. Scala History ● Martin Odersky is the creator ● Curious about compilers from beginning ● Created first compiler in 1995 - Pizza ○ Generics,Function pointers, Case class and pattern matching ● Led to javac compiler and generics eventually ○ Java is a strict language and hard to implement these new ideas
  • 19. Scala History cont.. ● 1999 - Started working on a language ○ Combines Object orientation with functional programming ○ Created Funnel language ○ It was not a user friendly language ○ New languages with less libraries are hard to adopt ● In 2001, design for Scala started, released first in 2003 ● Scala is not an extension of Java ● Scala translates into JAVA byte code ● One implementation of .Net compatible version, now out dated
  • 20. Scala Philosophy ● Growable language ○ Embed new DSL/add new types ● Scalable language ○ Same concepts in small and large applications ● Deep rather than broad ○ Focus on abstraction and composition
  • 21. Why scala ● Hybrid programming language ● Statically typed ● Runs on the jvm ● Can execute java code
  • 23. Everything is an Object ● Supports object oriented programming ● Including primitive types, everything in scala is an object Example: ● val x = 10 ● x.toString()
  • 24. Statically Typed Language ● var message = “hello” ● message = “bye” ● message = 100 //compilation error ● def abs(x:Int) : Int = { ??? } ● abs(“100”) //compilation error
  • 25. Type Inference ● val employeeName : String = “Joy” ● val department = “HR” //Type inferred as String ● val age = 35 //Type inferred as Int
  • 26. Collections ● List,Set,Map etc.. ● Mutable ● Immutable ● Functional API Example: val integerList = List(1,2,3)
  • 28. Operators are also a Function For example ● val sum = 2 + 3 Is same as, ● val sum = 2.+(3) //because ‘+’ operator here is a function
  • 29. Functions in Scala ● Higher order function ● Anonymous function ● Currying ● Tail recursion
  • 30. Defining a Function in Scala ● Defining a function In Scala - def sum(x:Int,y:Int) : Int = x+y In Java - public int sum(int x,int y){ return x+y; }
  • 31. Assigning function to variable Example : def add = (x:Int,y:Int) => x+y add(2,3) //should return 5
  • 32. Higher Order Functions Function that takes functions as a parameters or return function as a result is called higher order functions Example : com.ganesh.functions.HigherOrderFunctions.scala
  • 33. Anonymous Function Scala provides a relatively lightweight syntax for defining anonymous functions. Example: (x: Int) => x + 1 Shorthand for above function is, new Function1[Int, Int] { def apply(x: Int): Int = x + 1 }
  • 34. Currying Currying allows to turn a function that expects two arguments into a function that expects only one, and that function returns a function that expects the second argument. Creating basically a chain of functions. Example : com.ganesh.functions.Currying.scala
  • 35. Implicit Parameters The final parameter list on a method can be marked implicit, which means the values will be taken from the context in which they are called. ● It means that if no value is supplied when called, the compiler will look for an implicit value and pass it in for you. ● Implicits are useful in defining default implementations available. When you need a custom implementation, you can pass one in explicitly. Example : com.ganesh.implicits.Implicits.scala
  • 36. Tail Recursion ● Tail recursion is a special kind of recursion where the recursive call is the very last thing in the function. ● tail-recursive function will only use a single stack frame as opposed to hundreds or even thousands in case of normal recursive function. ● Tail recursion is much faster because it executes using single stack frame Example : com.ganesh.functions.TailRecursion.scala
  • 37. Transformation on Collections We can perform different transformations on collection using collection API. Example: com.ganesh.transformations.SimpleTransformations.scala com.ganesh.transformations.AdvancedTransformations.scala
  • 38. Error Handling Scala does not have checked exceptions like Java, so you can't do something like this to force a programmer to deal with an exception. Example : com.ganesh.exceptionhandling.ExceptionThrowing.scala com.ganesh.exceptionhandling.UsingOptions.scala
  • 39. References ● https://www.coursera.org/learn/progfun1/ - Scala course by Martin Odersky ● http://www.dipmat.unict.it/~barba/PROG-LANG/PROGRAMMI- TESTI/READING-MATERIAL/ShortIntroFPprog-lang.htm - Functional programming and lambda calculus ● Functional Programming in Scala