Introduction to Implicit usage and Implicit resolution in Scala

1. Understanding Implicits in
Scala
https://github.com/shashankgowdal/understanding-implicits

2. ● Shashank L
● Senior Software engineer at Tellius
● Big data consultant and trainer at
datamantra.io
● www.shashankgowda.com

3. Scala
Scala is a modern programming language designed to
express common programming patterns in a concise,
elegant, and type-safe way.

4. Scala is concise
● Do more with less code
● Type inference
● Higher level

5. Scala is concise
● Less boilerplate code

6. What are Implicits?

7. Implicits
● Language feature - not from external framework
● Implicits are next level of making Scala concise
● Scala Implicits allows to omit calling methods or
referencing variables explicitly
● Rely on the compiler to make the connections
● No runtime checks
○ Everything is type checked at compile time
● Enables advanced and elegant architecture design

8. Implicits are not
● Global variables
● Dynamically applied
● Dangerous

9. Implicit conversions in Java
static long sq(long a){
return a*a;
}
sq(10)
sq(10L)
● Java has implicit kind of type casting
● Only language features use them
● No public API to use it

10. Implicits
● Implicit parameters
● Implicit conversions

11. Implicit Parameters

12. Implicit parameters
● Function parameter annotated with the implicit keyword
def multiply (value:Int) (implicit by:Int) = value * by
● Parameter need not be passed explicitly
multiply(10)
● Tell the compiler once what needs to be passed
implicit val multiplier = 2
● Compilation error if implicit value not found
error: could not find implicit value for parameter by: Int
● Developer can be explicit
multiply(10)(3)

13. Implicit parameters
● Implicit can be a val, var or def
implicit def f: Int = if(monday) 2 else 4
● Can only use implicit once in the parameter list, all the parameters
following it will be implicit
def example1(implicit x:Int) //x is implicit
def example2(implicit x:Int, y:Int) //x & y is implicit
def example3(x:Int, implicit y:Int) //won't compile
def example4(x:Int)(implicit y:Int) //y is implicit
def example5(implicit x:Int)(y:Int) //won't compile
def example6(implicit x:Int)(implicit y:Int) //won't compile

14. Dependency injection

15. Dependency injection
● Separates creation of client’s dependencies from client’s
behaviour, allows program to be loosely coupled
● Similar to parameter passing. Dependencies are injected
rather than created inside
● This makes code
○ understandable
○ modular
○ reusable
○ testable

16. Dependency injection in Java
● Spring is the widely used framework in Java for DI
● Popular and well designed framework
● Relies on annotations for injection
● External configuration files is needed
● Dependency injection at runtime
○ Surprises at runtime if something isn’t right
● External library not a language feature

17. ● Scala way to do DI
● Language feature - No need of framework
● User Authentication service with Persistence
● UserRepository - Handles storage
● UserService - Handles authentication and depends on UserRepository for
persistence
● ComponentRegistry - Wires components and instantiates services
com.shashank.implicits.dependency_injection.cakepattern
Dependency injection - Cake pattern
Service
Storage
InMemory MongoDB

18. Dependency injection - Cake pattern
● Pros
○ Compile time checked
○ No need of external configuration files
○ Switch different implementations
○ Create multiple environments
○ Test individual components
● Cons
○ Lot of boilerplate code
○ Cannot fast prototype (requires complete architecture picture)
○ Hard to get it right initially

19. Dependency injection - Implicits
● UserRepository dependency is passed as an implicit parameter
to UserService
● No need of boilerplate code with components, implementations
● Enables faster prototyping with advantages of Cake pattern
● Switch between different implementation of UserRepository by
changing the instantiation of it at once place
com.shashank.implicits.dependency_injection.implicitway

20. Context passing

21. ● Similar to Dependency injection
● Keeping all dependencies in the form of context
● Common enough that simply "not having to pass it
everywhere" is itself a valuable goal
● Scala library uses it to pass around an ExecutionContext
to every code that needs to be run asynchronously.
Context passing

22. ExecutionContext in Future
object Future {
def apply[T](body: =>T)(implicit executor: ExecutionContext): scala.concurrent.Future[T] = {
val runnable = new PromiseCompletingRunnable(body)
executor.prepare.execute(runnable)
runnable.promise.future
}
}
Find first index of first occurrence of the word in a file
implicit val ec: ExecutionContext = scala.concurrent.ExecutionContext.Implicits.global
val firstOccurrence: Future[Int] = Future {
val source = scala.io.Source.fromFile("/very/big/file.txt")
source.toSeq.indexOfSlice("N628DL")
}
com.shashank.implicits.context_passing

23. Implicit Conversion

24. Implicits conversion
● Type conversions
● Extension methods

25. Type conversion
● Type conversion refers to changing an entity of one
datatype to another
● Type conversion can be
○ Explicit
Explicit conversion in some way is called casting
double doubleValue = 10 + (int)10.05;
○ Implicit
Implicit conversion is an automatic conversion by the
compiler
double doubleValue = 10 + 10.60;

26. Type conversion in Scala
● If an object’s type has to be converted from A to B
● Create an implicit conversion function
○ Takes parameter of type A
○ Implement conversion logic to convert to B and return
● Used extensively by Scala
● Conversion of Scala Boolean to Java Boolean object
implicit def boolean2Boolean(x: Boolean) = java.lang.Boolean.valueOf(x)
com.shashank.implicits.datatype_conversions

27. Extension methods
● As a library developer, providing custom methods to a class not
under your control is important
● Add special methods for specific type of your objects
● Spark has special methods for
○ PairRDD - reduceByKey, aggregateByKey
○ DoubleRDD - stats, mean, sum, histogram
● These methods are not in base RDD, It's added as extension
methods through implicits
● PairRDD methods are available only if the RDD type if
key-value pair
● DoubleRDD methods are available only if the RDD type is
Double

28. Extension methods
● If we want extend methods sum, mean for class A
● Create a custom class B with following properties
○ Single-parameter constructor expecting object of type A
○ Implement extension methods using object of type A
class JsonUser(user:User) {
def asJsObject: JsObject ={
JsObject(Map("userId" -> JsString(x.userId),
"email" -> JsString(x.email),
"isAdmin" -> JsBoolean(x.isAdmin)))
}
}
○ Define implicit type conversion to convert from A to B
implicit def toJsonUser(user:User) = new JsonUser(user)
com.shashank.implicits.extending_types

29. Implicit Class
● Syntactic sugar/shorthand for Extension methods
● Instead of creating a class and implicit def for conversion,
it can be combined together into a implicit class
implicit class JsonUser(user:User) {
def asJsObject: JsObject ={
JsObject(Map("userId" -> JsString(x.userId),
"email" -> JsString(x.email),
"isAdmin" -> JsBoolean(x.isAdmin)))
}
}
● Recommended to be in another trait/ class/ object
● Takes only one non-implicit parameter in constructor
com.shashank.implicits.implicit_class

30. Implicit Class
● Late trait implementation
implicit class UserOrdering(user: User) extends Ordered[User] {
override def compare(that: User): Int = user.userId.compare(that.userId)
}
com.shashank.implicits.implicit_class

31. Ad Hoc polymorphism

32. ● Polymorphic function that can be applied to arguments of different
types
def convertToJson(x:Jsonable) :String = x.serialize
● Values passed to x should be of type Jsonable (type checks)
trait Jsonable[T]{
def serialize(t: T): Json
}
● Making String type Jsonable
class StringJsonable(t: String) extends Jsonable[String]{
def serialize() = s""$t""
}
● Calling convertToJson method
convertToJson(StringJsonable("spark_meetup"))
com.shashank.implicits.adhoc_polymorphism.adapter
Ad Hoc polymorphism

33. ● Type system that supports ad hoc polymorphism
● Named after the language feature in Haskell
● Type class is used to provide evidence that a class String
satisfies an interface Jsonable.
def convertToJson[T](x: T)(implicit converter: Jsonable[T]): String = converter.serialize(x)
● Create an implicit converter from String type to Json String
implicit object StringJsonable(t: String) extends Jsonable[String]{
def serialize() = s""$t""
}
● Calling convertToJson method
convertToJson("spark_meetup")
com.shashank.implicits.adhoc_polymorphism.typeclasses
Type classes

34. ● Provides combination of Bridge pattern and adapter pattern
● Third parties can also implement Jsonable support
implicit object DateJsonable(t: Date) extends Jsonable[String]{
private val dateFormat = new SimpleDateFormat("yyyy-MM-dd")
def serialize() = s""${dateFormat.format(t)}""
}
● Calling convertToJson method
convertToJson(new Date())
com.shashank.implicits.adhoc_polymorphism.typeclasses_extension
Type classes

35. ● Represented in the form [T: Bound]
● Plain type parameter T together with an implicit parameter of
type Bound[T]
● Syntactic sugar for Methods expecting TypeClass
def convertToJson[T : Jsonable](x: T): String = x.serialize()
● More useful and clean when need to just pass them to other
methods that use them.
com.shashank.implicits.implicitly
Context bound

36. ● If Context bounds were used for Aesthetic purpose and there is
a need to access the implicit parameter
● implicitly with the implicit type can be used to get the implicit
parameter
def convertToJson[T : Jsonable](x: T): String = implicitly[Jsonable[T]].serialize()
com.shashank.implicits.implicitly
Implicitly

37. Behind the scenes

38. ● Compiler tries to find Implicit when
○ method being called doesn’t exist on the object’s class
○ method being called takes an implicit parameter
● Implicits are resolved at 2 priority levels
○ Resolve scope - Higher priority
○ Extended scope - Lower priority
Implicit resolution

39. ● Implicits defined in the current scope
implicit val n: Int = 5
def add(x: Int)(implicit y: Int) = x + y
add(5) // takes n from the current scope
● Explicit imports
import scala.collection.JavaConversions.mapAsScalaMap
def env = System.getenv() // Java map
val term = env("TERM") // implicit conversion from Java Map to Scala Map
● Wildcard imports
import scala.collection.JavaConversions._
val keySet = System.getenv().keySet() //returns Java Set collection
val sparkHomeSet = keySet.exists(key => key == "SPARK_HOME")
// implicit conversion from Java Set to Scala Set
Resolve scope

40. ● Companion object of the source type
case class User(name:String, score:Double)
object User {
implicit def toJson(user:User):JsObject =
JsObject(Map("user" -> JsString(user.name), "score" -> JsNumber(user.score)))
}
val json:JsObject = User("myname", 30)
● Companion object of the expected type
case class User(name:String, score:Double)
case class JsonUser(json:String)
object JsonUser {
implicit def toJsonUser(user:User):JsonUser =
JsonUser(JsObject(Map("user" -> JsString(user.name), "score" -> JsNumber(user.score))).prettyPrint)
}
val jsonUser:JsonUser = User("myname", 30)
Extended scope

41. ● Implicit Scope of Type arguments
case class User(name:String, score:Double)
object User {
implicit val ord = new Ordering[User] {
def compare(x: User, y: User): Int = implicitly[Ordering[Double]].compare(x.score, y.score)
}
}
val users = List(User("abc", 10), User("efg", 7), User("dcg", 9)).sorted
Extended scope

42. ● Resolve scope implicits has precedence over Extended
scope Implicits
● Multiple implicits in the same scope level is resolved using
Static overloading rule
○ Take the implicit which has more specific type
● If all the implicits have the same precedence according to Static
overloading rule, then it results in a compilation error
com.shashank.implicits.resolution
Implicit conflicts

43. Implicit usage in frameworks

44. ● Scala collection ordering
List(5, 3, 7).sorted
List("a", "x", "b").sorted
def sorted[B](implicit ord: Ordering[B])
def sorted[B](implicit ord: Ordering[B])
def max[U](implicit ord: Ordering[U])
● Scala collection numerical operations
List(5, 3, 7).sum
List(5, 3, 7).product
def product[B](implicit num: Numeric[B])
def sum[B](implicit num: Numeric[B])
Implicit usage in Frameworks

45. ● Scala and Java collection inter compatibility
import collection.JavaConverters._
val jul: java.util.List[Int] = ArrayBuffer(1, 2, 3).asJava
jul.add(4)
val jul = new java.util.ArrayList[Int]()
jul.add(10); jul.add(20)
import scala.collection.JavaConversions._
jul.map(x => x.toDouble)
● Scala and Java data type inter compatibility
val booleanValue:java.lang.Boolean = false
implicit def boolean2Boolean(x: Boolean) = java.lang.Boolean.valueOf(x)
Implicit usage in Frameworks

46. ● Spark Pair RDD functions
val rdd = sc.parallelize(List(("a", 10.0), ("c", 20.0), ("c", 20.0), ("d", 20.0))
rdd.groupByKey
rdd.reduceByKey
● Spark Double RDD functions
val rdd = sc.parallelize(List(1.0, 2.0, 3.0, 5.0, 20.0, 19.02, 19.29, 11.09, 21.0), 2)
rdd.sum
rdd.variance
● Spark Dataset API
import sparkSession.implicits._
val data = sparkSession.read.text("src/main/resources/data.txt").as[String]
def as[U : Encoder]: Dataset[U] = Dataset[U](sparkSession, logicalPlan)
implicit def newStringEncoder: Encoder[String] = Encoders.STRING
Implicit usage in Frameworks

47. ● Spray JSON
Int JsNumber
String JsString
CaseClass JsObject
implicit object IntJsonFormat extends JsonFormat[Int] {
def write(x: Int) = JsNumber(x)
def read(value: JsValue) = value match {
case JsNumber(x) => x.intValue
}
}
def toJson(implicit format: JsonFormat[T]): JsValue = format.write(any)
val score = 100
score.toJson.prettyPrint
case class User(userId:String, email:String, isAdmin:Boolean)
implicit val userFormat = jsonFormat3(User)
user.toJson.prettyPrint
Implicit usage in Frameworks

48. ● Do not overuse
● Avoid wildcard imports
import some.random.class.implicits._
● Don’t convert from one library type to another
● Avoid Implicit conversion which changes semantics
val jul:java.utilList = ...
import scala.collection.JavaConversions._
jul.map(a => a)
import scala.collection.JavaConverters._
jul.asScala.map(a => a)
● Use extended scope over resolve scope
What not to do?

49. ● Resolution rules can be difficult
● Automatic conversions
● Slow compilation
● IDE slows down
● Do not overuse
Beware

50. ● Under the hood of Implicits -
https://www.slideshare.net/DerekWyatt1/scala-implicits-not-to-be-feared
● Under the hood of Implicits -
https://www.youtube.com/watch?v=01zMZVf1ItU&t=12s
● What to leave Implicit by -
https://www.youtube.com/watch?v=Oij5V7LQJsA
● Implicit resolution -
https://stackoverflow.com/questions/5598085/where-does-scala-look-for-impli
cits
References

51. Thank you