Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Writing DSL with Applicative Functors
1. Writing DSL with
Applicative Functors
David Galichet
Freelance functional programmer
!
twitter: @dgalichet
2. Content normalization
• We want to parse heterogeneous data formats
(CSV, XML …) and transform them to a pivot format
(Scala object in our case)
• The transformation should be described as a DSL
• This DSL must be simple to use, and enable any
kinds of data transformations or verifications
3. Expected DSL format
val reader = ( !
Pick(0).as[String].map(_.capitalize) and !
Pick(1).as[Date].check(_.after(now())) !
).reduce(FutureEvent)!
!
reader("PSUG; 21/08/2014") // returns Success(FutureEvent("PSUG",
date))
Inspired by Play2 Json API
4. tag: step1
Conventions & material
• Code is available on Github : https://github.com/
dgalichet/PsugDSLWritingWithApplicative
• Code revision is define on the top of any slides
including source code (just checkout the specified
tag)
5. tag: step1
Reading single entry
• We have a CSV line and we want to read one column
• We will introduce several abstractions:
• Picker: fetch a data from CSV or XML
• Result: either a Success or a Failure
• Converter: convert value from Picker to Reader
• Reader: container with methods to process its
content
6. tag: step1
Introducing Picker
case class Picker(p: String => Result[String]) {!
def as[T](implicit c: Converter[T]): Reader[T] =
c.convert(p)!
}!
!
object CsvPicker {!
def apply[T](i: Int)(implicit separator: Char): Picker =
Picker { s: String =>!
val elems = s.trim.split(separator)!
if (i > 0 && elems.size > i) Success(elems(i).trim)!
else Failure(s"No column ${i} for ${s}")!
}!
}
7. tag: step1
Introducing Picker
case class Picker(p: String => Result[String]) {!
def as[T](implicit c: Converter[T]): Reader[T] =
c.convert(p)!
}!
!
object CsvPicker {!
def apply[T](i: Int)(implicit separator: Char): Picker =
Picker { s: String =>!
val elems = s.trim.split(separator)!
if (i > 0 && elems.size > i) Success(elems(i).trim)!
else Failure(s"No column ${i} for ${s}")!
}!
}
Picker wraps a function from String to Result
8. The Result
tag: step1
sealed trait Result[+T]
case class Success[T](t: T) extends Result[T]
case class Failure(error: String) extends Result[Nothing]!
9. The Converter
tag: step1
trait Converter[T] {
def convert(p: String => Result[String]): Reader[T]
}!
!
object Converter {!
implicit val string2StringConverter = new Converter[String] {!
override def convert(p: String => Result[String]) = Reader[String]
(p)!
// See code on Github for more converters!
}
10. The Converter
tag: step1
trait Converter[T] {
def convert(p: String => Result[String]): Reader[T]
}!
!
Convert the content of the Picker to a Reader
!
object Converter {!
implicit val string2StringConverter = new Converter[String] {!
override def convert(p: String => Result[String]) = Reader[String]
(p)!
// See code on Github for more converters!
}
11. The Reader
tag: step1
case class Reader[O](p: String => Result[O]) {
def apply(s: String): Result[O] = p(s)
}
A Reader doesn’t contain a value but a process to
transform original data (CSV line or XML) to a
Result
13. tag: step2
Enhancing the Reader
• The first defines a very simple Reader. We must
add a method to combine two instances of Reader
• We will also enhance Failure to store multiple
error messages
14. tag: step2
Enhancing the Reader
case class Reader[O](p: String => Result[O]) {!
def apply(s: String): Result[O] = p(s)!
!
def and[O2](r2: Reader[O2]): Reader[(O, O2)] = Reader { s: String =>!
(p(s), r2.p(s)) match {!
case (Success(s1), Success(s2)) => Success((s1, s2))!
case (Success(_), Failure(f)) => Failure(f)!
case (Failure(f), Success(_)) => Failure(f)!
case (Failure(f1), Failure(f2)) => Failure(f1 ++ f2)!
}!
}!
def map[T](f: O => T): Reader[T] = Reader { s: String =>!
p(s) match {!
case Success(o) => Success(f(o))!
case f: Failure => f!
}!
}!
def reduce[T] = map[T] _ // alias for map!
}
15. tag: step2
Enhancing Result type
sealed trait Result[+T]!
case class Success[T](t: T) extends Result[T]!
case class Failure(error: NonEmptyList[String]) extends
Result[Nothing]!
!
object Failure {!
def apply(s: String): Failure = Failure(NEL(s))!
}!
!
case class NonEmptyList[T](head: T, tail: List[T]) {
def toList = head::tail
def ++(l2: NonEmptyList[T]): NonEmptyList[T] = NonEmptyList(head,
tail ++ l2.toList)
}
object NEL {
def apply[T](h: T, t: T*) = NonEmptyList(h, t.toList)
}
16. Usage sample
tag: step2
implicit val separator = ';'
implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")
import Converter.string2StringConverter
import Converter.string2DateConverter!
!
val reader = (
CsvPicker(1).as[String] and
CsvPicker(2).as[Date]
).reduce { case (n, d) => FutureEvent(n, d) }!
reader("foo;bar;12/10/2014") === Success(FutureEvent("bar",
dtFormatter.parse("12/10/2014")))!
!
case class FutureEvent(name: String, dt: Date)
17. tag: step2
Usability problem
• The use of reduce (or map) method to transform a
Reader[(0, 02)] into an instance of
Reader[FutureEvent] for example is quite
verbose
• This will be even more verbose for instances of
Reader[(0, (02, 03))]
• We want the API to automatically bind tuple
elements to a constructor as we can encounter in
Play2 Json API
18. tag: step3
Applicative functors
• To tackle our problem, we will use Applicative
Functors and play2 functional library (and
especially FunctionalBuilder)
• This approach is inspired by @sadache (Sadek
Drobi) article https://gist.github.com/sadache/
3646092
• An Applicative Functor is a Type Class relying on
ad-hoc polymorphism to extends a Class with some
properties
• Play2 functional library (or Scalaz) provides
mechanism to compose Applicatives in a smart way
19. tag: step3
Applicative functors
M is an Applicative Functor if there exists the following methods :
def pure[A](a: A): M[A]
def map[A, B](m: M[A], f: A => B): M[B]
def apply[A, B](mf: M[A => B], ma: M[A]): M[B]!
with the following Laws :
• Identity: apply(pure(identity), ma) === ma where ma is an Applicative M[A]
• Homomorphism: apply(pure(f), pure(a)) === pure(f(a)) where f: A =>
B and a an instance of A
• Interchange: mf if an instance of M[A => B]
apply(mf, pure(a)) === apply(pure {(g: A => B) => g(a)}, mf)!
• Composition: map(ma, f) === apply(pure(f), ma)
20. tag: step3
Applicative functors
trait Applicative[M[_]] {
def pure[A](a: A): M[A]
def map[A, B](m: M[A], f: A => B): M[B]
def apply[A, B](mf: M[A => B], ma: M[A]): M[B]
} Applicative is an Higher Kinded type
(parameterized with M that take a single type parameter)
21. tag: step3
Reader is an Applicative
case class Reader[O](p: String => Result[O]) {
def apply(s: String): Result[O] = p(s)!
def map[T](f: O => T): Reader[T] = Reader { s: String =>
p(s) match {
case Success(o) => Success(f(o))
case f: Failure => f
}
}!
}!
object Reader {
def map2[O, O1, O2](r1: Reader[O1], r2: Reader[O2])(f: (O1, O2) =>
O): Reader[O] = Reader { s: String =>
(r1.p(s), r2.p(s)) match {
case (Success(s1), Success(s2)) => Success(f(s1, s2))
case (Success(_), Failure(e)) => Failure(e)
case (Failure(e), Success(_)) => Failure(e)
case (Failure(e1), Failure(e2)) => Failure(e1 ++ e2)
}
} …!
}
22. tag: step3
Reader is an Applicative
object Reader {
… // map2
implicit val readerIsAFunctor: Functor[Reader] = new
Functor[Reader] {
override def fmap[A, B](m: Reader[A], f: (A) => B) = m.map(f)
}
implicit val readerIsAnApplicative: Applicative[Reader] = new
Applicative[Reader] {
override def pure[A](a: A) = Reader { _ => Success(a) }
override def apply[A, B](mf: Reader[A => B], ma: Reader[A]) =
map2(mf, ma)((f, a) => f(a))
override def map[A, B](m: Reader[A], f: A => B) = m.map(f)
}
}
23. Usage sample
tag: step3
import Converter.string2StringConverter
import Converter.string2DateConverter!
import play.api.libs.functional.syntax._
import Reader.readerIsAnApplicative!
!
implicit val separator = ';'
implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")!
!
val reader = (
CsvPicker(1).as[String] and
CsvPicker(2).as[Date]
)(FutureEvent) // here we use CanBuild2.apply
reader("foo;bar;12/10/2014") === Success(FutureEvent("bar",
dtFormatter.parse("12/10/2014")))!
24. Usage sample
tag: step3
(errors accumulation)
import Converter.string2StringConverter
import Converter.string2DateConverter!
import play.api.libs.functional.syntax._
import Reader.readerIsAnApplicative!
!
implicit val separator = ';'
implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")!
!
val reader = (
CsvPicker(1).as[Int] and
CsvPicker(2).as[Date]
)((_, _))
reader(List("foo", "not a number", "not a date")) === Failure(NEL(!
"Unable to format 'not a number' as Int", !
"Unable to format 'not a date' as Date"))!
25. Benefits
tag: step3
• Making Reader an Applicative Functor give ability
to combine efficiently instances of Reader
• Due to Applicative properties, we still accumulate
errors
• Play2 functional builder give us a clean syntax to
define our DSL
26. tag: step4
Introducing XML Picker
case class Picker(p: String => Result[String]) {
def as[T](implicit c: Converter[T]): Reader[T] = c.convert(p)
}!
!
object XmlPicker {
def apply[T](query: Elem => NodeSeq): Picker = Picker { s: String =>
try {
val xml = XML.loadString(s)
Success(query(xml).text)
} catch {
case e: Exception => Failure(e.getMessage)
}
}
}!
27. Usage sample
tag: step4
import play.api.libs.functional.syntax._
import Reader.readerIsAnApplicative!
import Converter._
implicit val dF = new SimpleDateFormat("dd/MM/yyyy")!
val xml = """
<company name="Dupont and Co">
<owner>
<person firstname="jean" lastname="dupont" birthdate="11/03/1987"/>
</owner>
</company>"""
val r = (
XmlPicker(_ "person" "@firstname").as[String] and
XmlPicker(_ "person" "@lastname").as[String] and
XmlPicker(_ "person" "@birthdate").as[Date]
)(Person)!
r(xml) === Success(Person("jean","dupont",dF.parse("11/03/1987")))
case class Person(firstname: String, lastname: String, birthDt: Date)
28. tag: step4
Implementation problem
• The Reader[O] takes a type argument for the
output. The input is always a String
• With this implementation, an XML content will be
parsed (with XML.load) as many times as we use
XmlPicker. This will cause unnecessary overhead
• We will have the same issue (with lower overhead)
with our CsvPicker
29. tag: step5
Introducing Reader[I, 0]
To resolve this problem, we will modify Reader to
take a type parameter for the input
30. tag: step5
Introducing Reader[I, 0]
case class Reader[I, O](p: I => Result[O]) {
def apply(s: I): Result[O] = p(s)
def map[T](f: O => T): Reader[I, T] = Reader { s: I =>
p(s) match {
case Success(o) => Success(f(o))
case f: Failure => f
}
}!
}!
object Reader {
def map2[I, O, O1, O2](r1: Reader[I, O1], r2: Reader[I, O2])(f: (O1,
O2) => O): Reader[I, O] = Reader { s: I =>
(r1.p(s), r2.p(s)) match {
case (Success(s1), Success(s2)) => Success(f(s1, s2))
case (Success(_), Failure(e)) => Failure(e)
case (Failure(e), Success(_)) => Failure(e)
case (Failure(e1), Failure(e2)) => Failure(e1 ++ e2)
}
}
32. tag: step5
What are Type lambdas ?
If we go back to Applicative definition, we can see that it’s
an Higher Kinded type (same with Functor) :
!
trait Applicative[M[_]] { … } // Applicative accept
parameter M that take itself any type as parameter!
!
Our problem is that Reader[I, 0] takes two parameters
but Applicative[M[_]] accept types M with only one
parameter. We use Type Lambdas to resolve this issue:
!
new Applicative[({type λ[A] = Reader[I, A]})#λ]!
33. tag: step5
Go back to Reader[I, 0]
object Reader {!
implicit def readerIsAFunctor[I] = new Functor[({type λ[A] = Reader[I,
A]})#λ] {
override def fmap[A, B](m: Reader[I, A], f: A => B) = m.map(f)
}
implicit def readerIsAnApplicative[I] = new Applicative[({type λ[A] =
Reader[I, A]})#λ] {
override def pure[A](a: A) = Reader { _ => Success(a) }
override def apply[A, B](mf: Reader[I, A => B], ma: Reader[I, A]) =
map2(mf, ma)((f, a) => f(a))
override def map[A, B](m: Reader[I, A], f: A => B) = m.map(f)
}
34. tag: step5
Go back to Reader[I, 0]
object Reader {!
import scala.language.implicitConversions
// Here we help the compiler a bit. Thanks @skaalf (Julien Tournay) !
// and https://github.com/jto/validation
implicit def fcbReads[I] = functionalCanBuildApplicative[({type λ[A] =
Reader[I, A]})#λ]
implicit def fboReads[I, A](a: Reader[I, A])(implicit fcb:
FunctionalCanBuild[({type λ[x] = Reader[I, x]})#λ]) = new
FunctionalBuilderOps[({type λ[x] = Reader[I, x]})#λ, A](a)(fcb)
35. Converter[I, T]
tag: step5
trait Converter[I, T] {
def convert(p: I => Result[String]): Reader[I, T]
}!
object Converter {
implicit def stringConverter[I] = new Converter[I, String] {
override def convert(p: I => Result[String]) = Reader[I, String](p)
}!
!
implicit def dateConverter[I](implicit dtFormat: DateFormat) = new
Converter[I, Date] {
override def convert(p: I => Result[String]) = Reader[I, Date] { s:
I =>
p(s) match {
case Success(dt) => try { !
Success(dtFormat.parse(dt))
} catch { case e: ParseException => Failure(s"...") }
case f: Failure => f
}}}
36. Picker[I]
tag: step5
case class Picker[I](p: I => Result[String]) {
def as[T](implicit c: Converter[I, T]): Reader[I, T] = c.convert(p)
}
object CsvPicker {
def apply[T](i: Int): Picker[List[String]] = Picker { elems:
List[String] =>
if (i > 0 && elems.size > i) Success(elems(i).trim)
else Failure(s"No column ${i} found in ${elems.mkString(";")}")
}}
object XmlPicker {
def apply[T](query: Elem => NodeSeq): Picker[Elem] = Picker { elem:
Elem =>
try {
Success(query(elem).text)
} catch {
case e: Exception => Failure(e.getMessage)
}}}!
37. Usage sample
tag: step5
import play.api.libs.functional.syntax._
import Reader._!
import Converter._!
implicit val dF = new SimpleDateFormat("dd/MM/yyyy")!
val xml = XML.loadString("""
<company name="Dupont and Co">
<owner>
<person firstname="jean" lastname="dupont" birthdate="11/03/1987"/>
</owner>
</company>""")!
!
val r = (
XmlPicker(_ "person" "@firstname").as[String] and
XmlPicker(_ "person" "@lastname").as[String] and
XmlPicker(_ "person" "@birthdate").as[Date]
)(Person)!
r(xml) === Success(Person("jean", "dupont", dF.parse("11/03/1987")))
38. tag: step6
Adding combinators
• We now add new abilities to Reader
• We especially want a method to validate content
39. tag: step6
Adding combinators
case class Reader[I, O](p: I => Result[O]) {!
!
def flatMap[T](f: O => Reader[I, T]): Reader[I, T] = Reader { s: I =>
p(s) match {
case Success(o) => f(o)(s)
case f: Failure => f
}
}
def verify(f: O => Result[O]): Reader[I, O] = flatMap { o: O =>
Reader( _ => f(o)) }!
}
40. Usage sample
tag: step6
val r: Reader[String, String] = Reader { Success(_) }
r.verify { x => if (x == "OK") Success(x) else Failure("KO") }("OK")
=== Success("OK")
41. Conclusion
• We have created a simple and powerful DSL for
processing CSV and XML content
• This DSL give us ability to Pick data, transform and
verify it and also accumulate encountered errors
• We have seen that making Reader an instance of the
Applicative Functor Type Class add it new capabilities
• Using ad-hoc polymorphism using Type Classes gives
us ability to extends Reader without altering it
42. Follow-up
• Type Classes are defined by functions that must be
implemented with regards to their laws (left/right
identity …)
• Proving the correctness of Type Class Laws can
be a bit tricky we usual approach
• I will introduce the framework ScalaCheck at
scala.io 2014, and show how to test them
43. Follow-up
• In the roadmap that has been announced by
@typesafe (http://scala-lang.org/news/roadmap-next),
it seems that Scala 2.14 (aka « Don
Giovanni ») will clean up lambda types syntax