This document summarizes the NEScala2016 conference and unconference. The conference on March 4th included 11 talks on Scala topics like Scalaz streams and Spark. The following day's unconference had 26 talks across 2-5 tracks and covered additional Scala topics such as macros, logical programming, Akka streams, and type parameters vs members. Resources from the events were also listed.
2. About
● A media company - Creative Content Everyone Can Afford
● Since 2011, we saved more than $5B for our customers
● We reached 1M clips of inventory + $1M marketplace
sales faster than any company in history
● 15 engineers (total 60+ employees)
2
3. AboutMe
● Use Scala for about 5 years
● Not using Scala day to day for the past 9 months
● Scala developer, not sorcerer.
● Data Handyman @ Videoblocks - We Heart Data
● codingphilosophy.com
3
5. ● On 3/4 - 3/5
● Since 2011
● At Philadelphia this year (last year in Boston)
● 3/4: conference - 1 track, 11 talks
● 3/5: unconference - 2-5 tracks, 26 talks
● pretty academia as usual
● $60 - order of magnitude cheaper, and better
NEScala2016
5
You have to be there!
10. SparkRDD
● compile-time type safe
● api is based on Scala collection (good and bad)
10
def collect(): Array[T]
def collect[U](f: PartialFunction[T,U])(implicit arg0: ClassTag[U]): RDD[U]
//val rdd = sc.textFile("hdfs:/user/bmc/wikipedia‐pagecounts.gz")
//val parsedRDD = rdd.flatMap { line=>
// line.split("""s+""") match {
// case Array(project,_,numRequests,_)=>Some((project,numRequests))
// case_=>None
// }}
parsedRDD.filter { case (project,numRequest) => project=="en" }.
reduceByKey(_+_).
take(100).
Foreach{ case (project,requests) => println(s"$project:$requests") }
11. SparkDataFrame
● DSL
● Optimized by catalyst
● lost type safety
11
val df = parsedRDD.toDF("project", "numRequests")
df.groupBy($"page").
agg(sum($"numRequests").as("count")).
limit(100).
show(100)
scala>:typedf.collect()
Array[org.apache.spark.sql.Row]
12. SparkDataset
● DSL + RDD api
● Type safety
● Optimized
● Use Tungsten, faster (20x) and smaller (4x)
12
val df = sqlContext.read.json("people.json")
case class Person(name: String, age: Long)
val ds: Dataset[Person] = df.as[Person]
ds.groupBy(_.name).count()
experimental
18. MonadicLogging?
● Monad to accumulate effects (log)
● Write log of the same thread together
● Write log with logic
○ if (latency > 10ms) { mlog.toStdOut }
● Maybe too complicated and defeat the purpose of
logging?
○ Async + JVM crash, how to know what's going?
18
34. Rapture-GuavaofScala
34
● single point of failure is GREAT in code
● config through import
● Rapture JSON supports 8 JSON backend
● syntax error of html or json are checked by compiler
● use gitter to report bug to Jon
35. Scala.js
● All Scala code are supported, but not all libraries
● union type
● D3, react, autowire
35
36. SI-2712
36
Scala Issue: Implement higher-order unification for type
constructor inference.
object Test {
def meh[M[_], A](x: M[A]): M[A] = x
meh{(x: Int) => x} // should solve ?M = [X] X => X and ?A = Int ...
}
38. Resources
38
● Runar's blog: http://blog.higher-order.com/
● Slides of Brendan's talk: https://speakerdeck.com/bwmcadams/nescala-16-scala-
macros-for-mortals-or-how-i-learned-to-stop-worrying-and-mumbling-wtf
● Logic programming: https://github.com/stewSquared/ukanren
● Akka stream: https://github.com/pkinsky/akka-streams-example
● Type member vs type parameter: http://www.artima.com/weblogs/viewpost.jsp?
thread=270195
● Good blog of category theory: http://bartoszmilewski.com/2014/10/28/category-
theory-for-programmers-the-preface/
● Unconference grid: https://goo.gl/ei5ijy
● Rapture (The Grava of Scala) https://github.com/propensive/rapture
● Another javascript library in shapeless: https://github.com/travisbrown/circe
● Typesafe RPC: https://github.com/lihaoyi/autowire
● https://issues.scala-lang.org/browse/SI-2712