My talk from Scalar 2017 conference.

1. HOW TO EXTEND MAP?
OR WHY WE NEED COLLECTIONS REDESIGN?
SZYMON MATEJCZYK, SCALAR 2017

2. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
@SZYMONMATEJCZYK
▸ Engineer & data scientist.
▸ Scala fan since 2.8. Graphs lover.
▸ PhD student on leave of absence.
▸ OSS contributor (Cassovary,
Nebulostore).

3. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
GRID
type Grid[V] = Map[(Int, Int), V]
def row(i: Int): Seq[(Int, V)]

4. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
SCALA COLLECTIONS

5. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
SCALA COLLECTIONS - IMMUTABLE

6. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMMUTABLE MAP
package scala
package collection
package immutable
import generic._
trait Map[K, +V] extends Iterable[(K, V)]
// with GenMap[K, V]
with scala.collection.Map[K, V]
with MapLike[K, V, Map[K, V]
trait MapLike[K, +V, +This <: MapLike[K, V, This] with Map[K, V]]
extends scala.collection.MapLike[K, V, This]
with Parallelizable[(K, V), ParMap[K, V]]
package scala
package collection
trait MapLike[K, +V, +This <: MapLike[K, V, This] with Map[K, V]]
extends PartialFunction[K, V]
with IterableLike[(K, V), This]
with GenMapLike[K, V, This]
with Subtractable[K, This]
with Parallelizable[(K, V), ParMap[K, V]]

7. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMMUTABLE.MAP TRAITS HIERARCHY

8. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMMUTABLE MAP
package scala
package collection
package immutable
import generic._
trait Map[K, +V] extends Iterable[(K, V)]
// with GenMap[K, V]
with scala.collection.Map[K, V]
with MapLike[K, V, Map[K, V]
trait MapLike[K, +V, +This <: MapLike[K, V, This] with Map[K, V]]
extends scala.collection.MapLike[K, V, This]
with Parallelizable[(K, V), ParMap[K, V]]
package scala
package collection
trait MapLike[K, +V, +This <: MapLike[K, V, This] with Map[K, V]]
extends PartialFunction[K, V]
with IterableLike[(K, V), This]
with GenMapLike[K, V, This]
with Subtractable[K, This]
with Parallelizable[(K, V), ParMap[K, V]]

9. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
LET’S START WITH A INTERFACE…
trait Grid[V] extends immutable.Map[(Int, Int), V] {
def row(i: Int): Seq[(Int, V)]
}
class NestedGrid[V](
val underlying: immutable.Map[Int, immutable.Map[Int, V]]
)
extends Grid[V] {
...
}

10. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMPLEMENTATION
override def row(i: Int): Seq[(Int, V)] =
underlying.get(i).toSeq.flatten
override def get(key: (Int, Int)): Option[V] = {
for {
inner <- underlying.get(key._1)
res <- inner.get(key._2)
} yield res
}
override def iterator: Iterator[((Int, Int), V)] =
underlying.iterator.flatMap{
case (first, map) => map.iterator.map(x => ((first, x._1), x._2))
}
override def -(key: (Int, Int)): NestedGrid[V] = {
underlying.get(key._1) match {
case Some(e) => new NestedGrid(underlying + ((key._1, e - key._2)))
case None => this
}
}

11. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMPLEMENTATION
class NestedGrid[V](
val underlying: immutable.Map[Int, immutable.Map[Int, V]]
) extends Grid[V] {
override def +(kv: ((Int, Int), V)): NestedGrid[V] = {
...
}
}

12. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMPLEMENTATION
class NestedGrid[V](
val underlying: immutable.Map[Int, immutable.Map[Int, V]]
) extends Grid[V] {
override def +(kv: ((Int, Int), V)): NestedGrid[V] = {
...
}
}
Method + overrides nothing

13. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
FEW STEPS BACK
trait Grid[+V] extends immutable.Map[(Int, Int), V] {
def row(i: Int): Seq[(Int, V)]
}
class NestedGrid[+V](
protected val underlying: immutable.Map[Int, immutable.Map[Int, V]]
)
extends Grid[V]
{
override def +[V1 >: V](kv: ((Int, Int), V1)): NestedGrid[V1] = {
...
}
}

14. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMPLEMENTATION CONT’D
override def +[V1 >: V](kv: ((Int, Int), V1)): NestedGrid[V1] = {
underlying.get(kv._1._1) match {
case Some(e) => new NestedGrid(
underlying + ((kv._1._1, e + ((kv._1._1, kv._2))))
)
case None => new NestedGrid(
underlying + ((kv._1._1, immutable.Map(kv._1._2 -> kv._2)))
)
}
}

15. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMMUTABLE MAP COVARIANCE

16. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
SIMPLE
trait Key
case class DoorKey(id: Int) extends Key
case class EnglishKey(size: Int) extends Key
import scala.collection.immutable.Map
val cabinet = Map[Int, DoorKey](
1 -> DoorKey(7),
3 -> DoorKey(8)
)
cabinet + (5 -> EnglishKey(77))

17. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
IMMUTABLE MAP COVARIANCE
=+

18. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
TEST
def sampleMap = {
val map: NestedGrid[Figure] = NestedGrid.empty[Figure]
map + (((1, 1), Bishop)) + (((2, 2), Knight))
}
"Grid" should {
"contain elements" in {
val map = sampleMap
map((1, 1)) should equal (Bishop)
map.toSeq.sortBy(_._1) should equal (Seq(
((1, 1), Bishop),
((2, 2), Knight)
))
}
"support iteration" in {
val map = sampleMap
map.iterator.toSeq should equal (Seq(((1, 1), Bishop), ((2, 2), Knight)))
}
"support adding superclass" in {
val map: NestedGrid[Figure] = NestedGrid.empty[Figure]
(map + (((2, 2), Pawn))) shouldBe an[Grid[Piece]]
}
"support row-iteration" in {
val map = sampleMap
map.row(1) should equal (Seq((1, Bishop)))
}
}

19. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
FILTER AND OTHERS
def filter(p: A => Boolean): Repr = {
val b = newBuilder
for (x <- this)
if (p(x)) b += x
b.result
}
trait Map[K, +V] extends ...
with MapLike[K, V, Map[K, V]
trait MapLike[K, +V, +This <: MapLike[K, V, This] extends … {
override protected[this] def newBuilder: Builder[(K, V), This] =
new MapBuilder[K, V, This](empty)
}
class NestedGrid[+V](
protected val underlying: immutable.Map[Int, immutable.Map[Int, V]]
)
extends Grid[V]
with immutable.MapLike[(Int, Int), V, NestedGrid[V]]

20. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
MAP AND OTHERS
def map[B, That](f: A => B)(
implicit bf: CanBuildFrom[Repr, B, That]
): That = {
val b = bf(repr)
b.sizeHint(this)
for (x <- this) b += f(x)
b.result
}
implicit def canBuildFrom[B]: CanBuildFrom[NestedGrid[B], ((Int, Int), B), NestedGrid[B]] =
new CanBuildFrom[NestedGrid[_], ((Int, Int), B), NestedGrid[B]] {
override def apply(from: NestedGrid[_]): mutable.Builder[((Int, Int), B), NestedGrid[B]] =
newBuilder[B]
override def apply(): mutable.Builder[((Int, Int), B), NestedGrid[B]] = newBuilder
}
def newBuilder[B]: mutable.Builder[((Int, Int), B), NestedGrid[B]] =
new mutable.MapBuilder[(Int, Int), B, NestedGrid[B]](empty)

21. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
DEFAULT
override def withDefault[V1 >: V](d: ((Int, Int)) => V1): NestedGrid[V1] =
new NestedGrid.WithDefault[V1](this, d)
override def withDefaultValue[V1 >: V](d: V1): NestedGrid[V1] =
new NestedGrid.WithDefault[V1](this, x => d)
class WithDefault[+V](from: NestedGrid[V], d: ((Int, Int)) => V) extends NestedGrid[V](from.underlying) {
override def empty = new WithDefault(from.empty, d)
override def updated[V1 >: V](key: (Int, Int), value: V1): WithDefault[V1] =
new WithDefault[V1](from.updated[V1](key, value), d)
override def + [V1 >: V](kv: ((Int, Int), V1)): WithDefault[V1] = updated(kv._1, kv ._2)
override def - (key: (Int, Int)): WithDefault[V] = new WithDefault(from - key, d)
override def withDefault[V1 >: V](d: ((Int, Int)) => V1): NestedGrid[V1] = new WithDefault[V1](from, d)
override def withDefaultValue[V1 >: V](d: V1): NestedGrid[V1] = new WithDefault[V1](from, x => d)
override def default(x: (Int, Int)) = d(x)
}

22. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
RECAP
1. Decipher hierarchy
2. Implement logic
3. Be careful about variance and type bounds.
4. Change type params in MapLike for filter, take,…
5. Add implicit CanBuildFrom for map, flatMap, …
6. Manually create WithDefault implementation
(code duplication)

24. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?

25. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
COLLECTIONS REDESIGN (ODERSKY)
•simplicity
•separate interface from implementation
•fix complicated inheritance structure
•improve efficiency

26. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?

27. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
COLLECTIONS REDESIGN (SPIEWAK)
•to generic interfaces (Seq)
•collection/mutable/immutable
•general implementations inefficient
•extendability is important
•CanBuildFrom does more harm than good

28. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
BONUS - QUESTIONS
Did you spot a memory leak?
How would experienced Scala programmer tackle the
problem?
Is the extension of mutable.Map similar?

29. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
BONUS - ALTERNATIVES
•Implement row method outside of Map
•Implicit conversions
•Use composition, implement map, filter, default … on your
own

30. HOW TO EXTEND MAP?… OR WHY WE NEED COLLECTIONS REDESIGN?
REFERENCES
•http://www.artima.com/scalazine/articles/scala_collections_architecture.html
•https://github.com/lampepfl/dotty/issues/818
•https://gist.github.com/djspiewak/2ae2570c8856037a7738
•Code: https://github.com/szymonm/scalar-grid
•@szymonmatejczyk