A number of examples why Scala is in my rear view mirror, and a few ideas on how to improve the collections.

1. A Scala Corrections Library
Paul Phillips
paulp@improving.org
Source: xkcd, of course.

2. “When I'm working on a problem, I never think
about beauty. I think only how to solve the problem.”
!
“But when I have finished, if the solution is not
beautiful, I know it is wrong.”
– R. Buckminster Fuller
(syntax highlighting donated by paulp)

3. “When I'm working on a problem, I never
think about beauty. I think only how to
solve the problem.”
!
“But when I have finished, if the solution
is not beautiful, I know it is wrong.”
– R. Buckminster Fuller
trait ParSeqViewLike[
+T,
+Coll <: Parallel,
+CollSeq,
+This <: ParSeqView[T, Coll, CollSeq]
with ParSeqViewLike[T, Coll, CollSeq, This, ThisSeq],
+ThisSeq <: SeqView[T, CollSeq]
with SeqViewLike[T, CollSeq, ThisSeq]
] extends GenSeqView[T, Coll]
with GenSeqViewLike[T, Coll, This]
with ParIterableView[T, Coll, CollSeq]
with ParIterableViewLike[T, Coll, CollSeq, This, ThisSeq]
with ParSeq[T]
with ParSeqLike[T, This, ThisSeq]

4. The Winding Stairway
• Five years on scala
• Rooting for scala/typesafe
• But I quit a dream job...
• ...because I lost faith

5. Credentials

6. Credentials, cont.

7. Should you care?
•
I offer my credentials only to bear witness to my
credibility
•
I suspect I have written more scala code than
anyone else, ever.
•
What’s visible in compiler/library represents only
a small fraction of it

8. Caveats
•
I ran out of time. Slides are rushed. Forgive me.
•
Error messages and repl transcripts have been
heavily trimmed for clarity on a slide
•
This works counter to message when the point
involves complexity or incomprehensibility
•
So verbosify all compiler messages by a factor
of three for a more accurate feel

9. My axe is dull
•
I have been pulling my punches
•
This has left some thinking that I quit over
technical esoterica: java compatibility, jvm
limitations, intractable compiler challenges
•
This is not accurate

10. Subtext, people
•
Prevailing programmer culture frowns upon
criticism of named individuals
•
In this case that doesn’t leave much room for
additional specificity
•
All the relevant facts are available in the googles

11. Is Scala too complex?
•
I’ll field this one: YES
•
Is anyone fooled by specious comparisons
of language grammar size? Who cares?
•
Half the time when someone hits a bug they
can’t tell whether it is a bug in scala or the
expected behavior
•
That definitely includes me

12. Perceived Problem
C
•
A meme is going around that scala is too complex
•
Option A: Own it
•
Option B: Address it
•
Option C: Obscure it
p
O
i
t
n
o

13. Thus is born the “use case”
// A fictional idealized version of the genuine method
def map[B](f: (A)
B): Map[B]
!
// The laughably labeled "full" signature
def map[B, That](f: ((A, B))
B)
(implicit bf: CanBuildFrom[Map[A, B], B, That]): That
neither has any basis in reality!

14. the true name of map
// markers to distinguish Map's class type parameters
scala> class K ; class V
defined class K, V
!
scala> val host = typeOf[Map[K, V]]
host: Type = Map[K,V]
!
scala> val method = host member TermName("map")
method: Symbol = method map
!
// Correct signature for map has FOUR distinct identifiers
scala> method defStringSeenAs (host memberType method)
res0: String =
def map[B, That](f: ((K, V)) => B)
(implicit bf: CBF[Map[K,V],B,That]): That

15. •
Now you’re thinking “use case thing is a bug, big deal,
bugs get fixed.” Do they?
•
Surely as soon as it is known the documentation spins
these fabrications, it will be addressed? If not fixed, at
least it’ll be marked as inaccurate? Something?
•
Nope! To this day it’s the same. Your time is worthless.

16. Slightly Caricatured
map
def map[B](f: A => B): F[B]
Signature
Elegance
Advantages
Spokespicture
“map”
def map[B, That](f: A => B)
(implicit bf:
CanBuildFrom[Repr, B,
That]): That
Among the purest and
most reusable
<—- Not this.
abstractions known to
computing science
Can reason abstractly
about code
Can map a BitSet to a
BitSet without typing
“toBitSet”

17. The Bitset Gimmick
// Fancy, we get a Bitset back!
scala> BitSet(1, 2, 3) map (_.toString.toInt)
res0: BitSet = BitSet(1, 2, 3)
!
// Except…
scala> BitSet(1, 2, 3) map (_.toString) map (_.toInt)
res1: SortedSet[Int] = TreeSet(1, 2, 3)
!
// Um…
scala> (BitSet(1, 2, 3) map identity)(1)
<console>:21: error: type mismatch;
found
: Int(1)
required:
scala.collection.generic.CanBuildFrom[scala.collection.imm
utable.BitSet,Int,?]
(BitSet(1, 2, 3) map identity)(1)
^

18. similarly
scala> def f[T](x: T) = (x, new Object)
f: [T](x: T)(T, Object)
!
scala> SortedSet(1 to 10: _*)
res0: SortedSet[Int] = TreeSet(1, 2, 3,
!
scala> SortedSet(1 to 10: _*) map (x =>
res1: SortedSet[Int] = TreeSet(1, 2, 3,
!
scala> SortedSet(1 to 10: _*) map f map
res2: Set[Int] = Set(5, 10, 1, 6, 9, 2,
4, 5, 6, 7, 8, 9, 10)
f(x)._1)
4, 5, 6, 7, 8, 9, 10)
(_._1)
7, 3, 8, 4)

19. and in a similar vein
scala> val f: Int => Int = _ % 3
f: Int => Int = <function1>
!
scala> val g: Int => Int = _ => System.nanoTime % 1000000 toInt
g: Int => Int = <function1>
!
scala> Set(3, 6, 9) map f map g
res0: Set[Int] = Set(633000)
!
scala> Set(3, 6, 9) map (f andThen g)
res1: Set[Int] = Set(305000, 307000, 308000)

20. Java Interop: the cruelest joke
•
It’s impossible to call scala’s map from java!
•
See all the grotesque details at SI-4389
IX
F
T
ON
“I played with it until it got too tedious. I think the signatures work fine.
What does not work is that the variances of CanBuildFrom cannot be
modelled in Java, so types do not match. And it seems Java does not
even let me override with a cast. So short answer: You can't call these
things from Java because instead of declaration side variance you
have only a broken wildcard system.”
!
— Martin Odersky
W

21. Lightning Round
•
My time is running out and I can hear you saying…
•
“Just give us a laundry list of collections issues”
•
Okay, you asked for it (in my mind)

22. •
Implementation details infest everything
•
And every detail is implementation-defined
•
Capabilities should be designed around the laws
of variance; instead variance checks are
suppressed and key method contains is untyped
•
Specificity rules render contravariance useless
•
Implicit selection and type inference inextricably
bound - so type inference is largely frozen
because any change will break existing code

23. •
Extreme pollution of base objects - all collections
have “size: Int”, all Seqs have “apply”, etc.
•
Bundling of concerns (e.g. invariant Set)
•
Inheritance of implementation is the hammer for
every nail…
•
…yet “final” and “private”, critical for a hope of
correctness under inheritance, are almost
unknown
•
Semantics discovered instead of designed

24. assume the worst
In Set(x) ++ Set(x), which x wins?
!
Can xs filter (_ => true) return xs?
!
Are defaults preserved across
operations? Which operations? Is
sortedness? Will views and Streams
retain laziness when zipped?

25. xs map identity
scala> val m = Map(1 -> 2) withDefaultValue 10
m: Map[Int,Int] = Map(1 -> 2)
!
scala> m(1000)
res0: Int = 10
!
scala> (m map identity)(1000)
<console>:9: error: type mismatch;
found
: Int(1000)
required: CanBuildFrom[Map[Int,Int],(Int, Int),?]
(m map identity)(1000)
^
!
scala> m map identity apply 1000
java.util.NoSuchElementException: key not found: 1000
at MapLike$class.default(MapLike.scala:228)

26. types are for suckers
% find collection -name ‘*.scala’ |
xargs egrep asInstanceOf | wc -l
556

27. How could 556 casts ever go wrong
scala> val xs: Set[Int] =
(1 to 3).view.map(x => x)(breakOut)
!
java.lang.ClassCastException: SeqViewLike$$anon$3
cannot be cast to immutable.Set

28. get and apply
trivially fall into disagreement
!
scala> Map[Int,Int]() withDefaultValue 123
res0: Map[Int,Int] = Map()
!
scala> res0 contains 55
res1: Boolean = false
!
scala> res0 get 55
res2: Option[Int] = None
!
scala> res0 apply 55
res3: Int = 123

29. Why is covariance such an object of worship?
Types exist so we don’t have to live like this!
// WHY infer this utterly useless type?
scala> List(1, 2) ::: List(3, 4.0)
res0: List[AnyVal] = List(1, 2, 3.0, 4.0)
!
scala> PspList(1, 2) ::: PspList(3, 4.0)
<console>:23: error: type mismatch;
found
: PspList[Int]
required: PspList[Double]

30. Type Inference
+
Variance
——————————

31. Abstracting over mutability
•
•
•
An inherited implementation is ALWAYS wrong somewhere!!
•
Half the overrides in collections exist to stave off the
incorrectness which looms above. This is nuts.!
•
Not to mention “Map”, “Set”, etc. in three namespaces
Example: how do you write "drop" so it's reusable?!
In a mutable class, drop MUST NOT share, but in an
immutable class, drop MUST share!

32. How many ways are there to write ‘slice’ ?
% ack --no-filename 'def slice(' src/library/
!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
override def slice(from: Int, until: Int): Iterator[A] =
def slice(from: Int, until: Int): Iterator[A] = {
def slice(from: Int, until: Int): Repr =
def slice(from: Int, until: Int): Repr = {
def slice(from: Int, until: Int): Repr = {
def slice(start: Int): PagedSeq[T] = slice(start, UndeterminedEnd)
def slice(unc_from: Int, unc_until: Int): Repr
override /*IterableLike*/ def slice(from: Int, until: Int): Vector[A] =
override /*TraversableLike*/ def slice(from: Int, until: Int): Repr = {
override def slice(_start: Int, _end: Int): PagedSeq[T] = {
override def slice(from1: Int, until1: Int): IterableSplitter[T] =
override def slice(from1: Int, until1: Int): SeqSplitter[T] =
override def slice(from: Int, until: Int) = {
override def slice(from: Int, until: Int) = {
override def slice(from: Int, until: Int): List[A] = {
override def slice(from: Int, until: Int): Repr = self.slice(from, until)
override def slice(from: Int, until: Int): Repr = {
override def slice(from: Int, until: Int): Stream[A] = {
override def slice(from: Int, until: Int): String = {
override def slice(from: Int, until: Int): This =
override def slice(from: Int, until: Int): This =
override def slice(from: Int, until: Int): Traversable[A]
override def slice(from: Int, until: Int): WrappedString = {
override def slice(unc_from: Int, unc_until: Int): Repr = {

33. scala.conflation
•
Every collection must have size
•
Every sequence must have apply
•
Every call to map includes a "builder factory"
•
Every set must be invariant
•
Everything must suffer universal equality

34. predictability
One of these expressions returns 2 and one returns
never. Feeling lucky?
!
scala> (Stream from 1) zip (Stream from 1)
map { case (x, y) => x + y } head
!
scala> (Stream from 1, Stream from 1).zipped
map (_ + _) head

35. sets
Two complementary ways to define Set[A].
Complementary - and NOT the same thing!
Intensional
Extensional
Specification
Membership test
Members
Variance
Set[-A]
Set[+A]
Defining Signature
A => Boolean
Iterable[A]
Size
Unknowable
Known
Duplicates(*)
Meaningless
Disallowed

36. What's going on here?
scala> class xs[A] extends Set[A]
error: class xs has 4 unimplemented members.
!
// Intensional/extensional, conflated.
// Any possibility of variance eliminated.
def iterator: Iterator[A]
def contains(elem: A): Boolean
// What are these doing in the interface?
// Why can I define a Seq without them?
def -(elem: A): Set[A]
def +(elem: A): Set[A]

37. todo: also add all other methods
% git grep 'todo: also add' 607cb4250d
SynchronizedMap.scala: // !!! todo: also add all other methods
!
% git grep 'todo: also add' origin/master
SynchronizedMap.scala: // !!! todo: also add all other methods
!
commit 607cb4250d
Author: Martin Odersky <odersky@gmail.com>
Date:
Mon May 25 15:18:48 2009 (4 years, 8 months ago)
!
added SynchronizedMap; changed Set.put to Set.add, implemented
LinkedHashMap/Set more efficiently.

38. tyranny of the interface
•
Mandating "def size: Int" for all collections is the fast
track to Glacialville!
•
Countless times have I fixed xs.size != 0
•
Collections are both worlds: all performance/
termination trap, no exploiting of size information!
•
A universal size method must be SAFE and CHEAP

39. Psp Collections
•
So here is a little of what I would do differently
•
I realized since agreeing to this talk that I may
have to go cold turkey to escape scala’s orbit.
It’s just too frustrating to use.
•
Which means this may never go anywhere
•
But you can have whatever gets done

40. Conceptual Integrity
trait Collections {
type CC[+X]
type Min[+X]
type Opt[+X]
type CCPair[+X]
type ~>[-V1, +V2]
!
!
}
type
type
type
type
type
type
type
type
//
//
//
//
//
Iso[A]
Map[-A, +B]
FlatMap[-A, +B]
Grouped[A, DD[X]]
Fold[-A, +R]
Flatten[A]
Build[A]
Pure[A]
the overarching container type (in scala: any covariant collection, e.g. List, Vector)
least type constructor which can be reconstituted to CC[X] (scala: GenTraversableOnce)
the container type for optional results (in scala: Option)
some representation of a divided CC[A] (at simplest, (CC[A], CC[A]))
some means of composing operations (at simplest, Function1)
=
=
=
=
=
=
=
=
CC[A] ~> CC[A]
CC[A] ~> CC[B]
CC[A] ~> Min[B]
CC[A] ~> CC[DD[A]]
CC[A] ~> R
CC[Min[A]] ~> CC[A]
Min[A] ~> CC[A]
A ~> CC[A]
trait Relations[A] {
type MapTo[+B] = Map[A, B]
type FoldTo[+R] = Fold[A, R]
type This
= CC[A]
type Twosome
= CCPair[A]
type Self
= Iso[A]
type Select
= FoldTo[A]
type Find
= FoldTo[Opt[A]]
type Split
= FoldTo[Twosome]
}
//
//
//
//
//
//
//
//
//
//
//
//
//
//
//
//
e.g. filter, take, drop, reverse, etc.
e.g. map, collect
e.g. flatMap
e.g. sliding
e.g. fold, but also subsumes all operations on CC[A]
e.g. flatten
for use in e.g. sliding, flatMap
we may not need
an alias incorporating the known A
another one
the CC[A] under consideration
a (CC[A], CC[A]) representation
a.k.a. CC[A] => CC[A], e.g. tail, filter, reverse
a.k.a. CC[A] => A, e.g. head, reduce, max
a.k.a. CC[A] => Opt[A], e.g. find
a.k.a. CC[A] => (CC[A], CC[A]), e.g. partition, span

41. “Do not multiply entities
unnecessarily”
•
mutable / immutable
•
Seq / Set / Map
•
parallel / sequential
•
view / regular
24 Combinations!

42. Surface Area Reduced 96%
•
A Set is a Seq without duplicates.
•
A Map is a Set paired with a function K => V.
•
A mutable collection has nothing useful in common with an
immutable collection. Write your own mutable collections.
•
If we can’t get sequential collections right, we have no hope of
parallel collections. Write your own parallel collections.
•
“Views” should be how it always works.

43. predictability: size matters
scala> def f(xs: Iterable[Int]) = xs.size
f: (xs: Seq[Int])Int
!
// O(1)
scala> f(Set(1))
res0: Int = 1
!
// O(n)
scala> f(List(1))
res1: Int = 1
!
// O(NOES)
scala> f(Stream continually 1)
<ctrl-C>

44. Asking the right question
SizeInfo
/
Atomic
/
Infinite
Precise
Bounded

45. Don’t ask unanswerable questions
(Unless you enjoy hearing lies)
scala> val xs = Foreach from BigInt(1)
xs: psp.core.Foreach[BigInt] = unfold(1)(<function1>)
!
scala> xs.size
<console>:22: error: value size is not a member of
psp.core.Foreach[BigInt]
xs.size
^
!
scala> xs.sizeInfo
res0: psp.core.SizeInfo = <inf>

46. the joy of the invariant leaf
scala> List(1, 2, 3) contains "1"
res0: Boolean = false
!
scala> PspList(1, 2, 3) contains "1"
<console>:23: error: type mismatch;
found
: String("1")
required: Int
PspList(1, 2, 3) contains "1"
^

47. HEY! MAP NEED NOT BE RUINED!
scala> "abc" map (_.toInt.toChar)
res1: String = abc
!
scala> "abc" map (_.toInt) map (_.toChar)
res2: IndexedSeq[Char] = Vector(a, b, c)
!
// psp to the rescue
scala> "abc".m map (_.toInt) map (_.toChar)
res3: psp.core.View[String,Char] = view of abc
!
scala> res3.force
res4: String = abc