Slides for our talk given at Functional Conf 2017. Shared our experience of putting 34,000 lines of Haskell code in production at Vacation Labs. Please ping me on https://twitter.com/saurabhnanda if you'd like help with deploying Haskell in an industrial setting.

1. Joys & frustrations of putting
34,000 lines of Haskell into production
NOTE: Another version of these slides (with more text) is available at
https://www.slideshare.net/saurabhnanda/joys-frustrations-of-putting-34000-lines-
of-haskell-into-production-at-vacation-labs
Haskell

2. Source of all this gyaan
34,000 LoC of Haskell in production
Touched (almost) all aspects of a typical web-app stack
Except Haskell (or Purescript) in the frontend.
Using Angular2 + TypeScript instead.

4. Why Haskell?Why Haskell?
आ बैल मुझे मारआ बैल मुझे मार

5. Vacation Labs
Travel-Commerce Platform
Enabling travel companies with e-commerce technology.
Built mostly with Rails & Angular-v1 (Javascript)

6. Remember those numbers
“It's always good to remember where you come from and
celebrate it. To remember where you come from is part of
where you're going.”
- Anthony Burgess

7. Problems with Rails/JS
Code
Unable to refactor core data-structures confidently
Unable to remove dead-code without fear of breaking
Unable to review code effectively

11. Building webapps in Haskell
is harder than it ought to be
What I felt a year ago, just starting out...
http://tinyurl.com/haskell-is-needlessly-hard

12. Haskell.
First, the GOOD parts.

13. factoringre
actoringref
ctoringrefa
toringrefac
refactoring
toringrefact
oringrefacto
ingrefactor
ngrefactori

14. { “Type-safe” : “JSON” }
Aeson is brilliant. Auto-derives a lot of boilerplate.
OTOH writing JSON codecs in TypeScript by hand.

15. HTTP / Web / App Server
WAI/Warp has pretty good performance.
One lightweight thread per-incoming request.
Solid concurrency without an event-driven / callback model
(unlike node).

16. Now, for the
BAD parts.

17. Big problems faced first-hand
Library eco-system is effectively broken
Lack of established best-practices
Editor tooling
Records
Lack of ORM

18. Library ecosystem is
effectively broken
Lack of libraries it NOT a problem
Fragmentation, non-standardization
Heaps of cruft on Hackage
Broken library ecosystem

19. I can easily compare Haskell libraries
to select the best one
Strongly
disagree
Disagree Neutral Agree Strongly
agree
100
200
300
400
500

20. I can easily compare Haskell libraries
to select the best one
Strongly
disagree
Disagree Neutral Agree Strongly
agree
100
200
300
400
500
44% respondents
replied negatively

21. Emails
SES, Postmark, Mandrill, raw SMTP, etc. All there.
Fiasco with `smtp-mail` library.
Broken library ecosystem

22. Sendgrid library
`haskell-sendgrid` is incomplete. We wrote our own.
Ping us on Twitter if you want to help open-source our
code. (Open bounty)
http://www.vacationlabs.com/haskell-bounty-program/
Call for contribution

23. Testing
287 packages related to testing
None of them runs tests in isolated DB txns out-of-the-box.
Broken library ecosystem

24. DB libraries
persistent, hedgehog, hasql, pg-simple, opaleye,
mysql-simple, HaskellDB
Which to use when? Why so many?
Broken library ecosystem

25. Lack of established
best-practices
Community keeps arguing about core stuff
stack vs cabal
mtl vs free-monads
error handling
Is TH good or bad?
No best practices

26. Installing Haskell
FOUR different ways to install Haskell - seriously?!
PS: Just use “stack” - tried & tested. LTS snapshots are
awesome!
No best practices

28. Error handling
EIGHT different ways. Each library uses something else.
http://www.randomhacks.net/2007/03/10/haskell-8-ways-to-report-errors/
`ExceptT` vs `Conrol.Monad.Catch` fiasco
No best practices

29. do
x <- runExceptT (liftIO $ throwIO DivideByZero)
case x of
Left e -> pure “exception caught”
Right r -> pure “should never happen!”

30. do
x <- runExceptT (liftIO $ throwIO DivideByZero)
case x of
Left e -> pure “exception caught”
Right r -> pure “should never happen!”
-- OUTPUT
*** Exception: divide by zero
-- Reaction: Wtf ?!

31. do
x <- Control.Monad.Catch.try (liftIO $ throwIO DivideByZero)
case x of
Left (e :: SomeException) -> pure “exception caught”
Right r -> pure “should never happen!”

32. do
x <- Control.Monad.Catch.try (liftIO $ throwIO DivideByZero)
case x of
Left (e :: SomeException) -> pure “exception caught”
Right r -> pure “should never happen!”
-- OUTPUT
“exception caught”

33. Error reporting
Haskell code throws runtime errors. Please report them
properly.
Hype - “if it compiles, it runs”. Reality - “after a refactor, if it
compiles, then there is a high chance that you haven’t
broken anything”
No best practices

35. Which file?
Turns out “fastLogger” was trying to open a log-file in a non-existent
directory. (On LTS-9.0)

36. *** Exception: UnexpectedNull
{ errSQLType = “varchar”
, errSQLTableOid = Just (Oid 603560)
, errSQLField = “result22_2”
, errHaskellType = “Text”
, errMessage = “”
}

37. *** Exception: UnexpectedNull
{ errSQLType = “varchar”
, errSQLTableOid = Just (Oid 603560)
, errSQLField = “result22_2”
, errHaskellType = “Text”
, errMessage = “”
}
Which row? What SQL was being run?
Any other info that can help me debug this? Stacktrace?

39. Occurred during stress testing of JSON API. Wasted a lot of time!
Which file descriptor? Which library was accessing it? Nothing!
Turned out that DB pool size was larger than what DB could handle.

40. Absence of CallStacks can
probably be fixed
Call for contribution
Imperceptible perf-penalty with HasCallStack in IO code
Not being used by IO-heavy libraries. Raise some PRs!
GHC discussion to infer HasCallStack automatically
https://ghc.haskell.org/trac/ghc/ticket/13360

41. Testing
We drank the QuickCheck kool-aid
QuickCheck fiasco
No best practices

42. Editor tooling
SAD TRUTH - Nothing works as well as it should.
Tried a lot of things
Emacs + haskell-mode, Spacemacs + Intero, SublimeHaskell, Haskero, Haskelly, HIE, ghcid, etc.
Underlying GHCi hogs/leaks memory.
Editor tooling

44. Important caveat about
memory usage
Haskell DOES NOT leak memory in production.
Only present in GHC-i -- used in development.
Production Rails-app: 360 MB of RSS (x 3 processes)
Production Haskell-app: 78 MB of RSS (single process).

45. Recommendation:
Editor tooling
Use spacemacs + intero OR VSCode + HIE (Haskell IDE
Engine)
Keep ghci open in a terminal window and make friends with
:set -fobject-code & :load & :reload

46. Shout-out to
HIE (Haskell IDE Engine)
Call for contribution
Please use it, if it works for you. Make it work for you!
Contribute if you can.
https://github.com/haskell/haskell-ide-engine

47. Broken record
Records. Records. Records.
Records in Haskell are beyond horrible.
Some say, Haskell doesn’t even have records.
http://www.parsonsmatt.org/overcoming-records/

48. -- Your `users` table would probably map to the following…
data User = User
{ userId :: UserId
, userCreatedAt :: UTCTime
, userUpdatedAt :: UTCTime,
, userStatus :: UserStatus,
, userName :: Maybe Text
, userEmail :: Text
, userAge :: Maybe Int
}
-- But, you can’t use this in your `createUser` endpoint.
-- UI shouldn’t set `id, createdAt, updatedAt, status`
-- No easy way to create a “sub-record” from a master record.

49. -- Created our own library `record-splicer` to deal with this...
createRecordSplice SpliceArgs
{
sourcePrefix = "user"
, source = ''User
, requiredFields =
[
'userName
, 'userEmail
, 'userAge
]
, targetPrefix = "newuser"
, targetName = "NewUser"
, deriveClasses = [''Eq, ''Show, ''Generic]
}
makeLensesWith abbreviatedFields ''NewUser

50. // Same thing in TypeScript
type interface User {
readonly id: number
, readonly createdAt: DateTime
, readonly updatedAt: DateTime
, readonly status: ‘unconfirmed’ | ‘confirmed’ | ‘blocked’
, readonly name: string | null
, readonly email: string
, readonly age: string | null
}
// Making a typesafe sub-record
type NewUser = Pick <User, ‘name’ | ‘email’ | ‘age’>;
// Making a typesafe super-record
type Admin extends User {
readonly permissions: Array<Permission>
}

51. DB Library.
Dare I say ORM?
Either you use an ORM or you write an ad-hoc ORM.
Tried to understand how to build apps without an ORM.
Not convinced. At all. [1] [2] [3] [4]
Lack of ORM

52. Associations?
No DB library deals with associations.
Except two new discoveries - beam & postgresql-orm
Lack of ORM

53. Validations?
No DB library deals with validations.
Rails’ convention of putting strict invariants/validations in
the DB layer works ridiculously well.
Lack of ORM

54. Validations in DB-layer are an
anti-pattern?
Let’s look state-of-the-art: “digestive-functors”
Lack of ORM

56. DB Library Problem as well.
Incomplete / half-baked DB libraries
mysql-simple - (had?) concurrency issues
postgresql-simple doesn’t use PREPARE/EXECUTE
Opaleye generates slow queries
Lack of ORM

57. Applicative parsers?
Or “Hodor” parsers.
Special hate for applicative parsers!
Negate all benefits of static typing.
Especially true for postgresql-simple.
Lack of ORM

58. query_ = ([qc|
SELECT
c.id as client_id
, coalesce(c.custom_domain, c.domain) as domain
, c.home_page as home_page
, c.support_phone as support_phone
, c.support_email as support_email
, c.logo_photo_id as loto_photo_id
, logo_photo.image_file_name as logo_file_name
, logo_photo.image_fingerprint as logo_fingerprint
, c.favicon_image_id as favicon_image_id
, favicon.image_file_name as favicon_file_name
…

59. query_ = ([qc|
SELECT
c.id as client_id
, coalesce(c.custom_domain, c.domain) as domain
, c.home_page as home_page
, c.support_phone as support_phone
, c.support_email as support_email
, c.logo_photo_id as loto_photo_id
, logo_photo.image_file_name as logo_file_name
, logo_photo.image_fingerprint as logo_fingerprint
, c.favicon_image_id as favicon_image_id
, favicon.image_file_name as favicon_file_name
…
rowParser assetHost_ = Storefront
<$> field <*> field <*> field <*> field <*> field
<*> (photoEssentialsParser assetHost_ Photo.WebsiteLogo)
<*> (photoEssentialsParser assetHost_ Photo.Favicon)
<*> field <*> field <*> field <*> field <*> field <*> field
<*> field <*> field <*> field <*> field <*> field <*> field
<*> field <*> field <*> hstoreBoolParser <*> field
<*> field <*> field <*> field <*> field <*> field <*> field

60. query_ = ([qc|
SELECT
c.id as hodor
, coalesce(c.custom_domain, c.domain) as hodor
, c.home_page as hodor
, c.support_phone as hodor
, c.support_email as hodor
, c.logo_photo_id as hodor
, logo_photo.image_file_name as hodor
, logo_photo.image_fingerprint as hodor
, c.favicon_image_id as hodor
, favicon.image_file_name as hodor
…
rowParser assetHost_ = Storefront
<$> hodor <*> hodor <*> hodor <*> hodor <*> hodor
<*> (photoEssentialsParser assetHost_ Photo.WebsiteLogo)
<*> (photoEssentialsParser assetHost_ Photo.Favicon)
<*> hodor <*> hodor <*> hodor <*> hodor <*> hodor <*> hodor
<*> hodor <*> hodor <*> hodor <*> hodor <*> hodor <*> hodor
<*> hodor <*> hodor <*> hstoreBoolParser <*> hodor
<*> hodor <*> hodor <*> hodor <*> hodor <*> hodor <*> hodor

61. PG-Simple’s “hodor” parser
can be fixed
There is an open bounty.
https://github.com/lpsmith/postgresql-simple/issues/43
http://www.vacationlabs.com/haskell-bounty-program
Call for contribution

62. What did we really build in
38,000 lines of Haskell?
Flexi-payment plugin
DelayedJob clone (but using LISTEN/NOTIFY) along with UI
Sendgrid library
Opaleye boilerplate code-generator
Deep instrumentation wrappers over core libs

64. Logging & Instrumentation
“Deep instrumentation” in Haskell seems to be non-existent
Eg. kind of stuff provided by Rails out of box, or Skylight, or New Relic
Very hard to build drop-in instrumentation libraries for Haskell.
Probably because of the type-system itself.
http://tinyurl.com/haskell-instrumentation
Built our own ad-hoc logging & instrumentation layer by
wrapping important libs

65. Job Queue
Rails has DelayedJob (with an RDBMS backend).
Works really well for moderate workloads. ACID compliant and doesn’t add more moving parts to production.
Couldn’t find anything similar in Haskell world. Built our own.
Will open-source.
Except `yesod-job-queue` but we aren’t using Yesod.
Redis & SQS job-queues available, but we didn’t use first-hand.
Can’t comment.

66. JobAdmin UI built by intern in 4 months (included learning Haskell).
Built using Servant, Lucid, Aeson, Opaleye.

67. Initial (excruciating) pain for long-term gain
We have already gone through the pain
Now, we are reaping the gains
Haskell sucks.
Why continue with it?

68. What’s next?
Raising awareness about the issues we faced.
The core language is not as confusing
as the ecosystem is.
Contributing in any way to fix the ecosystem.

69. Thank you
@saurabhnanda on twitter
@vacationlabs on instagram
Haskell Bounty Program
Haskell Internship Program
We’re hiring Haskellers (and Haskell enthusiasts)