This document summarizes Rafal Wojdyla's presentation on how to be a productive data engineer. The presentation covers 4 areas: operations, development, organization, and culture. Under operations, it discusses the importance of automation for managing large Hadoop clusters to avoid putting out fires. For development, it emphasizes picking the right tools for the job, such as using Apache Crunch over Hadoop streaming at Spotify for its performance and type safety. In terms of organization, it notes that scaling machines is easy but scaling people and support is difficult. The presentation provides examples of automating tasks like map split size to help address this challenge.

1. How to Be
Productive
Data Engineer
Rafal Wojdyla - rav@spotify.com
Note: My views are my own and don't necessarily represent those of Spotify.
Speaker’s cut version – includes raw notes

2. Hi – my name is Rafal and I’m an engineer at Spotify, in this
presentation I will talk about how to be a productive data
engineer. I will combine knowledge of multiple productive
engineers at Spotify and touch different areas of our your
daily work life. I will use real world examples, failures,
success stories – but mostly failures. So if you are or want to
be a data engineer hopefully after this presentation every
single one of you will learn something now. I hope this
learning will improve your productivity, bring new feature to
your infrastructure or maybe spark a discussion inside your
team.

3. • Operations
• Development
• Organization
• Culture

4. We will go through lessons of productive data engineer and cover 4
different areas – operations, development, organization and culture.
So we will kinda work our way from low level admin tips and
spectacular disasters, after hard core operation we will talk about
development on Hadoop – what to avoid, how Spotify is overcoming
huge problem of legacy Hadoop tools. After development part we will
take a look at how organization structure can affect your productivity
and how one can tackle this problem. We will finish with ubiquitous
culture – how culture can help in being productive. So in a way we
start with low level scope of you productivity, how the cluster itself
can affect you, how operators can help you out, to later talk about
your development decision, structure of company and finish with how
environment can influence your work. There will be time for
questions at the end so please keep them till the end.

5. What is Spotify?
For everyone:
• Streaming Service
• Launched in October 2008
• 60 Million Monthly Users
• 15 Million Paid Subscribers
+ and for me:
• 1.3K nodes Hadoop cluster

6. Before we go deep into presentation let’s first talk about what spotify
is. Spotify is a streaming service, launched in 2008 in beautiful
Stockholm, Sweden. Current public numbers are that we have 60M
monthly users, 15M subscribers. And what’s unique about Spotify
service is that it can play a perfect song for every single moment, and
some of this is powered through Hadoop which makes it even cooler!
For me Spotify is also 1.3K Hadoop nodes – which is like a baby for
a team of 4 people. A baby that is sometimes very frustrating, shit
happens all the time and you have to wake up in the middle of the
night and clean it up, but it’s our baby and we love it. Without further
ado let’s move to the core of the topic and start with operations. If
there’s one lesson that comes from operating hadoop clusters from a
handful of nodes in the corner of the office to 1300 nodes – it will be
AUTOMATION.

7. Automation

8. Automation is crucial – especially if talking about Hadoop. Hadoop is
huge beast to manage, there is loads of moving parts, loads of new
stuff coming in and there’s always a reason for hadoop clusters to go
down. If hadoop was not enough there’s always something that your
company will push on poor operators – whether it’s a new linux distro
or maybe there’s a bug in libc and you need to restart all the
daemons and so on and so on.

9. ME
ADAM

10. You want to be proactive and do as little as possible – without
automation even coffee won’t help you. You want to be Adam – be
happy and work on new features, enhance hadoop – bring joy to
hadoop users. You don’t want to be the poor operator on the left,
focused primarily on putting out fires, exhausted. Btw this is the
picture of Adam and me after 40 hours of hadoop upgrade from
Hadoop 1 to 2 in 2013.

11. Apache Ambari
Cloudera Manager

12. So how to reach good enough automation of your cluster – let’s take Spotify as example first –
Spotify started with hadoop in 2009, very early, then there was a couple of tiny expansions, a
short episode of hadoop in EMR, and we went back to on premise with shinny new 60 nodes –
at that point we had to make a decision on how to manage hadoop – and because back then
CM was limited and Ambari didn’t exist and cause Spotify loves puppet we have decided to use
Puppet for this use case. It was rather big effort and took time, during which we had to drop
some work, put out fires and work on Puppet, but it was a great investment. Today after a few
iteration we like our puppet – as an example – most recent ongoing expansion is rather easy –
we name the machines using proper naming convention and puppet kicks in installs all the
services, configuration and keeps the machines in normalized state – very very important piece
of our infrastructure. But wait – the slide says something aboutAmbari and CM – yes – cause if
we were to set up a cluster today, we would most likely evaluate at least these two solution.
Like I said Spotify basically didn’t have a choice and we settled on puppet and we are happy
about it right now, but there’s huge leverage you can gain out of using these tools, loads of
features that you get out of the box, that we need to implement ourselves. So if you are
considering building hadoop cluster – make sure to give these tools a good try, they may not
solve all your issues and use cases but for sure will bring loads of value and in time your will get
even more features just from community – which is great, and is something that we are
missing.

13. + Puppet

14. That said – even if you decided to use Ambari of CM – mostly likely
you will still need some kind of configuration management tool –
whether it will be puppet, or chef or salt or whatever is your favorite –
you will need one, there will always be some extra library that you
need to install and configure or some user to create and so on.
There’s another interesting outcome of us building our own puppet
infrastructure – we know exactly how our hadoop is configured –
every single piece of it – which comes in handy in case of trouble
shooting. In this case we touched a little bit a problem of 3rd party
solutions vs let’s implement our own tailored solutions. How many of
you are aware of NIH problem?

15. Not Invented
Here

16. I will argue that there’s a number of cases and teams where this
problem occurs at Spotify. NIH problem is an nutshell when you
undervalue 3rd party solutions and convince others to implement your
own solutions – in most cases this is a huge problem. The lesson
that we have learned is that you need to give external tools a try,
experiment but don’t expect something to solve all your problems –
preferable define metrics of acceptance priori evaluation of tools.

17. Never Invented
Here

18. But what is actually very interesting in case of data areas is kinda
sibling problem on NIH – which is NeIH – a problem described I
believe by Michael O Church – it’s kinda opposite approach, it’s
when you overvalue 3rd party solutions, and end you in a messy
place of glue implementation madness. There’s loads of great tools
in BigData areas – not all of the work well with each other, not all of
them do well what them meant to be doing. I urge you to be critical,
sometimes implementation of your own tool or postponing a new,
shinny framework from infrastructure may be a good thing to do – but
it has to be a data driven decision that brings value. Think about this
two problems, and ask yourself are there examples of such solutions
at your company?

19. Wild Wild West

20. To illustrate this I will tell you a real story – a story of great failure and success at the end. So we had an
external consultant at Spotify – and his goal was to certify our cluster – basically 4 days of looking at
different corner of our infrastructure. First two days went really smooth, we went through our configuration,
state of the cluster and so on, we could not find a way to improve our cluster easily – which made as feel
proud, cause you know, we have this world class, talented hadoop expert over and he can’t find a way to
improve our cluster right? But oh boy was that a big mistake – so on day number 3, we are sitting in a
room, whole team and consultant and due to miscommunication and misconfiguration our standby NN and
RM go down – but that is still fine, cause RM starts in minute or two, standby can start in background – but
unfortunately during the troubleshooting by mistake we have killed our active NN – at this point basically
whole infrastructure was down – at our scale that means about 2 hours of downtime. It was bad! But wait
for day number 4 – so next day we are sitting in the room, again whole team, consultant but also our
managers and we listen to consultant saying that our testing and deployment procedures are like Wild Wild
West and we act like cowboys – it was hard to listen to but he was right and we knew it. Next thing we do,
was to go to a room with the team and come up something to solve this issue, we came up with something
that may be obvious – a preproduction cluster – a cluster made out of the same machine profile and almost
identical configuration which we will use for testing. But how to test was a real question. We went into
research mode and started reading and watching presentation – we were especially impressed by tool
called HIT by Yahoo, so we contacted the creators, unfortunately there was no plan to open source it – but
they gave us a nice tip – look atApache Bigtop.

21. Apache Bigtop

22. Apache Bigtop primarily facilities building, testing and deployment of hadoop
distribution – but you can also us it in a slightly different way – you can point
you bigtop at your preproduction cluster and use its smoke tests to test the
infrastructure. So our current flow of testing and deployment is to first deploy
to preproduction cluster, run bigtop tests – get instant feedback about the
change, if feedback is fine deploy to production if not – there’s something
wrong with the change and we know that before it is deployed to production.
Some findings from using BigTop is that it’s actually very easy to extend so we
were able to add smoke tests for our own tools like snakebite and luigi but
also what is very important we also run some production workloads as part of
smoke tests – which actually makes us feel sure about the change.
So in case of Apache Bigtop the problem was testing of hadoop infrastructure
– even tho Bigtop is not perfect for this it provides loads of value just out of the
box and thus it’s a great example of preventing NIH problem.

23. Enable log
aggregation

24. As an operator there are many ways to help yourself and also
delegate some of the work to developers themselves – one disabled
by default, but great feature of Hadoop 2 is log aggregation – how
many of you have log aggregation enable on your cluster? So in a
nutshell this feature will aggregate yarn logs from workers and store
them on HDFS for inspection, very useful for troubleshooting. So
most of you probably know how to enable it right?

25. To enable log aggregation
yarn.log-aggregation-enable = true
yarn.log-aggregation.retain-seconds = ?

26. It’s dead simple. But there’s one question – how long should we keep
the logs for? So we thought about it for a while, talked with HW a little
bit and since we have huge cluster why not story it for long time –
maybe we will need these logs for some analytics etc.

27. + <property>
+ <name>yarn.log-aggregation-enable</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>yarn.log-aggregation.retain-seconds</name>
+ <value>315569260</value>
+ <!--retention: 10 years-->
+ </property>

28. This is our initial change to configuration – 10 years. Does anyone
know what bad can happen if you do that?

29. Heap Memory used is 97%

30. If you run enough jobs/tasks after some time you will see something
like this in your NN – and when you see something like this in your
NN then you end up with hellelepahant!

31. Hellelephant

32. It’s a situation when your hadoop cluster spectacularly goes down.
What happen is that log aggregation will in time create many files,
very important consequence of many files on HDFS is growing heap
size, and then you get out of memory on NN. The lesson is that it’s
“good idea” to alert on heap usage for your master daemons but also
understand your configuration and its consequences, keep yourself
up to data on the changes in configuration, read code of hadoop
configuration keys and how they are connected between each other
– no all configuration parameters are documented.

33. Custom logs
• Profiling
• Garbage collection

34. It’s a situation when your hadoop cluster spectacularly goes down.
What happen is that log aggregation will in time create many files,
very important consequence of many files on HDFS is growing heap
size, and then you get out of memory on NN. The lesson is that it’s
“good idea” to alert on heap usage for your master daemons but also
understand your configuration and its consequences, keep yourself
up to data on the changes in configuration, read code of hadoop
configuration keys and how they are connected between each other
– no all configuration parameters are documented.

35. Right tool for
the job

36. There is a couple of interesting lessons about productive development – first
arguable most important is to pick right tool for the job – what is the current
most important value to bring. Let’s talk Spotify – in 2009 Spotify started with
hadoop streaming as a supported framework for MR development – hadoop
streaming basically enables you to implement MR jobs in languages different
than java – for many years it was THE framework – because Spotify loved
python and it enabled us to iterate faster thus provide knowledge for our
business. Time was passing and our hadoop cluster was growing – in time we
needed something different something better when it comes to performance
but also maturity. After long evaluation and I encourage you to watch
presentations by David Whiting about different frameworks, we have decided
to use Apache Crunch as supported framework for batch MR. Why – a couple
of reasons – first ease of testing and type safety.
David’s presentations:
* https://www.youtube.com/watch?v=XKY_0s7pESQ
* https://www.youtube.com/watch?v=-WOiZ2w7xtI

38. This graph shows number of successful and failed jobs divided by
framework for 6 month – and these are production jobs – as you can
see two most popular frameworks are hadoop streaming and crunch
– but the difference between failed and successful jobs is crucial.
Crunch jobs act much better and have better testing. Type safety
helps to discover problem at compile time and testing framework that
comes with crunch that we were able to enhance with hadoop
minicluster helps users to easily test their jobs – basically makes
testing easy, something that we missed for our hadoop streaming
jobs. But performance is another thing.

40. On this graph we can see map throughput for apache crunch and
hadoop streaming – these are production workloads for 6 months –
there’s huge difference. Crunch turns out to be on average 8 times
faster. What is more interesting is that we actually see higher
utilization of our cluster the more crunch jobs we see on the cluster –
which makes us super happy.

41. Right abstraction
for the job

42. Another thing that crunch provides is great abstraction – and that is
another thing that productive developer need to keep in mind – pick
the right abstraction for the jobs. In case of crunch we can start
thinking in terms of high level operations like filter, groupby, joins and
so on instead of old map/reduce legacy. This makes implementation
more intuitive and simply pleasant – thus make developer
experience much better. The interesting thing that we have observed
is that higher abstraction may remove some of the opportunities for
optimization thus it’s not as easy to implement the best performing
job – but on the other hand it reduces problem with premature
optimization, also on average performs really well – there’s very few
people that actually know how to optimize pure java MR jobs or
Hadoop streaming jobs at Spotify – but average optimization that we
get from crunch turns out to be really good as you could see on the
performance graph.

43. Scaling
machines is
easy, scaling
people is hard

44. We do have loads of nodes – and we have scaling machines nailed
down, crunch scales very well. But there’s big problem that we
currently have: scaling people. How you scale support and best
practices – we constantly see problems with code repetition, HDFS
mess, lack of data management, YARN resource contention – all this
brings our productivity down. There’s not enough time to go through
all of them but some of this problems we are trying to tackle with
nothing different then our beloved automation. Let’s see examples:

45. • Map split size
• Number of reducers
• HDFS data retention
• User feedback (ongoing)
Automation

46. We automate map split size calculation thus number of map tasks, but also number of
reducers therefor number and size of output files – all this is done by estimation and
historical data using our workflow manager Luigi – that I encourage you to take a look
at! Luigi github: https://github.com/spotify/luigi
We are about to finish our second iteration of HDFS retention policy that will
automatically remove data therefor reduce HDFS usage and in long term hopefully
reduce HDFS legacy mess.
Another ongoing effort is second iteration of automatic user feedback – we already
expose database with aggregated information about all MR jobs that our users can
query and learn how their jobs are performing – but we also plan another iteration
very simple iteration, focused on Crunch – that right after to workflow pipeline is done
will provide user with instant feedback – memory usage, garbage collection and so
on, very simple tweaks users can apply to improve their jobs – for example if a user
gives a pipeline 8GB of memory for each task and after going through counters we
see that tasks are actually using only max 3GB, instant feedback to reduce memory
could improve multitenancy of your cluster thus improve productivity.

47. Organization

48. With that let’s talk organization structure – how it can improve your
performance – but before that let’s take a look at a graph.

50. This graph show Hadoop availability by Quarter at Spotify, higher is
of course better – ok – so let’s see what happen here:

51. Ownerless

52. First part is hadoop cluster being ownerless, it was best effort
support by team of people that mostly didn’t even want to do
operations of hadoop, therefor multiple days of downtime happen,
and infrastructure was in bad shape, denormalized – overall terrible
state to be in. But there was a light at the end of the tunnel – in Q3
we have decided to create a squad – 3 people focused solely on
hadoop infrastructure.

53. Ownerless Squad

54. There was instant feedback right after squad was created – users
were happy and infrastructure was getting in shape, one of the first
decisions we have made was to move to yarn in Q4.

55. Ownerless
Squad
Upgrades

56. Q4 and beginnings of ‘14 we again saw drop in availability mostly
due to huge upgrade – and it’s consequences thereafter. The
upgrade itself took whole weekend, and after the upgrade we saw
many issues and fires that we had to put out, during this time we
were mostly reactive but also working on polishing our puppet
manifests. Whole situation stabilized after most fires were gone and
puppet was in good shape.

57. Ownerless
Squad
Upgrades Getting there

58. Our goal is too keep hadoop at 3 nines of availability – and we are
getting there since Q2 2014, hadoop squad is receiving constant
feedback from users and its common that hear that availability was
drastically improved which improved productivity and overall
experience, which is great and makes us want to work even harder
to achieve better results.

59. Culture

60. With that lets now talk about what surrounds us – the culture – I
strongly believe the culture at Spotify has huge influence on
productivity - there are tree main pillars of culture.

61. Experiment
Fail Fast
Embrace Failure

62. Experiment, fail fast and embrace failure. We love to experiment and
we have time to experiment whether it’s company wide hack week,
R&D days – if only one wish to experiment there’s time to do that and
there’s loads of curious people at Spotify there’s always something
going on. The most successful data based experiments are Luigi –
hadoop workflow manager and snakebite – pure python hdfs client –
I encourage you to take a look at them. Fail fast – do small steps –
don’t be afraid to admit failure, keep it as part of learning process to
the point of embracing it. Talk about your failures, share them publicly
for example through presentations both internally and externally – it
will make experimentation thus innovation flow much smoother. To
back this up by example let’s talk about one ongoing experiment.
Luigi github: https://github.com/spotify/luigi
Snakebite github: https://github.com/spotify/snakebite

63. Spark
But we
have
tried!
Non grata

64. Spark – it pretty much ongoing experiment that we come back to
every now and then, but officially it’s not welcome on production
cluster due to immaturity and poor multitenancy support – that said
most recent releases (>1.3) are very promising and we are
constantly playing with it and have high hopes for it, especially about
most recent dynamic resource allocation feature. There’s not so
much time left but I would like to share with you two important
lessons from our evaluation of a heavy spark job.

65. Spark
spark.storage.memoryFraction (0.6)
spark.shuffle.memoryFraction (0.2)
In shuffle heavy algorithms reduce
cache fraction in favour of shuffle.

66. First hint is about memory settings – there are two important settings
that can improve stability of your heavy Spark jobs – memory
available for caching – storage.memoryFraction and memory
available for shuffle – shuffle.memoryFraction. The default settings
are .6 and .2 leaving .2 for runtime. In our case we had a heavy
machine learning job that was doing almost terabyte of shuffle – but
very little (proportionally) caching – initially we had issues with shuffle
step, but reducing storage memory and leaving extra memory for
shuffle and runtime improved stability.

67. Spark
spark.executor.heartbeatInterval (10K)
spark.core.connection.ack.wait.timeout (60)
Increase in case of long GC pauses.

68. Another issue that we hit was long GC pauses – thus executors
would disappear which in turns triggers recomputation and in the end
potentially application failure. After tweaking hearbeat interval and
ack.wait timeout was saw improvement in stability and even tho GC
pauses still occurred they were less harmful.

69. Learnings
• Operations  Automation
• Development  Abstraction
• Organization  Team
• Culture  Experiment

70. Join the band
Engineers wanted in
NYC & Stockholm
http://spotify.com/jobs