1. © 2015 Concurrent, Inc. All rights reserved.
ACHIEVINGOPERATIONAL
READINESSONHADOOP
9 BEST PRACTICES FOR THE ENTERPRISE
BY SUPREET OBEROI
VP of Field Engineering, Concurrent

2. © 2015 Concurrent, Inc. All rights reserved.
Back then, only hobbyists and status seekers drove
their own vehicles, which were temperamental, to put
it mildly. To operate a car you pretty much had to be a
mechanic, because the only thing you could count on
was a breakdown. The dashboard showed little, if any,
actionable information. Only after that changed, and you
could reliably get from here to there without a wrench,
did automobiles really take off.
I draw the analogy to Hadoop because nearly every day
I hear from enterprise IT teams in industries like retail,
finance, health care, and insurance who thought Hadoop
was a Camry, and are learning that it’s more like whatever
predated the Model T. These teams were led to believe
they could replicate the business successes of Twitter,
LinkedIn, and Netflix simply taking a shiny new cluster
for a spin, but now they’re struggling to deploy Hadoop
applications with the standards of quality, reliability, and
manageability that they have come to expect.
In short, Hadoop, out of the box, is not operationally ready.
In this paper, I’ll share 9 best practices for how IT
organization can achieve operational readiness on
Hadoop. Of course, there are not yet formal certifications
or commonly accepted standards for overcoming the
many challenges. But there is an emerging consensus
around how Hadoop applications are best built, deployed,
and managed.
GettingstartedwithHadoop
isalittlelikebuyingacar—
100yearsago.

3. Buildcultureandtoolssupporting
collaborationbetweendevelopers,
operators,&otherHadoopteammembers
Operators are not trained — and often can’t or don’t want to be trained — to look
at the Java stack traces and debug code. Likewise, asking a developer to address
performance problems is a little like asking a passenger to get out of your car and look
under the hood (assuming your passenger is not a mechanic).
Therefore, at least for the foreseeable future, optimizing performance on Hadoop is
a team proposition. In the best case, an operator who detects a problem can easily
collaborate with the developers, data scientists, and even business managers who have
stakes in the application running smoothly. They should all see operational readiness as
a shared responsibility, and be armed with tools that show them the complete picture
when remediation is required.
1

4. © 2015 Concurrent, Inc. All rights reserved.
2
Connectexecutionproblemswith
applicationcontext
At newer companies, especially those in the San Francisco Bay Area, I find that one
person typically handles all the work — the data science, the development, and the
deployment to production — around a big data application. If there are problems when
the app runs, that same person can usually fix it. After all, she wrote it.
For big, traditional enterprises, it’s a different story. The operations team running fraud
and risk detection apps on Hadoop might live in Phoenix, while the team that developed
them sits ten time zones away in India. In some cases, the operations team today is
completely different from the one that first deployed an application.
Therefore, when a Hadoop job fails or takes too long to execute, operators should be
able to quickly link problems not only to the application that caused them, but also to
the relevant data flow logic inside the application. It’s also great if the operator can
immediately see detailed mapper/reducer stats tied to the problem. That way, they can
more quickly understand if the problem is with the code, the data or the hardware.
When a performance
problem arises, operators
should be able to investigate
app logic and cluster usage.

5. To an operator running hundreds or thousands of applications on a Hadoop cluster,
all of them look the same — until there’s a problem. So you need tools that let you
look at performance over groups of applications. Ideally, you should be able to
segment performance tracking by application types, departments, teams and data-
sensitivity levels.
Monitorthefleet,notthevehicle
3

6. © 2015 Concurrent, Inc. All rights reserved.
Monitoring a fleet still means knowing when an individual vehicle performs poorly.
Similarly, operators need to set SLA bounds on performance and define alerts and
escalation paths when they’re violated.
Every business is unique, so there’s no set list of performance metrics to monitor. But it’s
certainly looking at more than what you can see in log files.
SLA bounds should incorporate both raw metadata such as job status, as well as
business-level events like sensitive data access. Successful practitioners of operational
readiness also set up metrics that help predict future SLA violations, so they can
proactively address and avoid them.
Defineandenforceservicelevel
agreements—Yes,evenonHadoop
4

7. © 2015 Concurrent, Inc. All rights reserved.
Understandinter-appdependencies
5 At technology companies blessed with enough capital to run a dedicated Hadoop
cluster for every use case, applications run more or less independently. That’s not the
case, however, at larger, more traditional enterprises, which tend to run their clusters
as a shared service across lines of business. As a result, each application has at least
a few “roommates” in the cluster, some of which can be noisy, disruptive, or otherwise
detrimental to its own performance.
To understand what’s behind the errant behavior of one Hadoop application, in other
words, you have to first understand what others were doing on the cluster when it
ran. Did a rogue app hog resources, causing others to perform poorly? Was the poor
performance of one application actually due to its dependency on data from some other
application, upstream, that failed to operate as expected?
Provide your operations team with as much cluster-related context as you can. For
example, just by tracking cluster usage by application, you’ll more quickly understand
when an SLA violation is really about a rogue app, rather than a problem in the one that
triggered the alert.

8. © 2015 Concurrent, Inc. All rights reserved.
Tracking applications
that consume more
than 10,000 mappers
Establishing and enforcing the rules for rationing cluster resources is vital for achieving a
meaningful state of operational readiness and meeting SLA contracts. You may have to
handle unusual edge cases. For example, is it acceptable for a recommender engine to
meets its SLA contract in terms of spitting out recommendations but totally consuming a
700-node cluster for the duration of its execution? (I saw this happen in real life!)
RationYourCluster
6 To optimize cluster usage and ROI, operators must ration resources on the cluster and
enforce the limits.
For example, an operator can budget 10,000 mappers for the execution of a particular
application. Then, the onus is on the application to do two things: comply with the
budget restriction, and then demonstrate that compliance. Lacking such proof, rationing
rules should prevent the application from being deployed on the cluster. After all, the
application is not trustworthy.

9. © 2015 Concurrent, Inc. All rights reserved.
Solving for data lineage and governance in an unstructured environment like Hadoop is
no easy task. Traditional techniques to manually maintain a metadata dictionary quickly
lead to stale and old repositories. In addition, there is no proof that the model that is
deployed in production is using the fields described in the metadata repository.
What is required is visibility and enforcement at the operational level on the use of data
fields. If you can track if and when a data field is accessed by an app, you can make the
case you need to make.
Tracking the lineage
of data fields at an
operational level
Tracedataaccessattheoperationallevel
7 Good Hadoop management isn’t only about rationing compute resources; it also means
regulating access to sensitive data. This is especially true in industries with heightened
privacy concerns, such as financial services, health care, insurance, even, these days,
social media.
For example, a data scientist may develop a vastly improved new model for reducing
lending risk, but unless the enterprise can prove that the application does not use any
private data, it cannot deploy the application to production.

10. © 2015 Concurrent, Inc. All rights reserved.
Recorddatamisfires
8 Compliance folks at large enterprises also want proof that a Hadoop application
processed every record in a dataset, and they look for documentation when it fails
to do so. Failures can result from format changes in upstream data sets or plain old
data corruption. Keeping track of all records that the application failed to process is
particularly vital in regulated industries.

11. © 2015 Concurrent, Inc. All rights reserved.
Tuneyourenginebeforeyoureplaceit
9 With new compute fabrics emerging all the time, teams are sometimes too quick to junk
their old ones in pursuit of better performance. However, it’s very often the case that
you can achieve equal or greater performance gains just by optimizing code and data
flows on your existing fabrics. That way you can avoid expensive infrastructure upgrades
unless they’re truly necessary.

12. © 2015 Concurrent, Inc. All rights reserved.
OperationalreadinessonHadoop:
Youcangettherefromhere Supreet is the Vice President of Field
Engineering at Concurrent. Prior to that,
he was Director of Big Data application
infrastructure for American Express, where
he led the development of use cases for
fraud, operational risk, marketing and
privacy on Big Data platforms. He holds
multiple patents in data engineering and
has held leadership positions at Real-Time
Innovations, Oracle, and Microsoft.
When Henry Ford launched the Model T, it was rugged and robust, the first
car fit for a broader market. The car was not only affordable to buy; it was
also practical to own and operate.
Perhaps Hadoop will get there, too. Until then, there’s a lot you can do to
avoid getting left on the side of the big data road.
If you’re looking for help with operational readiness on Hadoop, or are
curious about the charts and displays I’ve shown here, get in touch with me
at sales@concurrentinc.com or visit concurrentinc.com.
ABOUT SUPREET OBEROI
@supreet_online
www.concurrentinc.com
sales@concurrentinc.com