Hadoop has become a backbone of many enterprises. While it can do wonders for businesses, it sometimes can be overwhelming for its operators and users. Amateurs as well as seasoned operators of Hadoop are caught unaware by common pitfalls of deploying, tuning and operating a Hadoop cluster. Having spent 5+ years working with 100s of Hadoop users, running clusters with 1000s of nodes, managing 10s of petabytes of data and running 100s of 1000s of tasks per day, we have seen people's unintentional acts, suboptimal configurations and common mistakes have resulted into downtimes, SLA violations, many hours of recovery operations and in some cases even data loss! Most of these traumas could have been easily avoided by applying easy to follow best practices that would protect data and optimize performance. In this talk we present real life stories, common pitfalls and most importantly, strategies on how to correctly deploy and manage Hadoop clusters. The talk will empower users and help make their Hadoop journey more fulfilling and rewarding. We will also discuss SmartSense. SmartSense can identify latent problems in a cluster and provide recommendations so that an operator can fix them before they manifest as a service degradation or outage.
SmartSense bundles include configuration, and metrics, and bundles used for Support Case troubleshooting included configuration, metrics, and log files. This data is captured for the Operating System of cluster nodes, as well as for all of the installed HDP services.
The capture process can be configured to exclude specific files from capture, or specific Hadoop properties within HDP configuration files. In order to provide protection to organization-specific data, such as customer ID’s, patient ID’s, Credit Card #’s, etc. We provide the capability to specify a regular expression that can be removed or replaced in any file that is captured by SmartSense. This allows protection of sensitive data in the event that data is unintentionally leaked into log files.
By default we remove all properties associated with clear text passwords. Ambari, Hive, and Oozie by default store DB credentials as cleartext, unless they’ve been configured to encrypt them. Just in case Hadoop Operators have not taken the time to do so, we exclude those properties by default.