Setting up Hadoop YARN Clustering

1. Danairat T., 2013, danairat@gmail.comBig Data Hadoop – Hands On Workshop
Setting up Hadoop Clustering
Hands-On Workshop
Danairat T.
Line ID: Danairat
FB: Danairat Thanabodithammachari
+668-1559-1446, danairat@gmail.com, Certified Java Programmer

2. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Big Data Introduction
Volume
Variety Velocity
DB Table
Delimited Text
XML, HTML
Free Form Text
Image, Music, VDO, Binary
Batch
Near real time
Real time
GB
TB
PB
XB
ZB

3. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Big Data Architecture
Big Data InfrastructureBig Data Infrastructure
BI/Report
Next Best
Action
Distributed Data Processing
Integration and Metadata Framework
Distributed Data Store and DWH
Monitoring
and
Management
Framework
Security
Framework
Predictive
Analytics
Descriptive
Analytics
Prescriptive
Analytics
Big Data Platform
Big Data Applications
Hardware, Storage, Network
Fraud
Analysis
Cyber
Security
Talent
Search

4. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Hadoop Timeline

5. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Apache Hadoop Core Technology
j2eedev.org/ecosystem-hadoop

6. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Apache Hadoop Ecosystem
j2eedev.org/ecosystem-hadoop

7. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Big Data Platform & Big Data Analytics
Hadoop Technology

8. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Block Size = 64MB
Replication Factor = 3
HDFS: Hadoop Distributed File System
Cost/GB is a few
¢/month vs $/month
apache.org/hadoop/

9. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
YARN: Yet Another Resource Negotiator
Hadoop.apache.org
MRV2 maintains API compatibility with previous stable release
(hadoop-1.x). This means that all Map-Reduce jobs should still run
unchanged on top of MRv2 with just a recompile.

10. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Hadoop 1.0 vs Hadoop 2.0
Hortonwork.com

11. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Hadoop 1.0 vs Hadoop 2.0
Hortonwork.com

12. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Hadoop 2
Hortonworks.com

13. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Hadoop Symbols and Reasons Behind
1
3

14. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Clone hadoop master to slave1 and slave2
master
slave1
slave2

15. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master node: Edit host file

16. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master node : Copy key file to slave1 and slave2
scp /home/ubuntu/.ssh/id_dsa.pub ip-172-31-1-8:/home/ubuntu/.ssh/master.pub
scp /home/ubuntu/.ssh/id_dsa.pub 172.31.15.16:/home/ubuntu/.ssh/master.pub

17. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
After this slide, we will use 3 cascaded
windows to represent master node, slave1
node and slave2 node
master node
slave1 node
slave2 node

18. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1 and slave2: cat /home/ubuntu/.ssh/master.pub >> /home/ubuntu/.ssh/authorized_keys

19. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: Test ssh to slave1 and slave 2
$ ssh ip-172-31-1-8
$ exit
$ ssh ip-172-31-15-16
$ exit

20. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: add slave1 and slave2 to Hadoop slave file

21. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: add slave1 and slave2 to Hadoop slave file

22. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: edit hdfs-site.xml

23. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: edit hdfs-site.xml for 2 replication servers

24. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At all nodes: remove directories of namenode and datanode

25. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: format namenode

26. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: format namenode

27. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: Execute start-dfs.sh

28. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1: Check jps result, you will see DataNode has been started

29. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave2: Check jps result, you will see DataNode has been started

30. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: Execute start-yarn.sh

31. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1: Check jps result, you will see NodeManager has been started

32. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave2: Check jps result, you will see NodeManager has been started

33. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Importing data into HDFS Cluster

34. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: import data to hdfs

35. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1: review imported result data from hdfs

36. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave2: review imported result data from hdfs

37. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Running MapReduce in Cluster Mode

38. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: execute YARN mapreduce program

39. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1, slave2: you will see Application Master and Yarn Child Container

40. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: review output file from hdfs

41. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: review output file from hdfs

42. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1, slave2: review output file from hdfs by using command:-
hdfs dfs -cat /outputs/wordcount_output_dir01/part-r-00000

43. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: review output result data from
web console

44. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: review output result data from
web console

45. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: review output result data from
web console

46. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: review output result data from
web console

47. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Stopping Hadoop Cluster

48. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: execute stop-yarn.sh

49. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1: use jps to review NodeManager has been stopped

50. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave2: use jps to review NodeManager has been stopped

51. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At master: execute stop-dfs.sh

52. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave1: use jps to review DataNode has been stopped

53. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
At slave2: use jps to review DataNode has been stopped

54. Danairat T., danairat@gmail.com:Big Data Hadoop – Hands On Workshop
Thank you very much
Danairat T.
Line ID: Danairat
FB: Danairat Thanabodithammachari
+668-1559-1446, danairat@gmail.com, Certified Java Programmer