Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Hadoop Architecture | HDFS Architecture | Hadoop Tutorial | Edureka

2.141 Aufrufe

Veröffentlicht am

This Edureka Hadoop Architecture Tutorial (Hadoop Architecture blog: https://goo.gl/hna58X) will help you understand the architecture of Apache Hadoop in detail. Below are the topics covered in this Hadoop Architecture Tutorial:

1) Hadoop Components
2) DFS – Distributed File System
3) HDFS Services
4) Blocks in Hadoop
5) Block Replication
6) Rack Awareness
7) HDFS Architecture
8) HDFS Read/Write Mechanisms
9) Hadoop HDFS Commands

Check our complete Hadoop playlist here: https://goo.gl/ExJdZs

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Hadoop Architecture | HDFS Architecture | Hadoop Tutorial | Edureka

  1. 1. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
  2. 2. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Agenda for today’s Session 1. Hadoop Components 2. DFS – Distributed File System 3. HDFS Services 4. Blocks in Hadoop 5. Block Replication 6. Rack Awareness 7. HDFS Architecture 8. HDFS Read/Write Mechanism
  3. 3. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Big Data Storage & Computation ? Storing Big Data was a Problem Even if a part of Big Data is Stored- Processing it, took Years
  4. 4. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Solves Big Data Problems Storing Big Data was no more a Problem And Processing did not take Years
  5. 5. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop has a Distributed File System. But Why?
  6. 6. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING DFS – Distributed File System
  7. 7. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING DFS – Distributed File System
  8. 8. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Components 2 main Hadoop Components Storage Processing
  9. 9. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop 2.x Daemons
  10. 10. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING NameNode and DataNode NameNode • Master daemon • Maintains and Manages DataNodes • Records metadata e.g. location of blocks stored, the size of the files, permissions, hierarchy, etc. • Receives heartbeat and block report from all the DataNodes DataNode • Slave daemons • Stores actual data • Serves read and write requests from the clients
  11. 11. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING A Quick Sneak peek of a Hadoop Cluster Architecture
  12. 12. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Cluster Architecture – Master Slave Topology Master Machine Slave Machines distributed over the cluster
  13. 13. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Let us talk about, how data is stored in HDFS?
  14. 14. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Blocks • Each file is stored on HDFS as blocks • The default size of each block is 128 MB in Apache Hadoop 2.x (64 MB in Apache Hadoop 1.x) • Let us say, I have a file example.txt of size 248 MB. Below is the representation of how it will be stored on HDFS
  15. 15. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Blocks • Each file is stored on HDFS as blocks • The default size of each block is 128 MB in Apache Hadoop 2.x (64 MB in Apache Hadoop 1.x) • Let us say, I have a file example.txt of size 248 MB. Below is the representation of how it will be stored on HDFS How many blocks will be created if a file of size 514 MB is copied to HDFS ?
  16. 16. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Blocks • Each file is stored on HDFS as blocks • The default size of each block is 128 MB in Apache Hadoop 2.x (64 MB in Apache Hadoop 1.x) • Let us say, I have a file example.txt of size 248 MB. Below is the representation of how it will be stored on HDFS How many blocks will be created if a file of size 514 MB is copied to HDFS ?
  17. 17. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Is it safe to have just 1 copy of each block? What do you think?
  18. 18. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture – Block Replication
  19. 19. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING How does Hadoop decide where to store the replica’s of the blocks created?
  20. 20. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture: Rack Awareness
  21. 21. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture: Rack Awareness
  22. 22. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture: Rack Awareness
  23. 23. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture: Rack Awareness
  24. 24. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture: Rack Awareness
  25. 25. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture: Rack Awareness
  26. 26. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop Architecture: Rack Awareness
  27. 27. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Architecture of HDFS
  28. 28. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Architecture
  29. 29. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Read/Write Mechanism
  30. 30. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Write Mechanism – Pipeline Setup
  31. 31. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Write Mechanism – Writing a Block
  32. 32. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Write Mechanism - Acknowledgment
  33. 33. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Multi-Block Write Mechanism For Block A: 1A -> 2A -> 3A -> 4A For Block B: 1B -> 2B -> 3B -> 4B -> 5B -> 6B
  34. 34. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING HDFS Read Mechanism
  35. 35. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Let us see few Hadoop/HDFS Commands
  36. 36. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop/HDFS Commands Check the version of Hadoop Command: hadoop version Check HDFS Health Command: hdfs fsck /
  37. 37. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop/HDFS Commands List all the HDFS files/directories Command: hdfs dfs –ls / Copy data from local system to HDFS Command: hdfs dfs –put /home/edureka/test /user
  38. 38. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Hadoop/HDFS Commands List all the HDFS commands and the syntax to use them Command: hdfs dfs –help
  39. 39. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Learning Resources  Hadoop Tutorial: www.edureka.co/blog/hadoop-tutorial  HDFS Tutorial: www.edureka.co/blog/hdfs-tutorial  Hadoop Architecture: www.edureka.co/blog/apache-hadoop-hdfs-architecture
  40. 40. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING Thank You … Questions/Queries/Feedback

×