HDFS stores files as blocks that are by default 64 MB in size to minimize disk seek times. The namenode manages the file system namespace and metadata, tracking which datanodes store each block. When writing a file, HDFS breaks it into blocks and replicates each block across multiple datanodes. The secondary namenode periodically merges namespace and edit log changes to prevent the log from growing too large. Small files are inefficient in HDFS due to each file requiring namespace metadata regardless of size.

1. HDFS Internals
Bhupesh Chawda
bhupesh@apache.org
DataTorrent

2. Image Source: https://help.marklogic.com/news/list/Index/10
Agenda

3. What are Blocks?
A physical storage disk has a block size - minimum amount of data it can
read or write. Normally 512 bytes.
File systems for a single disk also deal with data in blocks. Normally few kilo
bytes (4 kb).
Hadoop has a much larger block size. By default it is 64 mb.
Files in HDFS are broken down into block sized chunks and are stored as
independent units.
However, files smaller than a block size do not occupy the entire block.
Should I care?

4. Why so large blocks?
Minimize disk seek times
Assuming 10 ms of seek time, and 100 MB/s as disk transfer rate, if block
size if 100 MB, then seek time is 1% of transfer time which is small enough
to ignore.
Hence default is 64 MB while many production environments also use 128
MB.

5. HDFS Architecture
Image Source: https://hadoop.apache.org

6. Namenode and Datanode
Master - Namenode
Manages file system namespace
File system tree and metadata for all files and directories
Stores this info in -
Namespace image
Edit log
Knows for a given file which datanodes has the corresponding blocks. Reconstructed at
startup

7. HDFS Storage
Image Source: https://developer.yahoo.com/hadoop/tutorial/module2.html

8. Secondary Namenode
Image Source: http://www.quickmeme.com/meme/35ke38

9. Secondary Namenode
Not a backup namenode
Periodically merge the namespace image with the edit log, if edit log
becomes too large
Usually runs on a different machine than the namenode
The secondary however always lags behind primary and hence the merged
copy cannot be used in case of primary failure
In event of primary failure, copy the primary namespace image to the
secondary and run it as the new primary.

10. Writing a File in HDFS

11. Reading a file in HDFS

12. HDFS Block Placement

13. Small File Problem?
Each file occupies namespace irrespective of file size!!

14. Further Reading
HDFS Comics :-) https://docs.google.com/open?id=0B-
zw6KHOtbT4MmRkZWJjYzEtYjI3Ni00NTFjLWE0OGItYTU5OGMxYjc0N2M1

15. Sample:

16. Thank You!!
Please send your questions at:
bhupesh@apache.org / bhupesh@datatorrent.com