1. BIG DATA

2. The following topics will be covered in our
BIG DATA
Online Training:
Copyright @ 2015 Learntek. All Rights Reserved. 2

3. What is Hadoop?
Big Data Hadoop Training: Hadoop is a free, Java -based programming
framework that supports the processing of large data sets in a distributed
computing environment. It is part of the Apache project sponsored by the
Apache Software Foundation. Hadoop makes it possible to run applications on
systems with thousands of nodes involving thousands of terabytes of storage
capacity. Its distributed file system facilitates rapid data transfer rates among
nodes and allows the system to continue operating uninterrupted in case of a
node failure. This approach lowers the risk of catastrophic system failure, even
if a significant number of nodes become inoperative.
Copyright @ 2015 Learntek. All Rights Reserved.

4. Why Hadoop?
• Large Volumes of Data: Ability to store and process huge amounts of variety (structure,
unstructured and semi structured) of data, quickly. With data volumes and varieties
constantly increasing, especially from social media and the Internet of Things (IoT), that’s a
key consideration.
• Computing Power: Hadoop’s distributed computing model processes big data fast. The more
computing nodes you use, the more processing power you have.
• Fault Tolerance: Data and application processing are protected against hardware failure. If a
node goes down, jobs are automatically redirected to other nodes to make sure the
distributed computing does not fail. Multiple copies of all data are stored automatically.
• Flexibility: Unlike traditional relational database, you don’t have to process data before
storing it, You can store as much data as you want and decide how to use it later. That
includes unstructured data like text, images and videos etc.
• Low Cost: The open-source framework is free and used commodity hardware to store large
quantities of data.
• Scalability: You can easily grow your system to handle more data simply by adding nodes.
Little administration is required.
Copyright @ 2015 Learntek. All Rights Reserved. 4

5. Big Data Hadoop Training: Hadoop Introduction
• Big Data Hadoop Training:
Introduction to Data and System
• Types of Data
• Traditional way of dealing large
data and its problems
• Types of Systems & Scaling
• What is Big Data
• Challenges in Big Data
• Challenges in Traditional
Application
• New Requirements
• What is Hadoop? Why Hadoop?
• Brief history of Hadoop
• Features of Hadoop
• Hadoop and RDBMS
• Hadoop Ecosystem’s overview
Copyright @ 2015 Learntek. All Rights Reserved. 5

6. Hadoop Installation
• Installation in detail
• Creating Ubuntu image in
VMwareDownloading Hadoop
• Installing SSH
• Configuring Hadoop, HDFS &
MapReduce
• Download, Installation &
Configuration Hive
• Download, Installation &
Configuration Pig
• Download, Installation &
Configuration Sqoop
• Download, Installation &
Configuration Hive
• Configuring Hadoop in Different
Modes
Copyright @ 2015 Learntek. All Rights Reserved. 6

7. Hadoop Distribute File System (HDFS)
Copyright @ 2015 Learntek. All Rights Reserved. 7
• File System – Concepts
• Blocks
• Replication Factor
• Version File
• Safe mode
• Namespace IDs
• Purpose of Name Node
• Purpose of Data Node
• Purpose of Secondary Name
Node
• Purpose of Job Tracker
• Purpose of Task Tracker
• HDFS Shell Commands –
copy, delete, create
directories etc.
• Reading and Writing in HDFS
• Difference of Unix
Commands and HDFS
commands
• Hadoop Admin Commands
• Hands on exercise with Unix
and HDFS commands
• Read / Write in HDFS –
Internal Process between
Client, NameNode &
DataNodes.
• Accessing HDFS using Java
API
• Various Ways of Accessing
HDFS
• Understanding HDFS Java
classes and methods
• Admin: 1. Commissioning /
DeCommissioning DataNode
• Balancer
• Replication Policy
• Network Distance / Topology
Script

8. Map Reduce Programming
• About MapReduce
• Understanding block and
input splits
• MapReduce Data types
• Understanding Writable
• Data Flow in MapReduce
Application
• Understanding MapReduce
problem on datasets
• MapReduce and Functional
Programming
• Writing MapReduce
Application
• Understanding Mapper
function
• Understanding Reducer
Function
• Understanding Driver
• Usage of Combiner
• Understanding Partitioner
• Usage of Distributed Cache
• Passing the parameters to
mapper and reducer
• Analysing the Results
• Log files
• Input Formats and Output
Formats
• Counters, Skipping Bad and
unwanted Records
• Writing Join’s in MapReduce
with 2 Input files. Join Types.
• Execute MapReduce Job –
Insights.
• Exercise’s on MapReduce.
• Job Scheduling: Type of
Schedulers.
Copyright @ 2015 Learntek. All Rights Reserved. 8

9. Hive
• Hive concepts
• Schema on Read VS Schema on
Write
• Hive architecture
• Install and configure hive on
cluster
• Meta Store – Purpose & Type of
Configurations
• Different type of tables in Hive
• Buckets
• Partitions
• Joins in hive
• Hive Query Language
• Hive Data Types
• Data Loading into Hive Tables
• Hive Query Execution
• Hive library functions
• Hive UDF
• Hive Limitations
Copyright @ 2015 Learntek. All Rights Reserved. 9

10. Pig
• Pig basics
• Install and configure PIG on a cluster
• PIG Library functions
• Pig Vs Hive
• Write sample Pig Latin scripts
• Modes of running PIG
• Running in Grunt shell
• Running as Java program
• PIG UDFs
Copyright @ 2015 Learntek. All Rights Reserved. 10

11. HBase
• HBase concepts
• HBase architecture
• Region server architecture
• File storage architecture
• HBase basics
• Column access
• Scans
• HBase use cases
• Install and configure HBase on a
multi node cluster
• Create database, Develop and
run sample applications
• Access data stored in HBase
using Java API
Copyright @ 2015 Learntek. All Rights Reserved. 11

12. Sqoop
• Install and configure Sqoop on cluster
• Connecting to RDBMS
• Installing Mysql
• Import data from Mysql to hive
• Export data to Mysql
• Internal mechanism of import/export
Copyright @ 2015 Learntek. All Rights Reserved. 12

13. Oozie
• Introduction to OOZIE
• Oozie architecture
• XML file specifications
• Specifying Work flow
• Control nodes
• Oozie job coordinator
Copyright @ 2015 Learntek. All Rights Reserved. 13

14. Flume
• Introduction to Flume
• Configuration and Setup
• Flume Sink with example
• Channel
• Flume Source with example
• Complex flume architecture
Copyright @ 2015 Learntek. All Rights Reserved. 14

15. ZooKeeper
• Introduction to ZooKeeper
• Challenges in distributed Applications
• Coordination
• ZooKeeper : Design Goals
• Data Model and Hierarchical namespace
• Cilent APIs
Copyright @ 2015 Learntek. All Rights Reserved. 15

16. YARN
• Hadoop 1.0 Limitations
• MapReduce Limitations
• History of Hadoop 2.0
• HDFS 2: Architecture
• HDFS 2: Quorum based storage
• HDFS 2: High availability
• HDFS 2: Federation
• YARN Architecture
• Classic vs YARN
• YARN Apps
• YARN multitenancy
• YARN Capacity Scheduler
Copyright @ 2015 Learntek. All Rights Reserved. 16

17. Prerequisites :
• Knowledge in any programming language, Database knowledge and
Linux Operating system. Core Java or Python knowledge helpful.
Copyright @ 2015 Learntek. All Rights Reserved. 17

18. Copyright @ 2015 Learntek. All Rights Reserved. 18