1. Pig Power Tools
a quick tour
By
Viswanath Gangavaram
Data Scientist
R&D, DSG, Ilabs, [24] 7 INC
4/27/2014 1
Pig provides a higher level of abstraction for data users, giving them access to the power and
flexibility of Hadoop without requiring them to write extensive data processing applications
in low-level Java Code(MapReduce code). From the preface of “Programming Pig”

2. What we are going to cover
 A very short introduction to Apache Pig
 Use Grunt shell to work with the Hadoop Distributed File System
 Advanced Pig Operators(relational)
 Pig Macros and Modularity features
 Embed Pig Latin in Python for Iterative Processing and other advanced tasks(SIMOD golden Journeys)
 Json Parsing
 XML Parsing
 UDFs(Jython)
 Pig Streaming
 UDFs Vs. Streaming
 Custom load and store Functions to handle data formats and storage mechanisms
 Single Row Relations
 Python in Pig(Bringing nltk, numpy, scipy, pandas into pig)
 Lipstick
 Hue
 Performance Tips
 External libraries
 Piggybank, DataFu, DataFu Hour Glass, SimpleJson, ElephantBird
Note:- This is general Pig tutorial, will have minimum references to any particular data set
4/27/2014 2

3. A short introduction to “Apache Pig” in five minutes
4/27/2014 3
• Apache Pig is a high-level platform for executing data flows in parallel on Hadoop. The language for this
platform is called Pig Latin, which includes operators for many of the traditional data operations (join,
sort, filter, etc.), as well as the ability for users to develop their own functions for reading, processing, and
writing data.
– Pigs fly
• Pig processes data quickly. Designers want to consistently improve its performance, and not
implement features in ways that weigh pig down so it can't fly.
• What does it mean to be Pig?
– Pigs Eats Everything
• Pig can operate on data whether it has metadata or not. It can operate on data that is
relational, nested, or unstructured. And it can easily be extended to operate on data beyond
files, including key/value stores, databases, etc.
– Pigs Live Everywhere
• Pig is intended to be a language for parallel data processing. It is not tied to one
particular parallel framework. Check for Pig on Tez
– Pigs Are Domestic Animals
• Pig is designed to be easily controlled and modified by its users.
• Pig allows integration of user code where ever possible, so it currently supports user defined
field transformation functions, user defined aggregates, and user defined conditionals.
• Pig supports user provided load and store functions.
• It supports external executables via its stream command and Map Reduce jars via its
MapReduce command.
• It allows users to provide a custom partitioner for their jobs in some circumstances and to set
the level of reduce parallelism for their jobs.

4. 4/27/2014 4
Apache Pig “Word counting is hello world of MapReduce”
inputFile = LOAD ‘mary’ as ( line );
words = FOREACH inputFile GENERATE FLATTEN( TOKENIZE(line) ) as word;
grpd = GROUP words by word;
cntd = FOREACH grpd GENERATE group, COUNT(words)
DUMP cntd;
Output:-
(This , 2)
(is, 2)
(my, 2 )
(first , 2)
(apache, 2)
(pig,2)
(program, 2)
“mary” file content:-
This is my first apache pig program
This is my first apache pig program

5. 4/27/2014 5
Apache Pig Latin: A data flow language
• Pig Latin is a dataflow language. This means it allows users to describe how data from one or more inputs
should be read, processed, and then stored to one or more outputs in parallel.
• To be mathematically precise, a Pig Latin script describes a directed acyclic graph (DAG), where the edges
are data flows and the nodes are operators that process the data.
Comparing query(HIVE/SQL) and data flow languages(PIG)
• After a cursory look, people often say that Pig Latin is a procedural version of SQL. Although there are
certainly similarities, there are more differences. SQL is a query language. Its focus is to allow users to
form queries. It allows users to describe what question they want answered, but not how they want it
answered. In Pig Latin, on the other hand, the user describes exactly how to process the input data.
• Another major difference is that SQL is oriented around answering one question. When users want to do
several data operations together, they must either write separate queries, storing the intermediate data
into temporary tables, or write it in one query using subqueries inside that query to do the earlier steps of
the processing. However, many SQL users find subqueries confusing and difficult to form properly. Also,
using subqueries creates an inside-out design where the first step in the data pipeline is the innermost
query.
• Pig, however, is designed with a long series of data operations in mind, so there is no need to write the
data pipeline in an inverted set of subqueries or to worry about storing data in temporary tables.
• SQL is the English of data processing. It has the nice feature that everyone and every tool knows it, which
means the barrier to adoption is very low. Our goal is to make Pig Latin the native language of parallel
data-processing systems such as Hadoop. It may take some learning, but it will allow users to utilize the
power of Hadoop much more fully. - Extracted from “Programming Pig”

6. Pig’s Data types
 Scalar types
• int, long, float, double, chararray, bytearray
 Complex types
• Map
– A map in Pig is a chararray to data element mapping, where that element can be any Pig
type, including a complex type.
– The chararray is called a key and is used as index to find the element, referred to as the
value.
– Map constants are formed using brackets to delimit the map, a hash between keys and
values, and a comma between key-value pairs.
» *‘dept’#’dsg’, ‘team’#’r&d’+
• Tuple
– A tuple is a fixed-length, ordered collection of Pig data elements. Tuples are divided into
fields, with each field containing one data element. These elements can be of any type.
– Tuple constants use parentheses to indicate the tuple and commas to delimit fields in
the tuple.
» (‘boss’, 55)
• Bag
– A bag is an unordered collection of tuples.
– Bag constants are constructed using braces, with the tuples in the bag separated by
commas.
» { (‘a’, 20), (‘b’, 20), (‘c’, 30) }
4/27/2014 6

7. • Nulls
– Pig includes the concept of a data element being null. Data of any type can be null. A null data
element means the value is unknown. This might be because the data is missing, an error occurred
in processing it, etc.
4/27/2014 7
• Schemas
– Pig has a very lax attitude when it comes to schemas. This is a consequence of Pig’s philosophy of
eating anything
• Casts

8. Basic operators
1. LOAD
2. STORE
3. LIMIT
4. DEFINE
5. FOREACH
6. FILTER
7. DISTINCT
8. (CO)GROUP
9. JOIN
10. UNION
11. CROSS
12. ORDER BY
4/27/2014 8

9. Grunt shell
• Grunt is Pig’s interactive shell. It enables users to enter Pig Latin interactively and
provides a shell to interact with HDFS
4/27/2014 9
• Command line history, editing, Tab completion.
• No-pipes, no re-directions and no-background executions
• Grunts shell commands
– Shell for HDFS*
• fs –ls, fs –du, fs –stat, etc…
– Shell for Unix commands (working in the local directory)
• sh ls, sh cat
– exec
– run
– Kill jobid
– Set
– dump
– explain
– describe
• *: fs is default

10. Advanced operators
1. ASSERT
2. CUBE
3. IMPORT
4. MAPREDUCE
5. ORDER BY
6. RANK
7. SAMPLE
8. SPLIT
9. STREAM
4/27/2014 10

11. Pig’s Debugging tools
 Use the DUMP operator to display results to your terminal screen.
 Use the DESCRIBE operator to review the schema of a relation.
 Use the EXPLAIN operator to view the logical, physical, or map reduce execution plans to
compute a relation.
 Use the ILLUSTRATE operator to view the step-by-step execution of a series of
statements.
4/27/2014 11
Shortcuts for Debugging operators
 d alias - shortcut for DUMP. If alias is ignored last defined alias will be used.
 de alias - shortcut for DESCRIBE. If alias is ignored last defined alias will be used.
 e alias - shortcut for EXPLAIN. If alias is ignored last defined alias will be used.
 i alias - shortcut for ILLUSTRATE. If alias is ignored last defined alias will be used.
 q - To quit grunt shell

12. Json Parsing
4/27/2014 12

13. XML Parsing
4/27/2014 13

14. User Defined Functions
4/27/2014 14

15. Pig Streaming
4/27/2014 15

16. UDFs Vs. Pig streaming
4/27/2014 16

17. Cython in Pig(Bringing nltk, numpy, scipy, pandas into pig)
4/27/2014 17

18. Lipstick:- Let’s add some color to Pig
4/27/2014 18

19. Hue:- Hadoop and its ecosystem in Browser
4/27/2014 19

20. Piggybank
4/27/2014 20

21. DataFu
4/27/2014 21

22. DataFu Hourglass
4/27/2014 22

23. SimpleJson
4/27/2014 23

24. Elephant Bird
4/27/2014 24

25. So what is pig?
4/27/2014 25
Pig is a champion