Currently telecom companies store their data in database or data warehouse, treating them through ETL process and working on statistics and analysis by using OLAP tools or data mining engines. However, due to the data explosion along with the spread of Smart Phones traditional data storages like DB and DW aren’t sufficient to cope with these “Big Data”. As an alternative the method of storing data in Hadoop and performing ETL process and Ad-hoc Query with Hive is being introduced, and China Mobile is being mentioned as the most representative example. But, they are adopted mainly by new projects, which have low barriers in applying the new Hive data model and HQL. On the other hand, it is extremely difficult to replace the existing database with the combination of Hadoop and Hive if there are already a number of tables and SQL queries. NexR is migrating the telecom company’s data from Oracle DB to Hadoop, and converting a lot of existing Oracle SQL queries to Hive HQL queries. Though HQL supports a similar syntax to ANSI-SQL, it lacks a large portion of basic functions and hardly supports Oracle analytic functions like rank() which are utilized mainly in statistical analysis. Furthermore, the difference of data types like null value is also blocking the application of it. In this presentation, we will share the experience converting Oracle SQL to Hive HQL and developing additional functions with MapReduce. Also, we will introduce several ideas and trials to improve Hive performance. http://sdec.kr/

1. SDEC 2011
Seoul Data Engineering Camp
June 27-28
Seoul, South Korea

2. Replacing Legacy Telco DB/DW
to Hadoop and Hive
JunHo Cho
NexR

3. Agenda

4. Agenda
• Motivation for Hive and Hadoop

5. Agenda
• Motivation for Hive and Hadoop
• Hive Internal

6. Agenda
• Motivation for Hive and Hadoop
• Hive Internal
• Oracle Migration UseCase

7. Agenda
• Motivation for Hive and Hadoop
• Hive Internal
• Oracle Migration UseCase
• Hive Optimization

8. Agenda
• Motivation for Hive and Hadoop
• Hive Internal
• Oracle Migration UseCase
• Hive Optimization
• Future Work

9. Telco Data

10. Telco Data

11. Telco Data

12. Telco Data

13. Telco Data

14. Telco Data

15. Telco Data

16. Telco Data

27. qu er
Co n
de &
Di vi

28. OpenSource

29. OpenSource
Storage & Computing

30. OpenSource

31. OpenSource
Collection

32. OpenSource

33. OpenSource
Search

34. OpenSource

35. OpenSource
Analysis

36. OpenSource

37. OpenSource
Coordination

38. OpenSource

42. What is HIVE ?

43. What is HIVE ?
• A system for managing and querying structured data
built on top of Hadoop
• Map-Reduce for execution
• HDFS for storage
• Metadata in an RDBMS

44. What is HIVE ?
• A system for managing and querying structured data
built on top of Hadoop
• Map-Reduce for execution
• HDFS for storage
• Metadata in an RDBMS
• Key Building Principles
• SQL is a familiar language
• Extensibility - Types, Functions, Formats, Scripts
• Performance

45. Why Hive ?

47. Count call-record per phone ?

49. public class CallCountMapper extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {
private final IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(WritableComparable key, Writable value,
OutputCollector output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line.toLowerCase());
word.set(itr.nextToken());
output.collect(word, one);
}
}

50. public class CallCountMapper extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {
private final IntWritable one = new IntWritable(1);
er
private Text word = new Text();
app
public void map(WritableComparable key, Writable value,
M
OutputCollector output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line.toLowerCase());
word.set(itr.nextToken());
output.collect(word, one);
}
}

51. public class CallCountMapper extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {
private final IntWritable one = new IntWritable(1);
er
private Text word = new Text();
app
public void map(WritableComparable key, Writable value,
M
OutputCollector output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line.toLowerCase());
word.set(itr.nextToken());
output.collect(word, one);
}
}
public class CallCountReducer extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator values,
OutputCollector output, Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
IntWritable value = (IntWritable) values.next();
sum += value.get(); // process value
}
output.collect(key, new IntWritable(sum));
}
}

52. public class CallCountMapper extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {
private final IntWritable one = new IntWritable(1);
er
private Text word = new Text();
app
public void map(WritableComparable key, Writable value,
M
OutputCollector output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line.toLowerCase());
word.set(itr.nextToken());
output.collect(word, one);
}
}
public class CallCountReducer extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator values,
er
OutputCollector output, Reporter reporter) throws IOException {
uc
int sum = 0;
ed
while (values.hasNext()) {
R
IntWritable value = (IntWritable) values.next();
sum += value.get(); // process value
}
output.collect(key, new IntWritable(sum));
}
}

53. public class CallCountMapper extends MapReduceBase public class CallCount {
implements Mapper<LongWritable, Text, Text, IntWritable> {
public static void main(String[] args) {
private final IntWritable one = new IntWritable(1); JobClient client = new JobClient();
er
private Text word = new Text(); JobConf conf = new JobConf(WordCount.class);
app
public void map(WritableComparable key, Writable value, // specify output types
M
OutputCollector output, Reporter reporter) throws IOException { conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line.toLowerCase()); // specify input and output dirs
word.set(itr.nextToken()); FileInputPath.addInputPath(conf, new Path("input"));
output.collect(word, one); FileOutputPath.addOutputPath(conf, new Path("output"));
}
} // specify a mapper
conf.setMapperClass(KeyCountMapper.class);
// specify a reducer
conf.setReducerClass(CallCountReducer.class);
conf.setCombinerClass(CallCountReducer.class);
public class CallCountReducer extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> { client.setConf(conf);
try {
public void reduce(Text key, Iterator values, JobClient.runJob(conf);
er
OutputCollector output, Reporter reporter) throws IOException { } catch (Exception e) {
uc
e.printStackTrace();
int sum = 0; }
ed
while (values.hasNext()) { }
R
IntWritable value = (IntWritable) values.next(); }
sum += value.get(); // process value
}
output.collect(key, new IntWritable(sum));
}
}

54. public class CallCountMapper extends MapReduceBase public class CallCount {
implements Mapper<LongWritable, Text, Text, IntWritable> {
public static void main(String[] args) {
private final IntWritable one = new IntWritable(1); JobClient client = new JobClient();
er
private Text word = new Text(); JobConf conf = new JobConf(WordCount.class);
app
public void map(WritableComparable key, Writable value, // specify output types
M
OutputCollector output, Reporter reporter) throws IOException { conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
String line = value.toString();
er
StringTokenizer itr = new StringTokenizer(line.toLowerCase()); // specify input and output dirs
riv
word.set(itr.nextToken()); FileInputPath.addInputPath(conf, new Path("input"));
output.collect(word, one); FileOutputPath.addOutputPath(conf, new Path("output"));
D
}
} // specify a mapper
conf.setMapperClass(KeyCountMapper.class);
// specify a reducer
conf.setReducerClass(CallCountReducer.class);
conf.setCombinerClass(CallCountReducer.class);
public class CallCountReducer extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> { client.setConf(conf);
try {
public void reduce(Text key, Iterator values, JobClient.runJob(conf);
er
OutputCollector output, Reporter reporter) throws IOException { } catch (Exception e) {
uc
e.printStackTrace();
int sum = 0; }
ed
while (values.hasNext()) { }
R
IntWritable value = (IntWritable) values.next(); }
sum += value.get(); // process value
}
output.collect(key, new IntWritable(sum));
}
}

56. SELECT pnum, count(pnum)
FROM cdr
GROUP BY pnum;

57. History of Hive
• Hive development cycle is fast and the developer
community is growing rapidly
• Product release cycle is accelerating
Project
started 0.3.0 0.4.0 0.5.0 0.6.0 0.7.0 0.7.1
03/08 4/09 12/09 02/10 10/10 03/11 06/11

58. History of Hive
• Hive development cycle is fast and the developer
community is growing rapidly
• Product release cycle is accelerating
Project
started 0.3.0 0.4.0 0.5.0 0.6.0 0.7.0 0.7.1
03/08 4/09 12/09 02/10 10/10 03/11 06/11

59. History of Hive
• Hive development cycle is fast and the developer
community is growing rapidly
• Product release cycle is accelerating
Project
started 0.3.0 0.4.0 0.5.0 0.6.0 0.7.0 0.7.1
03/08 4/09 12/09 02/10 10/10 03/11 06/11

60. Who use Hive?
http://wiki.apache.org/hadoop/Hive/PoweredBy

61. UseCase in Hive?

62. UseCase in Hive?
• Report and ad hoc query

63. UseCase in Hive?
• Report and ad hoc query
• Log Analysis

64. UseCase in Hive?
• Report and ad hoc query
• Log Analysis
• Social Graph Analysis

65. UseCase in Hive?
• Report and ad hoc query
• Log Analysis
• Social Graph Analysis
• Data mining and analysis

66. UseCase in Hive?
• Report and ad hoc query
• Log Analysis
• Social Graph Analysis
• Data mining and analysis
• Machine Learning

67. UseCase in Hive?
• Report and ad hoc query
• Log Analysis
• Social Graph Analysis
• Data mining and analysis
• Machine Learning
• Dataset cleaning

68. UseCase in Hive?
• Report and ad hoc query
• Log Analysis
• Social Graph Analysis
• Data mining and analysis
• Machine Learning
• Dataset cleaning
• Data Warehouse

69. Hive Architecture
UI Driver
DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result

70. Hive Architecture
UI Driver select col1 from tab1 where ...
DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result

71. Hive Architecture
UI Driver
DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result

72. Hive Architecture
UI Driver
DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result

73. Hive Architecture
UI Driver
DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result

74. Hive Architecture
a 123344
b 121211
c 342434
UI Driver
DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result

75. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

76. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

77. Parser
Parser
Select col1,col2 From tab1 Where col3 > 5
TOK_QUERY
TOK_FROM TOK_INSERT
TOK_DESTINATION TOK_SELECT TOK_WHERE
TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR
TOK_DIR
>
TOK_TABLE_OR_COL TOK_TABLE_OR_COL
TOK_TMP_FILE
TOK_TABLE_OR_COL 5

78. Parser
Parser
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_QUERY
TOK_FROM TOK_INSERT
TOK_DESTINATION TOK_SELECT TOK_WHERE
TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR
TOK_DIR
>
TOK_TABLE_OR_COL TOK_TABLE_OR_COL
TOK_TMP_FILE
TOK_TABLE_OR_COL 5

79. Parser
Parser
Select col1,col2 From tab1 Where col3 > 5
TOK_QUERY
TOK_FROM TOK_INSERT
TOK_DESTINATION TOK_SELECT TOK_WHERE
TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR
QB tab1
TOK_DIR
>
TOK_TABLE_OR_COL TOK_TABLE_OR_COL
TOK_TMP_FILE
TOK_TABLE_OR_COL 5

80. Parser
Parser
Select col1,col2 From tab1 Where col3 > 5
TOK_QUERY
TOK_FROM TOK_INSERT
TOK_DESTINATION TOK_SELECT TOK_WHERE
TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR
tab1
TOK_DIR
>
TOK_TABLE_OR_COL TOK_TABLE_OR_COL
TOK_TMP_FILE
TOK_TABLE_OR_COL 5
QB insclause-0

81. Parser
Parser
Select col1,col2 From tab1 Where col3 > 5
TOK_QUERY
TOK_FROM TOK_INSERT
TOK_DESTINATION TOK_SELECT TOK_WHERE
TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR
tab1
TOK_DIR
>
TOK_TABLE_OR_COL TOK_TABLE_OR_COL
TOK_TMP_FILE
col1 QB
TOK_TABLE_OR_COL 5
insclause-0

82. Parser
Parser
Select col1,col2 From tab1 Where col3 > 5
TOK_QUERY
TOK_FROM TOK_INSERT
TOK_DESTINATION TOK_SELECT TOK_WHERE
TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR
tab1
TOK_DIR
>
TOK_TABLE_OR_COL TOK_TABLE_OR_COL
col1 col2 QB
TOK_TMP_FILE
TOK_TABLE_OR_COL 5
insclause-0

83. Parser
Parser
Select col1,col2 From tab1 Where col3 > 5
TOK_QUERY
TOK_FROM TOK_INSERT
TOK_DESTINATION TOK_SELECT TOK_WHERE QB
TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR
tab1
TOK_DIR
>
TOK_TABLE_OR_COL TOK_TABLE_OR_COL
col1 col2
TOK_TMP_FILE
TOK_TABLE_OR_COL 5
insclause-0

84. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

85. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

86. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB

87. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM
TOK_WHERE
TOK_SELECT
TOK_DESTINATION

88. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM TableScanOperator
TOK_WHERE
TOK_SELECT
TOK_DESTINATION

89. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM TableScanOperator
TOK_WHERE
TOK_SELECT
TOK_DESTINATION

90. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM TableScanOperator
TOK_WHERE FilterOperator
TOK_SELECT
TOK_DESTINATION

91. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM TableScanOperator
TOK_WHERE FilterOperator
TOK_SELECT
TOK_DESTINATION

92. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM TableScanOperator
TOK_WHERE FilterOperator
TOK_SELECT SelectOperator
TOK_DESTINATION

93. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM TableScanOperator
TOK_WHERE FilterOperator
TOK_SELECT SelectOperator
TOK_DESTINATION

94. Plan
Plan
Select col1,col2 From tab1 Where col3 > 5
QB
TOK_FROM TableScanOperator
TOK_WHERE FilterOperator
TOK_SELECT SelectOperator
TOK_DESTINATION FileSinkOperator

95. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

96. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

97. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
TableScanOperator
FilterOperator
SelectOperator
FileSinkOperator

98. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FilterOperator
SelectOperator
FileSinkOperator

99. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FilterOperator
SelectOperator
FileSinkOperator

100. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
Context
TableScanOperator
FilterOperator
ColumnPruner
SelectOperator
FileSinkOperator

101. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
Context
TableScanOperator
FilterOperator FIL
ColumnPruner TS
SEL
SelectOperator
FileSinkOperator

102. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FilterOperator FIL
ColumnPruner TS
SEL
SelectOperator
FileSinkOperator Context

103. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FilterOperator
ColumnPruner
SelectOperator
FIL
FileSinkOperator Context TS
SEL

104. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FilterOperator
ColumnPruner
FIL
SelectOperator Context TS
SEL
FileSinkOperator

105. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FilterOperator
ColumnPruner
FIL
SelectOperator Context TS
SEL col1, col2
FileSinkOperator

106. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FilterOperator
ColumnPruner
FIL
SelectOperator Context TS
SEL
FileSinkOperator

107. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FIL col1, col2, col3
FilterOperator Context TS
ColumnPruner
SEL
SelectOperator
FileSinkOperator

108. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
TableScanOperator
FIL
FilterOperator Context TS
ColumnPruner
SEL
SelectOperator
FileSinkOperator

109. Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5
tab1 {col1, col2, col3, col4,col5,col6,col7}
FIL
TableScanOperator Context TS col1, col2, col3
SEL
FilterOperator
ColumnPruner
FilterOperator
SelectOperator
FileSinkOperator

110. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

111. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

112. Task
Task Select col1,col2 From tab1 Where col3 > 5
TS - GenMRTableScan1
TaskFactory
FS - GenMRFileSink1
QB

113. Task
Task Select col1,col2 From tab1 Where col3 > 5
TS - GenMRTableScan1
TaskFactory
FS - GenMRFileSink1
QB
FetchTask

114. Task
Task Select col1,col2 From tab1 Where col3 > 5
TS - GenMRTableScan1
TaskFactory
FS - GenMRFileSink1
QB
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

115. Task
Task Select col1,col2 From tab1 Where col3 > 5
TS - GenMRTableScan1
TaskFactory
FS - GenMRFileSink1
QB
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

116. Task
Task Select col1,col2 From tab1 Where col3 > 5
TaskFactory
FS - GenMRFileSink1
QB
MapRedTask
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

117. Task
Task Select col1,col2 From tab1 Where col3 > 5
TaskFactory
FS - GenMRFileSink1
QB
MapRedTask
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

118. Task
Task Select col1,col2 From tab1 Where col3 > 5
TaskFactory
FS - GenMRFileSink1
QB
MapRedTask
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

119. Task
Task Select col1,col2 From tab1 Where col3 > 5
TaskFactory
FS - GenMRFileSink1
QB
MapRedTask
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

120. Task
Task Select col1,col2 From tab1 Where col3 > 5
TaskFactory
FS - GenMRFileSink1
QB
MapRedTask
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

121. Task
Task Select col1,col2 From tab1 Where col3 > 5
TaskFactory
QB
MapRedTask
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

122. Task
Task Select col1,col2 From tab1 Where col3 > 5
TaskFactory
QB
MapRedTask
MapRedTask
TableScanOperator
FilterOperator FetchTask
FilterOperator
SelectOperator
FileSinkOperator

123. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF
FILOperator SELOperator
MetaStore Hive QL FILOperator FSOperator
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

124. Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF
FILOperator SELOperator
MetaStore Hive QL FILOperator FSOperator
Thrift API Parser ExecMapper/ExecReducer
Plan SerDe
Optimizer Input/OutputFormat
Task
HDFS StorageHandler
RCFile
DB ... HBase

125. Oracle Migration
to Hive

126. l
l
l
l
l

127. l l
l l
l l
l l
l l

128. l l
l l
l l
l l
l l

129. Oracle SQL

130. Data Model
Hive Entity Sample HDFS LOC

131. Data Model
Hive Entity Sample HDFS LOC
Table

132. Data Model
Hive Entity Sample HDFS LOC
Table Log /hive/Log

133. Data Model
Hive Entity Sample HDFS LOC
Table Log /hive/Log
Partition

134. Data Model
Hive Entity Sample HDFS LOC
Table Log /hive/Log
Partition time=hour /hive/Log/time=1h

135. Data Model
Hive Entity Sample HDFS LOC
Table Log /hive/Log
Partition time=hour /hive/Log/time=1h
Bucket

136. Data Model
Hive Entity Sample HDFS LOC
Table Log /hive/Log
Partition time=hour /hive/Log/time=1h
/wh/Log/time=1h/
Bucket phone-num
part-$hash(phone-num)

137. Data Model
Hive Entity Sample HDFS LOC
Table Log /hive/Log
Partition time=hour /hive/Log/time=1h
/wh/Log/time=1h/
Bucket phone-num
part-$hash(phone-num)
External
Table

138. Data Model
Hive Entity Sample HDFS LOC
Table Log /hive/Log
Partition time=hour /hive/Log/time=1h
/wh/Log/time=1h/
Bucket phone-num
part-$hash(phone-num)
External /app/meta/dir
customer (arbitrary location)
Table

139. Data Model
MetaStore HDFS
Table
Data Location Partition
Bucketing Info
Partitioning Info
part-001
Bucket
Partition
MetaStore DB
/hive/Log
/hive/Log/time=1h
/hive/Log/time=1h/part-0001

140. Column Data Types

141. Column Data Types
• Primitive Types
• int type : tinyint, smallint, int, bigint
• boolean, float, double, string

142. Column Data Types
• Primitive Types
• int type : tinyint, smallint, int, bigint
• boolean, float, double, string
• Nest-able Collections
• array : value(any-type)
• map : key(primitive) and value(any-type)

143. Column Data Types
• Primitive Types
• int type : tinyint, smallint, int, bigint
• boolean, float, double, string
• Nest-able Collections
• array : value(any-type)
• map : key(primitive) and value(any-type)
• User-defined types
• structures with attributes

144. DataType Convert

145. DataType Convert
NUMBER(n)

146. DataType Convert
NUMBER(n) TINYINT
INT/BIGINT

147. DataType Convert
NUMBER(n) TINYINT
INT/BIGINT
NUMBER(n,m)

148. DataType Convert
NUMBER(n) TINYINT
INT/BIGINT
NUMBER(n,m) FLOAT/DOUBLE

149. DataType Convert
NUMBER(n) TINYINT
INT/BIGINT
NUMBER(n,m) FLOAT/DOUBLE
VARCHAR2

150. DataType Convert
NUMBER(n) TINYINT
INT/BIGINT
NUMBER(n,m) FLOAT/DOUBLE
VARCHAR2 STRING

151. DataType Convert
NUMBER(n) TINYINT
INT/BIGINT
NUMBER(n,m) FLOAT/DOUBLE
VARCHAR2 STRING
DATE

152. DataType Convert
NUMBER(n) TINYINT
INT/BIGINT
NUMBER(n,m) FLOAT/DOUBLE
VARCHAR2 STRING
DATE STRING
“yyyy-MM-dd HH:mm:ss” format

153. Oracle DML
• HIVE supports ANSI-SQL
• Sub-Queries in FROM clause
• Join query : equi-join/inner-join , outer-join

154. Range Operator

155. Range Operator
BETWEEN ~ AND ~

156. Range Operator
BETWEEN ~ AND ~
SELECT * from Employee WHERE
salary BETWEEN 100 AND 500;

157. Range Operator
BETWEEN ~ AND ~
SELECT * from Employee WHERE
salary BETWEEN 100 AND 500;
SELECT * from Employee WHERE
salary >= 100 AND salary <=500;

158. Range Operator
BETWEEN ~ AND ~
SELECT * from Employee WHERE
salary BETWEEN 100 AND 500;
SELECT * from Employee WHERE
salary >= 100 AND salary <=500;
SELECT * from Employee WHERE
BETWEEN(salary,100,500);

159. IN / EXISTS Clause

160. IN / EXISTS Clause
IN / EXISTS SubQuery

161. IN / EXISTS Clause
IN / EXISTS SubQuery
SELECT * from Employee e WHERE e.DeptNo
IN(SELECT d.DeptNo FROM Dept d)

162. IN / EXISTS Clause
IN / EXISTS SubQuery
SELECT * from Employee e WHERE e.DeptNo
IN(SELECT d.DeptNo FROM Dept d)
SELECT * from Employee e WHERE
EXISTS(SELECT )
1 FROM Dept d WHERE e.DeptNo=d.DeptNo

163. IN / EXISTS Clause
IN / EXISTS SubQuery
SELECT * from Employee e WHERE e.DeptNo
IN(SELECT d.DeptNo FROM Dept d)
SELECT * from Employee e WHERE
EXISTS(SELECT 1 FROM Dept d WHERE e.DeptNo=d.DeptNo )
SELECT * from Employee e
LEFT SEMI JOIN Dept d ON (e.DeptNo=d.DeptNo)

164. NOT IN Clause

165. NOT IN Clause
NOT IN SubQuery

166. NOT IN Clause
NOT IN SubQuery
SELECT * from Employee e WHERE e.DeptNo
NOT IN(SELECT d.DeptNo FROM Dept d)

167. NOT IN Clause
NOT IN SubQuery
SELECT * from Employee e WHERE e.DeptNo
NOT IN(SELECT d.DeptNo FROM Dept d)
SELECT e.* from Employee e
LEFT OUTER JOIN Dept d ON (e.DeptNo=d.DeptNo)
WHERE d.DeptNo IS NULL

168. NOT EXIST Clause

169. NOT EXIST Clause
NOT EXIST SubQuery

170. NOT EXIST Clause
NOT EXIST SubQuery
SELECT * from Employee e WHERE
NOT EXISTS(SELECT 1 FROM Dept d WHERE e.DeptNo=d.DeptNo )

171. NOT EXIST Clause
NOT EXIST SubQuery
SELECT * from Employee e WHERE
NOT EXISTS(SELECT 1 FROM Dept d WHERE e.DeptNo=d.DeptNo )
SELECT e.* from Employee e
LEFT OUTER JOIN Dept d ON (e.DeptNo=d.DeptNo)
WHERE d.DeptNo IS NULL

172. LIKE Clause

173. LIKE Clause
LIKE / NOT LIKE

174. LIKE Clause
LIKE / NOT LIKE
SELECT * from Employee e WHERE name LIKE ’%steve’

175. LIKE Clause
LIKE / NOT LIKE
SELECT * from Employee e WHERE name LIKE ’%steve’
SELECT e.* from Employee e WHERE name LIKE ‘%steve’

176. LIKE Clause
LIKE / NOT LIKE
SELECT * from Employee e WHERE name LIKE ’%steve’
SELECT * from Employee e WHERE name NOT LIKE ’%steve’
SELECT e.* from Employee e WHERE name LIKE ‘%steve’

177. LIKE Clause
LIKE / NOT LIKE
SELECT * from Employee e WHERE name LIKE ’%steve’
SELECT * from Employee e WHERE name NOT LIKE ’%steve’
SELECT e.* from Employee e WHERE name LIKE ‘%steve’
SELECT e.* from Employee e WHERE NOT name LIKE ‘%steve’

178. LIKE Clause
LIKE / NOT LIKE
SELECT * from Employee e WHERE name LIKE ’%steve’
SELECT * from Employee e WHERE name NOT LIKE ’%steve’
SELECT e.* from Employee e WHERE name LIKE ‘%steve’
SELECT e.* from Employee e WHERE NOT name LIKE ‘%steve’

179. JOIN Operator (1/4)

180. JOIN Operator (1/4)
SELF JOIN

181. JOIN Operator (1/4)
SELF JOIN
SELECT *
FROM Employee e1, Employee e2 WHERE e1.ID = e2.Id

182. JOIN Operator (1/4)
SELF JOIN
SELECT *
FROM Employee e1, Employee e2 WHERE e1.ID = e2.Id
SELECT *
FROM Employee e1 JOIN Employee e2 ON (e1.ID = e2.Id )

183. JOIN Operator (2/4)

184. JOIN Operator (2/4)
CROSS JOIN (Cartesian Product)

185. JOIN Operator (2/4)
CROSS JOIN (Cartesian Product)
SELECT emp.Name, dept.Name FROM Employee emp, Dept dep

186. JOIN Operator (2/4)
CROSS JOIN (Cartesian Product)
SELECT emp.Name, dept.Name FROM Employee emp, Dept dep
SELECT emp.Name, dept.Name FROM Employee emp JOIN Dept dep

187. JOIN Operator (3/4)

188. JOIN Operator (3/4)
LEFT OUTER JOIN

189. JOIN Operator (3/4)
LEFT OUTER JOIN
FROM Emp, Dept
SELECT *
WHERE Emp.deptNo = Dept.deptNo(+)

190. JOIN Operator (3/4)
LEFT OUTER JOIN
FROM Emp, Dept
SELECT *
WHERE Emp.deptNo = Dept.deptNo(+)
FROM Emp
SELECT *
LEFT OUTER JOIN Dept ON Emp.deptNO = Dept.deptNo

191. JOIN Operator (4/4)

192. JOIN Operator (4/4)
RIGHT OUTER JOIN

193. JOIN Operator (4/4)
RIGHT OUTER JOIN
FROM Emp, Dept
SELECT *
WHERE Emp.deptNo(+) = Dept.deptNo

194. JOIN Operator (4/4)
RIGHT OUTER JOIN
FROM Emp, Dept
SELECT *
WHERE Emp.deptNo(+) = Dept.deptNo
FROM Emp
SELECT *
RIGHT OUTER JOIN Dept ON Emp.deptNO = Dept.deptNo

195. Oracle Function

196. Condition Function

197. Condition Function
CASE

198. Condition Function
CASE
CASE expr WHEN THEN r1
cond1
[WHEN cond2 THEN r2]* [ELSE r] END

199. Condition Function
CASE
CASE expr WHEN THEN r1
cond1
[WHEN cond2 THEN r2]* [ELSE r] END
CASE expr WHEN THEN r1
cond1
[WHEN cond2 THEN r2]* [ELSE r] END

200. Math Function

201. Math Function
ROUND

202. Math Function
ROUND ROUND

203. Math Function
ROUND ROUND
CEIL

204. Math Function
ROUND ROUND
CEIL CEIL/CEILING

205. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD

206. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD PMOD

207. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD PMOD
POWER

208. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD PMOD
POWER POW/POWER

209. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD PMOD
POWER POW/POWER
SQRT

210. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD PMOD
POWER POW/POWER
SQRT SQRT

211. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD PMOD
POWER POW/POWER
SQRT SQRT
SIN/COS

212. Math Function
ROUND ROUND
CEIL CEIL/CEILING
MOD PMOD
POWER POW/POWER
SQRT SQRT
SIN/COS SIN/COS

213. Character Function

214. Character Function
SUBSTR

215. Character Function
SUBSTR SUBSTR

216. Character Function
SUBSTR SUBSTR
TRIM

217. Character Function
SUBSTR SUBSTR
TRIM TRIM

218. Character Function
SUBSTR SUBSTR
TRIM TRIM
LPAD/RPAD

219. Character Function
SUBSTR SUBSTR
TRIM TRIM
LPAD/RPAD LPAD/RPAD

220. Character Function
SUBSTR SUBSTR
TRIM TRIM
LPAD/RPAD LPAD/RPAD
LTRIM/RTRIM

221. Character Function
SUBSTR SUBSTR
TRIM TRIM
LPAD/RPAD LPAD/RPAD
LTRIM/RTRIM LTRIM/RTRIM

222. Character Function
SUBSTR SUBSTR
TRIM TRIM
LPAD/RPAD LPAD/RPAD
LTRIM/RTRIM LTRIM/RTRIM
REPLACE

223. Character Function
SUBSTR SUBSTR
TRIM TRIM
LPAD/RPAD LPAD/RPAD
LTRIM/RTRIM LTRIM/RTRIM
REPLACE REGEXP_REPLACE

224. NULL Function

225. NULL Function
COALESCE

226. NULL Function
COALESCE COALESCE

227. NULL Function
COALESCE COALESCE
NVL

228. NULL Function
COALESCE COALESCE
NVL Custom UDF

229. NULL Function
COALESCE COALESCE
NVL Custom UDF
NVL2

230. NULL Function
COALESCE COALESCE
NVL Custom UDF
NVL2 Custom UDF

231. Custom UDF Function
• Condition Function
• DECODE
• Null Comparison Function
• NVL / NVL2
• Type Conversion
• TO_NUMBER
• TO_CHAR
• TO_DATE

232. Oracle Analytic
Function

233. Analytic Function

234. Analytic Function
Joins, WHERE, GROUP BY clauses are performed

235. Analytic Function
Joins, WHERE, GROUP BY clauses are performed
the analytic functions are performed
with the result set

236. Analytic Function
Joins, WHERE, GROUP BY clauses are performed
the analytic functions are performed
with the result set
ORDER BY clause is processed

237. Analytic Function
Rank salary in dept
name
dept
salary
---------------------
a
Research
100
b
Research
100
c
Sales
200
d
Sales
300
e
Research
50
f
Accounting
200
g
Accounting
300
h
Accounting
400
i
Research
10

238. Analytic Function
name
dept
salary
---------------------
a
Research
100
b
Research
100
c
Sales
200
d
Sales
300
e
Research
50
f
Accounting
200
g
Accounting
300
h
Accounting
400
i
Research
10

239. Analytic Function
Map
name
dept
salary
---------------------
a
Research
100
b
Research
100
c
Sales
200
d
Sales
300
e
Research
50
Map
f
Accounting
200
g
Accounting
300
h
Accounting
400
i
Research
10
Map

240. Analytic Function
a
Research
100
b
Research
100
c
Sales
Map
200
d
Sales
300
e
Research
Map
50
f
Accounting
200
g
Accounting
300
h
Accounting
Map
400
i
Research
10

241. Analytic Function
DISTRIBUTED BY dept
a
Research
100
b
Research
100
c
Sales
Map
200
d
Sales
300
e
Research
Map
50
f
Accounting
200
g
Accounting
300
h
Accounting
Map
400
i
Research
10

242. Analytic Function
DISTRIBUTED BY dept
a
Research
100
b
Research
100
c
Sales
Map
200
Reduce
d
Sales
300
e
Research
Map
50
f
Accounting
200
Reduce
g
Accounting
300
h
Accounting
Map
400
i
Research
10

243. Analytic Function
DISTRIBUTED BY dept
c Sales 200
Map g Accounting
300
h
d
Accounting
Sales
400
300
Reduce
f Accounting
200
Map g
Research
300
h
Research
400
e Research 300 Reduce
i Research 10
Map

244. Analytic Function
SORT BY dept, salary
c Sales 200
Map d Sales 300
f Accounting
200
g Accounting
300 Reduce
h Accounting
400
Map i Research 10
g
Research
300
e Research 300 Reduce
h
Research
400
Map

245. Analytic Function
c Sales 200
Map d Sales 300
f Accounting
200
g Accounting
300 Reduce
h Accounting
400
Map i Research 10
g
Research
300
e Research 300 Reduce
h
Research
400
Map

246. Analytic Function
RANK(dept,salary)
c Sales 200 1
Map d Sales 300 2
f Accounting
200 1
Reduce g Accounting
300 2
h Accounting
400 3
Map i Research 10 1
g
Research
300 2
Reduce e Research 300 3
h
Research
400 4
Map

247. Analytic Function

248. Analytic Function
RANK

249. Analytic Function
RANK
SELECT name,dept,salary,RANK() OVER (PARTITION BY dept
ORDER BY salary DESC) FROM emp

250. Analytic Function
RANK
SELECT name,dept,salary,RANK() OVER (PARTITION BY dept
ORDER BY salary DESC) FROM emp
SELECT e.name,e.dept,e.salary,RANK( e.dept,e.salary)
FROM (SELECT name, dept, salary FROM empDISTRIBUTED
BY dept SORT BY dept, salary DESC) e

251. Analytic Function
RANK
SELECT name,dept,salary,RANK() OVER (PARTITION BY dept
ORDER BY salary DESC) FROM emp
RANK(arg1,arg2) - Custom UDF
SELECT e.name,e.dept,e.salary,RANK( e.dept,e.salary)
FROM (SELECT name, dept, salary FROM empDISTRIBUTED
BY dept SORT BY dept, salary DESC) e

252. Hive Optimization
& Future Work

253. Tuning Parameter

254. Tuning Parameter
• Hadoop Tunning

255. Tuning Parameter
• Hadoop Tunning
• mapred.job.reuse.jvm.num.task

256. Tuning Parameter
• Hadoop Tunning
• mapred.job.reuse.jvm.num.task
• mapred.child.java.opts

257. Tuning Parameter
• Hadoop Tunning
• mapred.job.reuse.jvm.num.task
• mapred.child.java.opts
• mapred.min.split.size / mapred.max.split.size

258. Tuning Parameter
• Hadoop Tunning
• mapred.job.reuse.jvm.num.task
• mapred.child.java.opts
• mapred.min.split.size / mapred.max.split.size
• dfs.block.size

259. Tuning Parameter
• Hadoop Tunning
• mapred.job.reuse.jvm.num.task
• mapred.child.java.opts
• mapred.min.split.size / mapred.max.split.size
• dfs.block.size
• Hive Tunning

260. Tuning Parameter
• Hadoop Tunning
• mapred.job.reuse.jvm.num.task
• mapred.child.java.opts
• mapred.min.split.size / mapred.max.split.size
• dfs.block.size
• Hive Tunning
• hive.input.format = CombineHiveInputFormat

261. UDF/UDAF
• Develop UDF to optimize number of MR jobs
• Extend GenericUDF to avoid java reflection
• Avoid creating new objects in UDF

262. Future Work

263. Future Work
• HiveQL SQL Compliance
• HIVE-282 - IN statement for WHERE clauses
• HIVE-192 - Add TIMESTAMP column type
• HIVE-1269 - Support Date/Datetime/Time/Timestamp Primitive Types

264. Future Work
• HiveQL SQL Compliance
• HIVE-282 - IN statement for WHERE clauses
• HIVE-192 - Add TIMESTAMP column type
• HIVE-1269 - Support Date/Datetime/Time/Timestamp Primitive Types
• Analytic Function
• HIVE-896 - Add LEAD/LAG/FIRST/LAST analytical windowing functions to Hive
• HIVE-952 - Support analytic NTILE function

265. Future Work
• HiveQL SQL Compliance
• HIVE-282 - IN statement for WHERE clauses
• HIVE-192 - Add TIMESTAMP column type
• HIVE-1269 - Support Date/Datetime/Time/Timestamp Primitive Types
• Analytic Function
• HIVE-896 - Add LEAD/LAG/FIRST/LAST analytical windowing functions to Hive
• HIVE-952 - Support analytic NTILE function
• Optimization
• HIVE-1694 - Accelerate GROUP BY execution using indexes
• HIVE-482 - Optimize Group By + Order By with the same keys

268. Hive
Oracle 2 Hive

269. Hive
A system for managing and querying
structured data built on top of Hadoop
Oracle 2 Hive

270. Hive
A system for managing and querying
structured data built on top of Hadoop
Oracle 2 Hive
data model
ANSI-SQL
built-in function / custom UDF
analytic function

273. Question ?