Intro to Big Data and the power of women in Goryeo dynasty with the Mongol Empire in Korea

1. HiPIC
Recent IT Development and Women:
Big Data and The Power of Women in Goryeo
KWiSE Annual Meeting
Chapman University, CA
Oct 20th 2012
Jongwook Woo (PhD)
High-Performance Internet Computing Center (HiPIC)
Educational Partner with Cloudera and Grants Awardee of Amazon AWS
Computer Information Systems Department
California State University, Los Angeles
Jongwook Woo
CSULA

2. HiPIC Contents
Part I. Big Data
Fundamentals of Big Data
Data-Intensive Computing: Hadoop
Big Data Supporters and Use Cases
Part II. The Power of Women in Goryeo
Dynasty
North East Asia before the Mongol Empire
Korea and Mongol
The Empress Gi
CSULA
Jongwook Woo

3. HiPIC Part I
Big Data
Fundamentals of Big Data
NoSQL DB: HBase, MongoDB
Data-Intensive Computing: Hadoop
Big Data Supporters and Use Cases
CSULA
Jongwook Woo

4. HiPIC Experience in Big Data
 Grants
 Received Amazon AWS in Education Research Grant (July
2012 - July 2014)
 Received Amazon AWS in Education Coursework Grants (July
2012 - July 2013, Jan 2011 - Dec 2011
 Partnership
 Received Academic Education Partnership with Cloudera since
June 2012
 Certificate
 Certificate of Achievement in the Big Data University Training
Course, “Hadoop Fundamentals I”, July 8 2012
 Cloud Computing Blog
 http://dal-cloudcomputing.blogspot.com/
CSULA
Jongwook Woo

5. What is Big Data, Map/Reduce, Hadoop, NoSQL DB on
HiPIC Cloud Computing
CSULA
Jongwook Woo

6. HiPIC Big Data
Too much data
Tera-Byte (1012), Peta-byte (1015)
– Because of web
– Sensor Data, Bioinformatics, Social
Computing, smart phone, online game…
Cannot handle with the legacy
approach
Too big
Un-/Semi-structured data
CSULA
Jongwook Woo

7. HiPIC Two Issues in Big Data
How to store Big Data
NoSQL DB
How to compute Big Data
Parallel Computing with multiple cheap
computers
– Not need super computers
CSULA
Jongwook Woo

8. HiPIC Contents
Fundamentals of Big Data
Data-Intensive Computing: Hadoop
Big Data Supporters and Use Cases
CSULA
Jongwook Woo

9. HiPIC Data nowadays
• Data Issues
o data grows to 10TB, and then 100TB.
o Unstructured data coming from sources
 like Facebook, Twitter, RFID readers, sensors,
and so on.
 Need to derive information from both the
relational data and the unstructured data
• as soon as possible.
• Solution to efficiently compute Big
Data
o Hadoop Map/Reduce
CSULA
Jongwook Woo

10. HiPIC Solutions in Big Data Computation
 Map/Reduce by Google
(Key, Value) parallel computing
 Apache Hadoop
 Big Data
Data Computation (MapReduce, Pig)
 Integrating MapReduce and RDB
 Oracle + Hadoop
 Sybase IQ
 Vertica + Hadoop
 Hadoop DB
 Greenplum
 Aster Data
 Integrating MapReduce and NoSQL DB
 MongoDB MapReduce
 HBase
CSULA
Jongwook Woo

11. HiPIC Apache Hadoop
 Motivated by Google Map/Reduce and GFS
 open source project of the Apache Foundation.
 framework written in Java
– originally developed by Doug Cutting
• who named it after his son's toy elephant.
 Two core Components
 Storage: HDFS
– High Bandwidth Clustered storage
 Processing: Map/Reduce
– Fault Tolerant Distributed Processing
 Hadoop scales linearly with
 data size
 Analysis complexity
CSULA
Jongwook Woo

12. HiPIC Hadoop issues
 Map/Reduce is not DB
 Algorithm in Restricted Parallel Computing
 HDFS and HBase
 Cannot compete with the functions in RDBMS
 But, useful for
 Semi-structured data model and high-level dataflow query
language on top of MapReduce
– Pig, Hive, Jsql, Cascading, Cloudbase
 Useful for huge (peta- or Terra-bytes) but non-complicated data
– Web crawling
– log analysis
• Log file for web companies
– New York Times case
CSULA
Jongwook Woo

13. HiPIC MapReduce Pros & Cons Summary
Good when
Huge data for input, intermediate, output
A few synchronization required
Read once; batch oriented datasets (ETL)
Bad for
Fast response time
Large amount of shared data
Fine-grained synch needed
CPU-intensive not data-intensive
Continuous input stream
CSULA
Jongwook Woo

14. HiPIC MapReduce in Detail
Functions borrowed from functional
programming languages (eg. Lisp)
Provides Restricted parallel programming
model on Hadoop
User implements Map() and Reduce()
Libraries (Hadoop) take care of
EVERYTHING else
– Parallelization
– Fault Tolerance
– Data Distribution
– Load Balancing
CSULA
Jongwook Woo

15. HiPIC Map
Convert input data to (key, value) pairs
map() functions run in parallel,
 creating different intermediate (key, value)
values from different input data sets
CSULA
Jongwook Woo

16. HiPIC Reduce
reduce() combines those intermediate values
into one or more final values for that same
key
reduce() functions also run in parallel,
each working on a different output key
Bottleneck:
reduce phase can‟t start until map phase is
completely finished.
CSULA
Jongwook Woo

17. HiPIC Example: Sort URLs in the largest hit order
Compute the largest hit URLs
 Stored in log files
Map()
 Input <logFilename, file text>
 Output: Parses file and emits <url, hit counts> pairs
– eg. <http://hello.com, 1>
Reduce()
 Input: <url, list of hit counts> from multiple map
nodes
 Output: Sums all values for the same key and emits
<url, TotalCount>
– eg.<http://hello.com, (3, 5, 2, 7)> => <http://hello.com, 17>
CSULA
Jongwook Woo

18. HiPIC Map/Reduce for URL visits
Input Log Data
Map1() Map2() … Mapm()
(http://hi.com, 1) (http://halo.com, 1)
(http://hello.com, 3) (http://hello.com, 5)
… …
Data Aggregation/Combine
(http://hi.com, <1, 1, …, 1>) (http://halo.com, <1, 5,>)
(http://hello.com, <3, 5, 2, 7>)
Reduce1 () Reduce2() … Reducel()
(http://hi.com, 32) (http://halo.com, 6)
(http://hello.com, 17)
CSULA
Jongwook Woo

19. HiPIC Legacy Example
In late 2007, the New York Times
wanted to make available over the web
its entire archive of articles,
11 million in all, dating back to 1851.
four-terabyte pile of images in TIFF format.
needed to translate that four-terabyte pile of TIFFs
into more web-friendly PDF files.
– not a particularly complicated but large computing chore,
• requiring a whole lot of computer processing time.
CSULA
Jongwook Woo

20. HiPIC Legacy Example (Cont’d)
In late 2007, the New York Times
wanted to make available over the web
its entire archive of articles,
a software programmer at the Times, Derek Gottfrid,
– playing around with Amazon Web Services, Elastic
Compute Cloud (EC2),
• uploaded the four terabytes of TIFF data into Amazon's
Simple Storage System (S3)
• In less than 24 hours, 11 millions PDFs, all stored
neatly in S3 and ready to be served up to visitors to the
Times site.
 The total cost for the computing job? $240
– 10 cents per computer-hour times 100 computers times 24 hours
CSULA
Jongwook Woo

21. HiPIC Contents
Fundamentals of Big Data
Data-Intensive Computing: Hadoop
Big Data Supporters and Use Cases
CSULA
Jongwook Woo

22. HiPIC Supporters of Big Data
 Apache Hadoop Supporters
 Cloudera
– Like Linux and Redhat
– HiPIC is an Academic Partner
 Hortonworks
– Pig,
– Consulting and training
 Facebook
– Hive
 IBM
– Jaql
 NoSQL DB supporters
 MongoDB
– HiPIC tries to collaborate
 HBase, CouchDB, Apache Cassandra (originally by FB) etc
CSULA
Jongwook Woo

23. HiPIC Similarities in Pig, Hive, and Jaql
• translate high-level languages into MapReduce jobs
o the programmer can work at a higher level
 than writing MapReduce jobs in Java or other
lower-level languages
• programs are much smaller than Java code.
• option to extend these languages,
o often by writing user-defined functions in Java.
• Interoperability
o programs written in these high-level languages can
be imbedded inside other languages as well.
CSULA
Jongwook Woo

24. HiPIC Pig
• developed at Yahoo Research around 2006
o moved into the Apache Software Foundation in
2007.
• PigLatin,
o Pig's language
o a data flow language
o well suited to processing unstructured data
 Easy to write MapReduce codes
CSULA
Jongwook Woo

25. HiPIC Hive
• developed at Facebook
o turns Hadoop into a data warehouse
o complete with a dialect of SQL for querying.
• HiveQL
o a declarative language (SQL dialect)
• Difference from PigLatin,
o you do not specify the data flow,
 but instead describe the result you want
 Hive figures out how to build a data flow to
achieve it.
o a schema is required,
CSULA
Jongwook Woo

26. HiPIC Jaql
• developed at IBM.
• a data flow language
o its native data structure format is JSON (JavaScript
Object Notation).
CSULA
Jongwook Woo

27. HiPIC Use Cases
Amazon AWS
Facebook
Twitter
Craiglist
HuffPOst | AOL
CSULA
Jongwook Woo

28. HiPIC Amazon AWS
 amazon.com
 Consumer and seller business
 aws.amazon.com
 IT infrastructure business
– Focus on your business not IT management
 Pay as you go
– Pay for servers by the hour
– Pay for storage per Giga byte per month
– Pay for data transfer per Giga byte
 Services with many APIs
– S3: Simple Storage Service
– EC2: Elastic Compute Cloud
• Provide many virtual Linux servers
• Can run on multiple nodes
– Hadoop and HBase
– MongoDB
CSULA
Jongwook Woo

29. HiPIC Amazon AWS (Cont’d)
Customers on aws.amazon.com
Samsung
– Smart TV hub sites: TV applications are on AWS
Netflix
– ~25% of US internet traffic
– ~100% on AWS
NASA JPL
– Analyze more than 200,000 images
NASDAQ
– Using AWS S3
CSULA
Jongwook Woo

30. HiPIC Facebook [7]
Using Apache HBase
For Titan and Puma
HBase for FB
– Provide excellent write performance and good reads
– Nice features
• Scalable
• Fault Tolerance
• MapReduce
CSULA
Jongwook Woo

31. HiPIC Titan: Facebook
Message services in FB
Hundreds of millions of active users
15+ billion messages a month
50K instant message a second
Challenges
High write throughput
– Every message, instant message, SMS, email
Massive Clusters
– Must be easily scalable
Solution
Clustered HBase
CSULA
Jongwook Woo

32. HiPIC Puma: Facebook
 ETL
 Extract, Transform, Load
– Data Integrating from many data sources to Data Warehouse
 Data analytics
– Domain owners‟ web analytics for Ad and apps
• clicks, likes, shares, comments etc
 ETL before Puma
 8 – 24 hours
– Procedures: Scribe, HDFS, Hive, MySQL
 ETL after Puma
 Puma
– Real time MapReduce framework
 2 – 30 secs
– Procedures: Scribe, HDFS, Puma, HBase
CSULA
Jongwook Woo

33. HiPIC Twitter [8]
Three Challenges
Collecting Data
– Scribe as FB
Large Scale Storage and analysis
– Cassandra: ColumnFamily key-value store
– Hadoop
Rapid Learning over Big Data
– Pig
• 5% of Java code
• 5% of dev time
• Within 20% of running time
CSULA
Jongwook Woo

34. HiPIC Craiglist in MongoDB [9]
Craiglist
~700 cities, worldwide
~1 billion hits/day
~1.5 million posts/day
Servers
– ~500 servers
– ~100 MySQL servers
Migrate to MongoDB
Scalable, Fast, Proven, Friendly
CSULA
Jongwook Woo

35. HiPIC
HuffPost | AOL [10]
Two Machine Learning Use Cases
Comment Moderation
– Evaluate All New HuffPost User Comments Every
Day
• Identify Abusive / Aggressive Comments
• Auto Delete / Publish ~25% Comments Every Day
Article Classification
– Tag Articles for Advertising
• E.g.: scary, salacious, …
build a flexible ML platform running on
Hadoop
Pig for Hadoop implementation.
CSULA
Jongwook Woo

36. HiPIC Conclusion
Era of Big Data
Need to store and compute Big Data
Storage: NoSQL DB
Computation: Hadoop MapRedude
Need to analyze Big Data in mobile
computing, SNS for Ad, User Behavior,
Patterns, Bioinformatics, Medical data …
CSULA
Jongwook Woo

37. HiPIC Part II
The power of Women in Goryeo
Dynasty
North East Asia before the Mongol Empire
Korea and Mongol
The Empress Gi
CSULA
Jongwook Woo

38. HiPIC Three kingdoms (AD 907 - 1125)
CSULA
Jongwook Woo

39. HiPIC Before Mongol
Three kingdoms balanced power
Goryeo, Yo (Liao, Cathay, Khitan, 契丹),
Song
–Goryeo-Yo: 3 wars
• First invasion (AD 993): 서희,
• Second invasion with 400K (AD 1010):
강조
• Third invasion with 100K (AD 1018):
강감찬
– Goryeo became famous after this victory
CSULA
Jongwook Woo

40. HiPIC Three kingdoms (AD 1115- 1234)
CSULA
Jongwook Woo

41. HiPIC Before Mongol
Three kingdoms balanced power
(AD 1115 - 1234)
Goryeo, Gum (Jin, Jurchen, Yojin, 金朝),
South Song
–윤관 invaded Jurchen Wanyan (完顏) clan
(AD 1111) and many battles
–Jin defeated Liao dynasty at AD 1121
– wanted to keep a peace with Goryeo
• From the emperor of big brother to the
king of little brother
CSULA
Jongwook Woo

42. HiPIC
Part II. The power of Women in Goryeo Dynasty
Korea and Mongol
Wars since AD1231 (고종 18)
Goryeo (Korea) dynasty
Military dictatorship of Choe family ended at AD1258
(고종 45)
Mongol
Was conquering China (the South Song dynasty)
since AD1257
– Möngke Kahn
• Right battalion
– Kublai
• Left battalion
CSULA
Jongwook Woo

43. HiPIC Korea and Mongol (Cont’d)
Mongol Empire in 1227 at Genghis Khan„s death
[http://en.wikipedia.org/wiki/Timeline_of_the_Mon
gol_Empire]
CSULA
Jongwook Woo

44. HiPIC Korea and Mongol (Cont’d)
1236 Beginning
invading Europe by
Hulagu
Ariq Böke controlled
1231 Beginning
Mongol at Karakorum
invading Korea
1236 Beginning
invading South Asia
By Möngke Khan and
Kublai
Mongol Empire after Genghis Khan„s death (1227)
under Möngke Khan
[http://en.wikipedia.org/wiki/Timeline_of_the_Mongol
_Empire]
CSULA
Jongwook Woo

45. HiPIC Korea and Mongol (Cont’d)
World in AD1257 – 1260
1257: Mongols was attacking Vietnam
1258: Mongols occupied Baghdad
1259: Mongols was invading Syria
– The death of Möngke Khan
1260: The succession war had begun
– By Möngke‟s brothers : Kublai Khan and Ariq Böke.
– Kublai and the youngest brother Hulugu returned to
KaraKorum: Capital of the Mongol empire
• Kara: north, Korum: Khori (Space, 골, 고을)
CSULA
Jongwook Woo

46. HiPIC Korea and Mongol (Cont’d)
Again Goreyo and Mongol in 1259
Decided to have a peace treaty with Mongol
– Actually to surrender
April 21 1259 (고종 46): The Crown Prince left to
meet the Khan
May 17th 1259: The Crown Prince met Mongol army
at Yoyang (Liao liang) who was about to invade
Goreyo
– Stop the Mongol army
 June 30 1259: The king Go-Jong passed away
 July 30 1259: The Khan passed away
– Mongol army stopped the prince to hide the khan‟s
death
The prince met Kublai at Gaebong close to the
Yellow river
– Dec 1259: Kublai was returning back to KaraKorum CSULA
Jongwook Woo

47. HiPIC Korea and Mongol (Cont’d)
Hulagu
Ariq Böke controlled
Mongol at Karakorum Goryeo‟s Crown
Prince
Kublai
Mongol Empire after Möngke Kahn' death (1227)
[http://en.wikipedia.org/wiki/Timeline_of_the_Mon
gol_Empire]
CSULA
Jongwook Woo

48. HiPIC Goreyo and Mongol in 1260-1264
The great meeting and the great Khan
Kublai welcomed the prince with the glad favor
– Kublai was so happy and said
• “The god is helping me. Goryeo kingdom surrendered
to me, who was never defeated even by the Chinese
emperor Dang Tae-Jong”
• He knew that Goryeo is originated from GoGuRyeo
Kublai appointed the prince to the king of Goryeo
(Won-Jong)
– as Go-Jong passed away
They came together to Beijing on Jan 1260.
April 1260: Won-Jong‟s enthronement ceremony in
Goryeo
 August 21 1264: Ariq Böke surrendered to Kublai
at Xanadu (KaraKorum)
CSULA
Jongwook Woo

49. HiPIC The great meeting and the marriage
 Sept 1264: King Won-Jong went to Beijing and meet
the Khan
 Another great welcoming from the Khan
 1269: Kublai decided his daughter to marry the
crown price of Goryeo
 1269, Aug 1270: Won-Jong and the crown prince asked Kublai for the
marriage
 1271, 1272: the prince went to Beijing and returned back
– Volunteer to lead the invasion of Japan
 April 1273: Defeated Sambyolcho at Jeju island
 May 1274: The crown prince of Goryeo and the
princess of the Mongol (Holdorogerimisil, 제국공주)
empire married at the palace of the capital in the
Mongol empire
 Aug 1274: The prince became the king (충렬왕)
CSULA
Jongwook Woo

50. HiPIC Korea and Mongol (Cont’d)
Mongol Empire in 1300 -1405: this map is not
correct as Goryeo was an independent
kingdom
[http://en.wikipedia.org/wiki/Timeline_of_the_Mon
gol_Empire] CSULA
Jongwook Woo

51. HiPIC Korea and Mongol (Cont’d)
The Mongol Empire and the
Kingdom of Goryeo tied with
marriages
Mongol Empire in
[http://en.wikipedia.org/wiki/Kublai_Khan]
CSULA
Jongwook Woo

52. HiPIC The political position
 The position of the king was the 7th ranked in the
Mongol empire
 It is the power of the princess
– A daughter of Kublai
 Should know that Kublai Khan has 12 sons.
 Goryeo received many benefits from the empire
– “Only Goryeo in the world kept the king and kingdom”
– When the king went to the palace of the empire, all mongol
officials wanted to give presents.
– The king asked the Khan to suppress Mongol generals in
Goryeo
 The position of the king was the 4th ranked in the
empire
 The next great Khan Temur:
 The princess is his aunt
 The khan asked the king be the 4th ranked at the empire
CSULA
Jongwook Woo

53. HiPIC The Empress Gi (기황후, 奇皇后)
born to Gi Ja-o (奇子敖)
in Haengju (幸州), Gor
yeo
Became a concubine of
Toghun Temür Khan
– Became the first
empress in 1365
Her son Ayurshiridar was
designated Crown Prince
in 1353.
– Supported by Korean
eunuch Bak Bulhwa
(朴不花)
– became a Khan called
Biligtü Khan in 1370.
CSULA
Jongwook Woo

54. HiPIC The Empress Gi (기황후, 奇皇后)
 Good for Goryeo
She prohibited the culture to send Korean women to
the Mongol empire for marriage and slavery
 She eliminated any discussion to make Goryeo
kingdom as one of provinces in the Mongol empire
CSULA
Jongwook Woo

55. HiPIC The Empress Gi (기황후, 奇皇后)
 An elder brother named Gi Cheol (奇轍,
Bayan Bukha).
 Came to threaten the position of the king of Goryeo
 King Gongmin exterminated the Gi family in 1356
CSULA
Jongwook Woo

56. HiPIC The Empress Gi (기황후, 奇皇后)
 The Ming China occupied the capital of the
empire, Dadu (大都, Beijing), in 1368
The empress was disappointed that Goryeo did not
send any reinforcements
Fled north to Shangdu (上都, Xanadu)
CSULA
Jongwook Woo

57. HiPIC Conclusion II
Woman has a power to control husband:
King and Khan (Emperor)
 can promote their social positions to the higher
Woman can make a son to a Khan
Woman possess a political power to
positively affect the motherland
We need to know history and educate kids
CSULA
Jongwook Woo

58. HiPIC Question?
CSULA
Jongwook Woo

59. HiPIC References Part I
1) Introduction to MongoDB, Nosh Petigara, Jan 11, 2011
2) Hadoop Fundamental I, Big Data University
3) “Large Scale Data Analysis with Map/Reduce”, Marin
Dimitrov, Feb 2010
4) “BFS & MapReduce”, Edward J Yoon
http://blog.udanax.org/2009/02/breadth-first-search-
mapreduce.html, Feb 26 2009
5) “Market Basket Analysis Algorithm with no-SQL DB HBase
and Hadoop”,Jongwook Woo, Siddharth Basopia, Yuhang
Xu, Seon Ho Kim, The Third International Conference on
Emerging Databases (EDB 2011), Songdo Park Hotel,
Incheon, Korea, Aug. 25-27, 2011
CSULA
Jongwook Woo

60. HiPIC References
6) “Market Basket Analysis Algorithm with Map/Reduce of
Cloud Computing”, Jongwook Woo and Yuhang Xu, The 2011
international Conference on Parallel and Distributed
Processing Techniques and Applications (PDPTA 2011),Las
Vegas (July 18-21, 2011)
7) Building Realtime Big Data Services at Facebook with
Hadoop and Hbase, Jonathan Gray, Facebook, Nov 11, 2011,
Hadoop World NYC
8) Analyzing Big Data at Twitter, Kevin Well, Web 2.0 Expo, NYC,
Sep 2010
9) Lessons Learned from Migrating 2+ Billion Documents at
Craigslist, Jeremy Zawodny, 2011
10) Machine Learning on Hadoop at Huffington Post | AOL, Thu
Kyaw and Sang Chul Song, Hadoop DC, Oct 4, 2011
CSULA
Jongwook Woo

61. HiPIC References
11) “MapReduce Debates and Schema-Free”, Woohyun Kim,
www.coordguru.com, http://blog.naver.com/wisereign, March
3 2010
12) “Large Scale Data Analysis with Map/Reduce”, Marin
Dimitrov, Feb 2010
13) “HBase Schema Design Case Studies”, Qingyan Liu, July 13
2009
CSULA
Jongwook Woo

62. HiPIC References Part II
1) 고려에 시집온 징기스칸의 딸들, 이한수, Nov 8 2006, 김영사
2) 쿠빌라이 칸의 일본원정과 충렬왕, 이승한, 2009, 푸른역사
CSULA
Jongwook Woo