1. What is “Data Engineering?”
Data Engineering Lab.
Kim Yong Dam
DataPub 12/3
Data Engineering Lab. in Sogang Univ.
2. <Contents>
1. Introduction
2. What is Data Engineering?
3. Role of Data Engineer
4. What I’m doing..?
5. Future Work
DataPub 12/3
Data Engineering Lab. in Sogang Univ.
4. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Introduction
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
5. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Introduction
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Features / value
6. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Introduction
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Features / value
Price / Analysis
7. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Introduction
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Features / value
???? Price / Analysis
8. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Introduction
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Features / value
Optimization Price / Analysis
9. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Introduction
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Features / value
Optimization Price / Analysis
How?
11. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Data Engineering
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Tons of to do..
Tons of to do..
Tons of to do..
Tons of to do..
12. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Tons of to do..
Tons of to do..
Tons of to do..
Tons of to do..
Build Systems
with respect to
each data domain
Data Engineering
13. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
https://blog.hackerrank.com/the-biggest-misconception-about-data-scientists/
Tons of to do..
Tons of to do..
Tons of to do..
Tons of to do..
“On Computer Architecture”
Data Engineering
15. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Role of Data Engineer
https://jobs.apple.com/us/search?job=86260820&openJobId=86260820#&ss=Data%20Engineer&t=0&so=&pN=0&openJobId=99607161
16. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Role of Data Engineer
https://cloud.google.com/certification/data-engineer
17. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Role of Data Engineer
https://cloud.google.com/certification/data-engineer
18. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Role of Data Engineer
https://cloud.google.com/certification/data-engineer
19. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Role of Data Engineer
1. Designing data processing systems
2. Building and maintaining data structures and databases
3. Analyzing data and enabling machine learning
4. Modeling business processes for analysis and optimization
5. Ensuring reliability
6. Visualizing data and advocating policy
7. Designing for security and compliance
20. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Role of Data Engineer
1. Designing data processing systems
2. Building and maintaining data structures and databases
3. Analyzing data and enabling machine learning
4. Modeling business processes for analysis and optimization
5. Ensuring reliability
6. Visualizing data and advocating policy
7. Designing for security and compliance
“Should focus on something”
22. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
My Voyage
1. Designing data processing systems
2. Building and maintaining data structures and databases
3. Analyzing data and enabling machine learning
4. Modeling business processes for analysis and optimization
5. Ensuring reliability
6. Visualizing data and advocating policy
7. Designing for security and compliance
23. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
My Voyage
1. Designing data processing systems
2. Building and maintaining data structures and databases
3. Analyzing data and enabling machine learning
4. Modeling business processes for analysis and optimization
5. Ensuring reliability
6. Visualizing data and advocating policy
7. Designing for security and compliance
For what?
For what?
For what?
24. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
My Voyage
http://www.ibmbigdatahub.com/infographic/four-vs-big-data
25. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
My Voyage
http://www.jobs.ac.uk/enhanced/industry/lifesciences-london/
26. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
My Voyage
http://www.jobs.ac.uk/enhanced/industry/lifesciences-london/
Make a implemented connection
27. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
My Voyage
http://www.jobs.ac.uk/enhanced/industry/lifesciences-london/
As a TEAM!
29. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Future Work
1. Tree Optimization for Spatial data in Non-Volatile Memory
2. Keyword Clustering for SNS data analysis
3. Clustering technique as unsupervised learning
4. Spatial Web Querying using Spatial Database
30. DataPub 12/3
Data Engineering Lab. in Sogang Univ.
Future Work
1. PB+ tree, R-tree for PCM
2. Ontology-based Keyword Clustering, Review on Sematic
Document Clustering
3. An efficient K-Means Algorithm integrated with Jaccard Distance
Measure for Document Clustering, A New Mallows Distance
Based Metric for Comparing Clusterings, Measuring Similarity
between Sets of Overlapping Clusters
4. Efficient Processing of Spatial Group Keyword Queries, Keyword
Search in Spatial Databases: Toward Searching by Document
