Data Integration with Embulk Data Science Weekend 2016, Yogyakarta December 3rd, 2016

1. Data Integration
with Embulk
DATA SCIENCE WEEKEND 2016, YOGYAKARTA
TEGUH NUGRAHA

2. Multi Data Formats and
Storages
MySQL
PostgreSQL
MongoDB
CSV files
BigQuery
Redshift
HDFS
Google Analytics
Mixpanel
Zendesk
Elasticsearch

3. Multi Data Sources
Users data in MySQL
Offline data in CSV
Traffics data in Google Analytics
Log data
Bulk Data Loading: Load data from A to B

4. Problems
Parsing files
Error handling
Idempotent Retrying
Performance
Scalability
Format compatibility

5. Solution
Reliable framework with parallel execution, data validation, error recovery,
auto guessing, resuming and extensive plugins
github.com/embulk/embulk

6. Embulk: Bulk Data Loader

7. Plugins by Category
•Input plugins
•Output plugins
•Filter plugins
•File parser plugins
•File decoder plugins
•File formatter plugins
•File encoder plugins
•Executor plugins

8. Getting Started
1. Embulk requires Java
2. Download embulk:
http://dl.embulk.org/embulk-latest.jar
3. Make it executable
$ embulk --version
4. Run an example:
$ embulk example

9. Installing Embulk Plugin
$ embulk gem install embulk-input-mysql
$ embulk gem install embulk-output-postgresql
List of plugins:
https://embulk.org/plugins

10. Embulk Configuration File

11. Embulk Configuration File
(YAML)
in: Input plugin options.
◦ parser: If the input is file-based, parser plugin parses a file format (built-in csv,
json, etc).
◦ decoder: If the input is file-based, decoder plugin decodes compression or
encryption (built-in gzip, bzip2, zip, tar.gz, etc).
out: Output plugin options.
◦ formatter: If the output is file-based, formatter plugin formats a file format (such
as built-in csv, JSON)
◦ encoder: If the output is file-based, encoder plugin encodes compression or
encryption (such as built-in gzip or bzip2)
filters: Filter plugins options (optional).
exec: Executor plugin options. An executor plugin control parallel processing
(such as built-in thread executor, Hadoop MapReduce executor)

12. Using Guess Command
Guess command guesses parser and decoder options
$ embulk guess seed.yml –o config.yml

13. Using guess command

14. Previewing and Running
$ embulk preview config.yml
$ embulk run config.yml
Setup cron schedule

15. embulk-input-mysql
https://github.com/embulk/embulk-input-jdbc/tree/master/embulk-
input-mysql
$ embulk gem install embulk-input-mysql

16. embulk-output-postgresql
https://github.com/embulk/embulk-output-jdbc/tree/master/embulk-
output-postgresql
$ embulk gem install embulk-output-postgresql

17. Using Variables
configuration file name must end with .yml.liquid
Environment variables are set to env variable

18. Include file
File will be searched from the relative path of the input
configuration file and file name will be _<name>.yml.liquid

20. Thank You
TEGUH NUGRAHA
DATA SCIENCE LEAD, BUKALAPAK
TEGUH@BUKALAPAK.COM
HTTP://WWW.SLIDESHARE.NET/TEGUHN

21. References
https://embulk.org
https://github.com/embulk/embulk
http://www.slideshare.net/frsyuki/fighting-against-chaotically-
separated-values-with-embulk