This document discusses Apache Kafka and Kafka Connect. It provides an overview of Kafka Connect and how it can be used for ETL processes. Kafka Connect allows data to be exported from or imported to Kafka and integrated with other systems through customizable connectors. The document describes how to run Kafka Connect in standalone and distributed modes and highlights some popular connectors available for integrating Kafka with other data sources and sinks.

1. Apache Kafka & Kafka Connect
( ETL )
Future Architect ,Inc
Keigo Suda
2016/11/25
D&S Data Night vol.04

2. 12
! Kafka Connect ETL
( )
( ) /

3. 13
! Future Architect ,Inc ( )
!
! IoT

4. 14

5. 15
Kafka Connect

6. 16
! Kafka ver 0.9
! Kafka
!
!
! /
Kafka Connect

7. 17
Connectors
https://www.confluent.io/product/connectors/

8. Kafka Connect

11. 21

12. 22
!
!
!
Kafka Connect ( )

13. 23
ETL Kakfa Connect( )
P T
// / /
AC
KM
I
M
KM

14. Kafka Connect

15. 25http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
Kakfa Connect
Kafka Connect

16. 26
Kakfa Connect
http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started

17. 27
Stream & Partition(RDB )
http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect

18. 28
N:1
Stream & Partition(RDB )
http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect

19. 29
Kakfa Connect
http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started

20. 30
Worker & Connector
http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started

21. 31
CONNECTER
WORKER
STREAM
PARTITION
STANDALONE MODE
DISTRIBUTED MODE
TASK

22. 32
$ bin/connect-standalone.sh config/connect-standalone.properties connector1.properties
(Standalone mode)
!
!
! Worker
! Connector
Worker Connector

23. 33
name=local-file-source
connector.class=FileStreamSource
tasks.max=1
file=test.txt
topic=connect-test
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
...
Worker
Connector

25. 35
!
!
! ETL

26. 36
! Kafka Connect Source Sink
! Sink/Source Sink
Kafka Connect
Source Sink

27. 37
! Connector/Task OK
! API OK
( )
( )

28. 38
Connector( )

29. 39
!!(source pull)
Task

30. 40
! Task Connector
!
Deploy
name=sample-sink
connector.class=SampleSinkConnector
tasks.max=1
topic=connect-test
Connector

32. 42
! Kafka
!
Kafka Connect

33. 43
!
API
Kafka Connect

34. 44
(kafka-connect-jdbc)

35. 45
! Confluent
Confuluent Platform
Kafka Connect

36. 46
! Kakfa ( )
! Confluent….