"Kafka Connectors are ready-to-use components useful for importing and exporting data between Kafka topics and external systems such as databases or file systems. Along with the connectors, there are ready-made and easy-to-use Single Message Transformations (SMT) for making modifications to message values and keys. SMTs are relatively easy to build but testing them is another story. We faced several challenges in how to do a proper test because we couldn't faithfully replicate what could happen inside Kafka Connect when using Unit Tests. To make matters worse, we often got data in different formats or patterns than we expected, resulting in several errors while the SMT was being used in different environments. Join us for a lightning session where we'll show you how to do proper integration testing for custom SMTs, using Testcontainers as the best, most accurate, and solid way to test these complex Kafka Connect components. In the end, we get an SMT that is more robust, cleaner, easier to debug and, most importantly, ensure it does what we expected when running it inside our Connect clusters."

1. Testing SMTs?
Testcontainers to the rescue!
Fábio Sequeira | Mafalda Santos
Marionete

2. 2
© 2023 Marionete Limited
• Kafka Connect is a tool for scalability and reliable data transmission between Apache Kafka and other data systems.
• Kafka Connectors are ready-to-use components useful for importing and exporting data between Kafka topics and external
systems (e.g. databases).
Testing SMTs
Kafka Connect & Connectors
Source Connector Sink Connector
Data Source Data Sink
Kafka

3. 3
© 2023 Marionete Limited
• Single Message Transforms (SMTs) are used to transform message values and keys.
• Just like the connectors, there are ready-made and easy-to-use SMTs available.
Testing SMTs
SMTs
Source Connector Sink Connector
Data Source Data Sink
Kafka
SMT SMT

4. 4
© 2023 Marionete Limited
There are plenty of SMTs available but…
Testing SMTs

5. 5
© 2023 Marionete Limited
There are plenty of SMTs available but…
Testing SMTs
Sometimes they are not enough.

6. 6
© 2023 Marionete Limited
We need to build
CUSTOM SMTs!
Testing SMTs

7. 7
© 2023 Marionete Limited
SMTs are relatively easy to build…
Testing SMTs

8. 8
© 2023 Marionete Limited
SMTs are relatively easy to build…
Testing SMTs

9. 9
© 2023 Marionete Limited
Testing SMTs
Unit Tests Limitations
Data Source
Kafka Cluster
Kafka Connect
Connector
Instance
SMTs
Converter
Connect
Record

10. 10
© 2023 Marionete Limited
Testing SMTs
Unit Tests Limitations
Data Source
Kafka Cluster
Kafka Connect
Connector
Instance
SMTs
Converter
Connect
Record
Unit Tests
?
?
?

11. 11
© 2023 Marionete Limited
Testing SMTs
Unit Tests Limitations
• Hard to understand how the Connector reads the value types coming from a DB:
• Often got data in different formats or patterns than we expected
• Different connectors could “read” data differently

12. 12
© 2023 Marionete Limited
Testing SMTs
Unit Tests Limitations
• Hard to understand how the Connector reads the value types coming from a DB:
• Often got data in different formats or patterns than we expected
• Different connectors could “read” data differently
• Certain DB-specific types are very difficult to define in unit tests:
• SQL Server types: Datetime2? Datetimeoffset?

13. 13
© 2023 Marionete Limited
Testing SMTs
Unit Tests Limitations
• Hard to understand how the Connector reads the value types coming from a DB:
• Often got data in different formats or patterns than we expected
• Different connectors could “read” data differently
• Certain DB-specific types are very difficult to define in unit tests:
• SQL Server types: Datetime2? Datetimeoffset?
• Difficult to map/identify specific topic schema fields:
• Protobuf schemas: optional/oneof fields

14. 14
© 2023 Marionete Limited
How to complement the SMTs tests with a more robust and effective test?
Testing SMTs

15. 15
© 2023 Marionete Limited
How to complement the SMTs tests with a more robust and effective test?
Testing SMTs

16. © 2023 Marionete Limited
Testcontainers
to the rescue!

17. 17
© 2023 Marionete Limited
Testcontainers to the rescue!
Testcontainers
• Set up, configure, and run Docker
containers
• Simplify integration testing
• Available in multiple languages
• (We used Java)

18. 18
© 2023 Marionete Limited
Testcontainers to the rescue!
Containers for SMT Testing
• The Testcontainers library has a module for Kafka
• Example:
KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.3.2")
)
• There are also modules for various types of databases.
• For our use cases, we used a Container object based on the Testcontainers MS SQL Server module.
• But we needed more than that…

19. 19
© 2023 Marionete Limited
Testcontainers to the rescue!
Custom Testcontainer Library
KafkaContainer SchemaRegistryContainer ConnectContainer MsSqlServerContainer

20. 20
© 2023 Marionete Limited
Testcontainers to the rescue!
SchemaRegistryContainer
public final class SchemaRegistryContainer extends GenericContainer<SchemaRegistryContainer> {
public SchemaRegistryContainer(DockerImageName image) {
super(image);
this.addExposedPorts(8081);
this.withEnv("SCHEMA_REGISTRY_HOST_NAME", this.getHost());
}
public SchemaRegistryContainer withKafka(KafkaContainer kafkaContainer) {
this.withNetwork(kafkaContainer.getNetwork());
this.withEnv(
"SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS",
"PLAINTEXT://" + kafkaContainer.getNetworkAliases().get(0) + ":9092"
);
return this;
}
}

21. 21
© 2023 Marionete Limited
Testcontainers to the rescue!
SchemaRegistryContainer (cont)
public SchemaRegistryContainer setupSchemaRegistryContainer(
KafkaContainer kafkaContainer,
String alias,
String confluentVersion
) {
return new SchemaRegistryContainer(
DockerImageName
.parse("confluentinc/cp-schema-registry:" + confluentVersion))
.withNetworkAliases(alias)
.withKafka(kafkaContainer);
}

22. 22
© 2023 Marionete Limited
Testcontainers to the rescue!
ConnectContainer
public final class ConnectContainer extends GenericContainer<ConnectContainer> {
public ConnectContainer(ImageFromDockerfile image) {
super(image);
this.addExposedPorts(8083);
this.withEnv("CONNECT_GROUP_ID", "testcontainer-connect-group");
this.withEnv("CONNECT_CONFIG_STORAGE_TOPIC", "connect-config");
this.withEnv("CONNECT_OFFSET_STORAGE_TOPIC", "connect-offsets");
this.withEnv("CONNECT_STATUS_STORAGE_TOPIC", "connect-status");
this.withEnv("CONNECT_REST_ADVERTISED_HOST_NAME", this.getHost());
this.withEnv(
"CONNECT_PLUGIN_PATH",
"/usr/share/java, /usr/share/confluent-hub-components/"
);
// ...
}
public ConnectContainer withKafka(KafkaContainer kafka, SchemaRegistryContainer registry) {
this.withNetwork(kafka.getNetwork());
this.withEnv(
"CONNECT_BOOTSTRAP_SERVERS",
"PLAINTEXT://" + kafka.getNetworkAliases().get(0) + ":9092”
);
this.dependsOn(kafka, registry);
return this;
}
}

23. 23
© 2023 Marionete Limited
Testcontainers to the rescue!
ConnectContainer (cont)
public ConnectContainer setupConnectContainer(
KafkaContainer kafkaContainer,
SchemaRegistryContainer registryContainer,
String alias,
String confluentVersion
) {
return new ConnectContainer(
new ImageFromDockerfile().withDockerfileFromBuilder(
dockerfileBuilder -> dockerfileBuilder
.from("confluentinc/cp-kafka-connect:" + confluentVersion)
.run("/bin/bash", "-c", "confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.6.0")
.build()
)
)
.withKafka(kafkaContainer, registryContainer)
.withNetworkAliases(alias);
}

24. 24
© 2023 Marionete Limited
Testcontainers to the rescue!
Custom Library Example Helper Methods
• createTopic() • registerSchema() • installSMT()
• registerConnector()
• runSQLFile()
KafkaContainer SchemaRegistryContainer ConnectContainer MsSqlServerContainer

25. 25
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer

26. 26
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Custom SMT
Generate jar

27. 27
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Custom SMT

28. 28
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
Custom SMT

29. 29
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
output-topic
Custom SMT

30. 30
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
output-topic
output-topic-value
Custom SMT

31. 31
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
output-topic
output-topic-value
Consumer
Custom SMT

32. 32
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
output-topic
output-topic-value
Consumer
test-source-connector
Custom SMT

33. 33
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
output-topic
output-topic-value
Consumer
test-source-connector
Custom SMT
{
"name": "test-source-connector",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"tasks.max": "1",
"key.converter":
"org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schema:8081",
"value.converter.auto.register.schemas": "false",
"topic.prefix": "output-topic",
"connection.url":
"jdbc:sqlserver://mssql:1433;databaseName=TestDB;(...)",
"table.whitelist": "Input_Table",
// ...
"transforms": "myCustomSMT",
"transforms.myCustomSMT.type": "org.example.MyCustomSMT$Value",
"transforms.myCustomSMT.targetFields": "field1,field2"
}
}

34. 34
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
output-topic
output-topic-value
Consumer
test-source-connector
Input records
Custom SMT

35. 35
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Source Connectors
KafkaContainer SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer
MsSqlServerContainer
Input table
output-topic
output-topic-value
Consumer
test-source-connector
Input records
Compare retrieved records
with expected records
Custom SMT

36. 36
© 2023 Marionete Limited
Testcontainers to the rescue!
Test Setup – Sink Connectors
KafkaContainer
SchemaRegistryContainer
Installed plugins:
• JDBC connector
• Custom SMT
ConnectContainer MsSqlServerContainer
Output table
input-topic
input-topic-value
Producer
test-sink-connector
Compare retrieved records
with expected records

37. 37
© 2023 Marionete Limited
Thank you!

38. © 2023 Marionete Limited
Contacts
www.marionete.co.uk
FÁBIO SEQUEIRA
fabiosequeira
MAFALDA SANTOS
mafaldajsantos
solutions@marionete.co.uk
@marionete_io
https://www.linkedin.com/company/marionete
Core Technology
Specialist
Core Technology
Specialist