What do you do when you've two different technologies on the upstream and the downstream that are both rapidly being adopted industrywide? How do you bridge them scalably and robustly? At Wework, the upstream data was being brokered by Kafka and the downstream consumers were highly scalable gRPC services. While Kafka was capable of efficiently channeling incoming events in near real-time from a variety of sensors that were used in select Wework spaces, the downstream gRPC services that were user-facing were exceptionally good at serving requests in a concurrent and robust manner. This was a formidable combination, if only there was a way to effectively bridge these two in an optimized way. Luckily, sink Connectors came to the rescue. However, there weren't any for gRPC sinks! So we wrote one.
In this talk, we will briefly focus on the advantages of using Connectors, creating new Connectors, and specifically spend time on gRPC sink Connector and its impact on Wework's data pipeline.
5. Architecture (continued)
❏ Importance of the architecture
❏ Kafka-gRPC combination
❏ Java/Scala-Go
❏ Why gRPC?
❏ How the combination worked perfectly?
6. Architecture (continued)
Machine readable API contract, so
platform neutral
Throughput
Protobuf, so polyglot and faster,
smaller and simpler, safer payload
Distributed, HA
HTTP/2 - multiplexing, binary, safer,
future
Scalability
Streaming vs request/response Complementary components -
KStream, KTable, KSQL, Connectors
Community Community
gRPC Kafka
7. Life Without Connect
In the very beginning, there was no Connect
Kafka sources were connected to a variety of sinks at Wework- gRPC,
Elasticsearch etc. And hey, it worked well.
Yes, but what about
❏ Scalability?
❏ Security?
❏ Configurability?
❏ Error handling?
❏ Extendability?
Do not reinvent the wheel!
9. Then there was Connect...
Very simply, Kafka Connect is a framework to stream data into and out of Kafka
Properties:
❏ Broad copying, scalability, streaming and batch apps, parallelism.
❏ Does one and one thing very well - copying data
❏ Extensible through Connectors
Models:
❏ Connector
❏ Worker
❏ Data
10. Connect components
❏ Connectors – abstraction that handles data streaming by managing “tasks”
❏ Tasks – the implementation of how data is copied to or from Kafka
❏ Workers – the running processes that execute connectors and tasks
❏ Converters – translates data between Connect and the source/sink
❏ Transforms – alters messages produced by or sent to a connector
❏ Dead Letter Queue – error handling
11. Types of Connectors
❏ Source Connector - Kinesis, Zendesk, Jira, Twitter, email
❏ Sink Connector - S3, MongoDB, HDFS, YouNameIt
❏ Connect Hub - https://www.confluent.io/hub/
❏ Also available are different Transform and Converter
❏ Availability in Confluent Cloud - fully managed
❏ Licenses, levels of verification
13. Writing your own Connectors. Yes, we can!
❏ Why?
❏ Oh, we don’t have that Connector!
❏ We have a Connector, but we need to customize it to our needs
❏ Complete control on how we want to move the data
❏ Give it back to the community
❏ How?
❏ Kafka Connect API to the rescue!
❏ Implement/extend your Connector, Task, Config interfaces/abstract classes
18. GrpcClient Interface
Crux of bulkSend() AKA how is gRPC Connector different from MySql Connector?
gRPC glossary
❏ stub: generated when protoc is run if a service declaration is in the proto file
❏ service class
❏ rpc name and args
❏ channel: provides a connection to a gRPC server on a specified host and port
❏ along with grpc server url and port
19. bulkSend()
bulkSend() forms the crux of the gRPC Sink Connector’s data copying
❏ Handles channel readiness (connectivity state)
❏ Manages security and all logic related to error-handling
❏ Controls the rate of data copying
❏ Potentially retry logic
22. Testing, Deployment, and Monitoring
❏ Testing Connector by mocking is hard and tricky as it involves two systems -
Kafka and the external source/sink
❏ Independent unit-testing is recommended for all Task and Connector classes
❏ End-to-end testing using gRPC servers created on the fly in Docker
containers of CircleCI testing plan
❏ Extremely difficult to test gRPC channel connectivity states and other error
scenarios
23. Testing, Deployment, and Monitoring (contd…)
❏ Packaging - for easy installing into into Kafka Connect installations
❏ By creating an Archive
❏ create a tarball or ZIP archive
❏ contains a single directory with unique name (name and version likely)
❏ all JAR files and other resource files needed by the connector are in tld
❏ doesn’t include Kafka Connect API or runtime libraries
❏ By creating an Uber JAR
❏ create an uber JAR that contains all JAR files and other resource files
❏ Installation -
❏ User simply unpacks the archive or places the uber JAR in a directory listed in Kafka
Connect’s plugin path
24. Monitoring Connectors
❏ Monitored via Connect’s extensive REST interface
❏ current status of a connector and its tasks
❏ worker ids to whom tasks are assigned
❏ pause/resume APIs
❏ active connectors, connector tasks, restart a connector, restart a task, update config, delete
connector
❏ Logging
❏ Connect comes with default Java-based logging utility Apache Log4j to collect runtime data
and record component events
25. Connect metrics and metrics using Prometheus
❏ Reports a variety of metrics through Java Management Extensions (JMX)
❏ task and worker metrics - status, running-ratio,
offset-commit-success-percentage, offset-commit-avg-time-ms, task-count,
connector-count, rebalancing metrics
❏ A variety of client metrics like connection-count, connection-close-rate,
network-io-rate, outgoing-byte-rate, request-rate etc
❏ gRPC Sink Connector metrics:
❏ sink-record-read-rate
❏ sink-record-active-count
❏ sink-record-read-total
❏ sink-record-send-rate
26. Connect metrics and metrics using Prometheus (cntd..)
❏ The monitoring tool Prometheus ingests metrics, makes them graphable, and
helps build alerts on top of metrics
❏ pulls metrics from HTTP endpoints added to the Prometheus configuration file
❏ provides JMX Exporter, a collector that can configurably scrape and expose
mBeans of a JMX target
27. Challenges and Lessons Learned
❏ Configuring more than one rpcs in a service
❏ Configuring rpcs with multiple arguments
❏ Testing the two components of the system
❏ Connectors have the capability to be extremely flexible, and can also hide
intricate logic when used off the shelf