Weitere ähnliche Inhalte Ähnlich wie Pulsar for Kafka People (20) Mehr von Jesse Anderson (13) Kürzlich hochgeladen (20) Pulsar for Kafka People1. Pulsar For Kafka People
1 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
3. Apache Pulsar is a distributed
event streaming system
Apache Pulsar
• It uses a distributed log to
durably store messages
• Pulsar was originally created
at Yahoo
• Open sourced in 2016
• Graduated to a top-level
Apache project in 2018
3 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
4. Kafka is a distributed publish
subscribe system
Apache Kafka
• It uses a commit log to track
changes
• Kafka was originally created
at LinkedIn
• Open sourced in 2011
• Graduated to a top-level
Apache project in 2012
4 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
6. Basic Kafka Architecture
Publisher SubscriberKafka
Publisher sends data and
doesn't know about the
subscribers or their status.
Subscriber recieves data from
publisher and never directly
interacts with it.
All interactions go through
Kafka and it handles all
communication.
6 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
7. Basic Pulsar Architecture
Producer ConsumerPulsar
Producers do not directly
interact with the BookKeeper
cluster.
Consumers do not directly
interact with the BookKeeper
cluster.
All Brokers in the Pulsar
cluster are stateless and can
be scaled independently.
BookKeeper All Bookies in the BookKeeper
cluster are stateful and can
be scaled independently.
7 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
8. Kafka Partitions
Producer 0
Data is divided into partitions.
Partitions are both logical
and physical divisions.
Producer 1
Topic
Partition 0 Partition 1 Partition 2
All data is sent and received
on topics. Topics group like
data together.
8 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
9. Pulsar Ledgers
The individual messages that
are produced are stored as
records in the ledger.
Stream
Ledger 1 Ledger 2 Ledger 3
Record 1 Record 2 Record 3 Record 4
Each topic has it's own stream
and all data for a topic
is stored in it.
As more data is added to a
topic, new ledgers are
allocated to store the data.
9 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
10. Kafka Consumers
Producer 0
Consumer recieves data from
all topic partitions and connects
to brokers 0, 1, and 2.
Producer 1
Broker 0 Broker 1 Broker 2
Topic
Partition 0 Partition 1 Partition 2
Consumer (P012)
10 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
11. Pulsar Subscriptions
Producer 0
In failover, all partitions are
consumed by one consumer and
will fail over to hot spare on fail.
Producer 1
Broker 0 Broker 1 Broker 2
Topic
Partition 0 Partition 1 Partition 2
Failover Sub. (P012)
Shared Sub. (P012)
In shared, messages are
sent in a round robin way to
all consumers.
Shared Sub. (P012)
Key Shared (P012)
Key Shared (P012)
Failover Sub. (P)
In key shared, messages with
the same key are consistently
routed to the same consumer.
11 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
13. import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import static org.apache.kafka.clients.producer.ProducerConfig.*;
Properties props = new Properties();
// Configure brokers to connect to
props.put(BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
// Create a producer with the key as a string and value as a string
KafkaProducer<String, String> producer = new KafkaProducer<>(props,
new StringSerializer(), new StringSerializer());
// Create ProducerRecord and send it
String key = "mykey";
String value = "myvalue";
ProducerRecord<String, String> record = new
ProducerRecord<>("hello_topic", key, value);
producer.send(record);
producer.close();
Kafka Producer API
13 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
14. PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://broker1:6650")
.build();
// Create a producer that will send values as strings
// Default is byte[]
Producer<String> producer = client
.newProducer(Schema.STRING)
.topic("hellotopic")
.create();
// Create a new message, send it, and block until it is
// acknowledged
producer.newMessage()
.key("mykey")
.value("myvalue")
.send();
// Create a new message, send it, and don't block until it is
// acknowledged
producer.newMessage()
.key("mykey2")
.value("myvalue2")
.sendAsync();
// Close producer and client
producer.close();
client.close();
Pulsar Producer API
14 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
15. import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import static org.apache.kafka.clients.consumer.ConsumerConfig.*;
@SuppressWarnings("unused")
public class HelloConsumer {
KafkaConsumer<String, String> consumer;
public void createConsumer() {
String topic = "hello_topic";
Properties props = new Properties();
// Configure initial location bootstrap servers
props.put(BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
// Configure consumer group
props.put(GROUP_ID_CONFIG, "group1");
// Create the consumer with the key as a string and value as a string
consumer = new KafkaConsumer<>(props, new StringDeserializer(),
new StringDeserializer());
Kafka Consumer API (1/2)
15 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
16. consumer.subscribe(Arrays.asList(topic));
while (true) {
// Poll for ConsumerRecords for a certain amount of time
ConsumerRecords<String, String> records = consumer.poll(
Duration.ofMillis(100));
// Process the ConsumerRecords, if any, that came back
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
String value = record.value();
// Do something with message
}
}
}
public void close() {
consumer.close();
}
public static void main(String[] args) {
HelloConsumer consumer = new HelloConsumer();
consumer.createConsumer();
consumer.close();
}
Kafka Consumer API (2/2)
16 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
17. client = PulsarClient.builder()
.serviceUrl("pulsar://broker1:6650")
.build();
String myTopic = "hellotopic";
String mySubscriptionName = "my-subscription";
// Create a consumer that will receive values as strings
// Default is byte[]
consumer = client.newConsumer(Schema.STRING)
.topic(myTopic)
.subscriptionName(mySubscriptionName)
.subscribe();
while (true) {
// Block and wait until a single message is available
Message<String> message = consumer.receive();
try {
// Do something with the message
System.out.println("Key is "" + message.getKey()
+ "" value is "" + message.getValue()
+ """);
// Acknowledge the message so that it can be
// deleted by the message broker
consumer.acknowledgeCumulative(message);
} catch (Exception e) {
// Message failed to process, redeliver later
consumer.negativeAcknowledge(message);
}
}
Pulsar Consumer API
17 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
18. Both projects have an ecosystem
associated with them
Ecosystem
Projects
• Kafka Streams -> Pulsar
Functions
• KSQLDB (prop) -> Pulsar SQL
• Kafka Connect -> Pulsar IO
• Kafka API compatibility for
Pulsar
18 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
20. Kafka++
All Kafka use cases plus
more
20 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
23. Unified
Do both MQ-style and
Pub/Sub-style with the
same cluster
23 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9
24. Lots of Topics
Supports millions of topics
24 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9