This document summarizes a presentation about Apache Pulsar multi-tenancy and security features at Verizon Media. It discusses how Pulsar implements tenant and namespace isolation through storage quotas, throttling policies, broker and bookie isolation. It also covers authentication, authorization, encryption in transit and at rest, and how Pulsar proxy supports SNI routing for hybrid cloud deployments and cross-organization replication. Future plans include tenant-based broker virtualization and hybrid cloud deployments with geo-replication.

1. Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or
distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Apache Pulsar
Multi-tenancy and Security
June 17, 2021
Rajan Dhabalia rdhabalia@verizonmedia.com
Ludwig Pummer ludwig@verizonmedia.com
1

2. Speakers
2
Rajan Dhabalia
Principal Software Engineer, Verizon Media
Ludwig Pummer
Principal Production Engineer, Verizon Media

3. Agenda
● Pulsar in Yahoo/Verizon Media
● Multi tenancy
● Security
● SNI routing and proxy support
● Future
● QA
3

4. Pulsar journey in Yahoo
● Developed as a hosted pub-sub service within Yahoo/VMG
○ open-sourced in 2016
● Global deployment
○ 6 DC (Asia, Europe, US)
○ full mesh replication
● Mission critical use cases
○ Serving applications
○ Lower latency bus for use by other low latency services
○ Write availability
4

5. ● Pulsar scale and storage evolution talk
https://pulsar-summit.org/en/event/virtual-conference-2020/sessions/pulsar-storage-on-bookkeepe
r-seamless-evolution
● Pulsar growth since 2015
○ 120+ tenants and 15M rps
○ Storage evolution : HDD, SDD, NVMe, PMEM
○ On-prem, public-cloud and cross org integration
● Scale but what about multi-tenancy?
Scale & Multi-Tenancy
5

6. 6
Secured multi-tenant system with Apache Pulsar

7. Multi-tenancy & Security Requirement
7
Multi-tenancy
Tenant and Namespace
IO isolation
Quota and Throttling
Broker and Bookie isolation
Anti-affinity group
Security
Authentication & Authorization
Encryption in transit
Encryption at rest
Pulsar proxy
Support ATS, HAProxy, Nginx

8. Multi-tenancy
8

9. Tenant
● Highest level of provisioning
● Unit of administration
● Managed by Pulsar
administrators
● Usually one team
9
Tenant and Namespace
Namespace
● Middle level of provisioning
● Unit of data policy
● Managed by Pulsar
administrators and/or Tenants
● Usually one application/use
case
persistent://tenant/namespace/topic

10. 1. Portal find User to Team mapping
2. User creates or modifies tenant
○ Tenant name, Admin Authorization Principals
○ Clusters, WPS & RPS Estimates
○ Jira project, Contact Info, Documentation Link
3. Portal reviews capacity & calls Admin API to manage tenant
○ Jira ticket for Pulsar operator if needed
10
Self-Service Tenant Management

11. 11
IO Isolation
Writer Reader
Journal Data File
Data Device
Journal Device
Write Reads (cold)

12. Storage Quota
● Tenant-controlled
● Namespace-level and
Topic-level
● Storage Limit
● Policy
Throttling
● Pulsar Administrator-controlled
● Namespace-level
● Publish Rate (broker)
● Dispatch Rate
● Replicator Dispatch Rate
● Max
○ Producers
○ Subscriptions
○ Consumers
○ Unacked Messages
12
Quota & Throttling

13. Broker Isolation
● Regex of Namespaces to
Regex of Brokers/IP Range
● Primary and Secondary broker
Regexes
13
Broker Isolation
Why
● High Profile/Reserved capacity
● Misbehaving tenants
● Debugging
bin/pulsar-admin ns-isolation-policy set
--auto-failover-policy-type min_available
--auto-failover-policy-params min_limit=5,usage_threshold=80
--namespaces ‘my-tenant/.*’
--primary ‘broker-mytenant[0-9]+.mydomain’ --secondary
‘spare[0-9]+.mydomain’ my-cluster policy-name

14. Bookie Isolation
● Bookies to “Affinity Group”
● Namespace(s) to
Primary/Secondary Affinity
Group
● Rack-Aware within group
14
Bookie Isolation
Why
● SLA
● High Profile/Reserved capacity
bin/pulsar-admin bookies set-bookie-rack -b 1.1.1.1:3181
-g group-bookie1 --hostname bookie1.mydomain -r /default-rack
...
bin/pulsar-admin namespaces set-bookie-affinity-group
my-tenant/my-namespace1 --primary-group group-bookie1

15. ● Common unit of failure
for multiple brokers
15
Failure Domain
bin/pulsar-admin clusters
create-failure-domain
cluster-name
--domain-name domain-1
--broker-list
broker-1,broker-2
Broker-1
Broker-2
Domain-1
Broker-3
Broker-4
Domain-2
Namespace-1
Namespace-2
Namespace-3
Namespace-4
1
2
3
4
Loadbalancer: Namespace
assignment sequence
Anti-affinity-namespaces: “Namespace-X”

16. ● Assign Namespaces to
Anti-Affinity Group
● Changes Load Balancer
Behavior
16
Anti-affinity group
bin/pulsar-admin namespaces
set-anti-affinity-group
tenant/namespace1 --group
tenant-aag-a
bin/pulsar-admin namespaces
set-anti-affinity-group
tenant/namespace2 --group
tenant-aag-a
Broker-1
Broker-2
Domain-1
Broker-3
Broker-4
Domain-2
Namespace-1
Namespace-2
Namespace-3
Namespace-4
1
2
3
4
Loadbalancer: Namespace
assignment sequence
Anti-affinity-namespaces: “Namespace-X”

17. Security
17

18. ● Authentication
○ TLS Authentication
○ Athenz
○ Kerberos
○ JSON Web Token Authentication
○ Pluggable authentication provider
● Authorization
○ Pluggable authorization provider
○ Default authorization provider on metadata service
18
Authentication & Authorization

19. 19
Encryption over the wire
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://pulsar-broler:6651/")
.tlsTrustCertsFilePath("/ca.cert.pem")
.authentication(AUTH, "tlsCertFile:/cert.pem,"+"tlsKeyFile:/key.pem")
.enableTlsHostnameVerification(true)
.build();

20. Producer creation
Producer producer = pulsarClient.newProducer()
.topic(
"persistent://my-tenant/my-ns/my-topic"
)
.addEncryptionKey("myappkey")
.cryptoKeyReader(new MyCryptoKeyReader())
.create();
20
Encryption at rest
Consumer creation
Consumer consumer = pulsarClient.newConsumer()
.topic(
"persistent://my-tenant/my-ns/my-topic"
)
.subscriptionName(
"my-subscriber-name"
)
.cryptoKeyReader(new MyCryptoKeyReader())
.subscribe();

21. ● Proxy for hybrid could application
● Gateway in a cloud environment or on
Kubernetes
21
Pulsar Proxy: Public cloud access
Proxy Configuration
brokerServiceURLTls=pulsar+ssl://brokers.example.com:6651
brokerWebServiceURLTls=https://brokers.example.com:8443

22. ● Proxy server creates a TLS tunnel between remote client and server
● The goal is to enable external clients to connect to internal services and do their
own client certificate verification, possibly because distribution of private keys to
the edge Traffic Server instances is too difficult or too risky.
22
Support Layer-4 SNI Routing

23. 23
Pulsar client: SNI Routing
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://pulsar-broker:6651/")
.enableTls(true).tlsTrustCertsFilePath("/ca.cert.pem")
.proxyServiceUrl(proxyUrl, ProxyProtocol.SNI)
.authentication(AUTH, "tlsCertFile:/cert.pem,"+"tlsKeyFile:/key.pem")
.build();

24. 24
Cross Organization geo-replication
pulsar-admin clusters create orgB-cluster
--broker-url-secure pulsar+ssl:// orgB-broker-vip:6651
--proxy-protocol SNI
--proxy-url pulsar+ssl:// orgA-proxy:443
pulsar-admin clusters create orgA-cluster
--broker-url-secure pulsar+ssl:// orgA-broker-vip:6651
--proxy-protocol SNI
--proxy-url pulsar+ssl:// orgB-proxy:443
For more info: PIP-60:
https://github.com/apache/pulsar/wiki/PIP-60%3A-Support-Proxy-server-with-SNI-routing

25. Future Roadmap
● Tenant based broker virtualization
○ Container based brokers on BookKeeper service
● Hybrid cloud deployment with geo-replication
25

26. Questions?
26

27. Thank you
Rajan Dhabalia rdhabalia@verizonmedia.com
Ludwig Pummer ludwig@verizonmedia.com