Mingyu Bao presented on the application of Apache Pulsar in Tencent's billing and advertising systems. Tencent developed their own distributed transaction framework (TDXA) and message queue (TDMQ) based on Pulsar to ensure high transaction consistency and availability across regions. TDMQ supports asynchronous data transmission between systems, exception handling, and high data consistency. Pulsar is also used in Tencent's advertising system to support real-time and offline data processing with low latency, handle massively delayed messages through wheel timers and delayed topics, and currently processes over 100 billion messages per day across 70+ clusters with maximum throughput of 166w+ requests per second.
How to Remove Document Management Hurdles with X-Docs?
Application of Apache Pulsar in Tencent Billing and Tencent Advertising - Pulsar Virtual Summit Europe 2021
1. Pulsar Virtual Summit Europe 2021
Application of Apache pulsar
in Tencent Billing and
Tencent Advertising
Mingyu Bao
Senior Software Engineer,Tencent
2. Pulsar Virtual Summit Europe 2021
Self-Introduction
MingYu Bao.
Senior Software Engineer at Tencent.
Professional in distributed financial systems.
Focused on distributed message queue
system and transaction engine architecture,
providing billing services of high stability for
Tencent's business.
Open-source enthusiast.
Contributed of Apache Pulsar community.
3. Pulsar Virtual Summit Europe 2021
Midas
米大师
Standard Edition
Games,Video,Club etc
30+ Billions of Accounts
Billions of Transactions
Enterprise Editon
WeChat Ads
Cloud Up to 1,000,000 TPS
OverSeas Editon
• 100+ Payment Channels
• 180+ Countries
• Full link Risk Control
Midas platform
Industry Edition
Education,Travel,Traffic etc
5. Pulsar Virtual Summit Europe 2021
High consistency of transaction, ensuring the consistency of payments and goods
delivery
Failure and timeout: A single payment of Midas often involves many internal and external systems. This leads to
longer call chains and more exceptions, especially in network timeouts (e.g. overseas payment services), DB
manipulation, goods delivery Failures.
High data reliability crossing regions
Oceans of timed processing
subscription & recurring billing, in-time account reconciliation
High Performance
High Scalability
automatically scales elastically according to the business scope
Pain Points of Billing platform
6. Pulsar Virtual Summit Europe 2021
Our Solution
Producer
Consumer
http
proxy
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster-1
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster-1
Consumer
Business side Op side
Second meter monitoring
In/Out message tracing
Flow control
High data reliability
Core side
Import/export tool
7. Pulsar Virtual Summit Europe 2021
Our Solution
TDXA+TDMQ
TDXA, a framework solution for transaction control. Ensuring high consistency of transactions, high fault tolerance, high scalability.
TDMQ, based on Pulsar,distributed Message Queue. EnablesTDXA to deal with messaging with high consistency and availability, and makes
convergence of various exceptions with the state table.
8. Pulsar Virtual Summit Europe 2021
Deployment of Tencent billing system
…
TDSQL
…
TDSQL
TDXA TDXA TDXA
App App App
TDMQ TDMQ
Wechat
Pay
Wechat
Pay
bank
Shenzhen Shanghai Hongkong
TDSQL
TDMQ’s Achievements:
1. Asynchronous data
transmission
between sytems
2. System Exception
handling
3. High data
consistency
3
1
2
9. Pulsar Virtual Summit Europe 2021
TDsort
Function
broker
Application of Pulsar in Apache InLong
SDK Agent http
Bus/DataProxy
CKafka Hippo TubeMQ Pulsar
Sort
Real-time/offli
ne calculation
SDK
Aggregate
Cache
Consume
Data access universal kit – Apache InLong
DBsync
Collection
Apache InLong
10. Pulsar Virtual Summit Europe 2021
Pulsar in Tencent Advertising
Advertising pool
Apache InLong
DBsync Pulsar
Sync consumer
Push consumer
Public
BM CRM TCC Etc.
Subscribe
Wide table
Click house
Auth ES/Provider ES
Auth Service
Sync Pub Service
Adv play
Adv sort
Adv Search
Adv index
Real time/offline
Low latency
More topics
Massively delayed messages
11. Pulsar Virtual Summit Europe 2021
Real-time and offline data processing
binlog
TDSQL
…… 5 4 3 2 1
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster for real time consumer
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster for offline consumer
GEO
DBSync
12. Pulsar Virtual Summit Europe 2021
Massively Delayed Messages in Adv
Produce delayed messages to some certain time
forever()
Massively delayed message
Unfixed delay time
1-1 1-3 5-1 1-4 1-5 5-2 1-2 5-3
WheelTi
mer/Sub
-1
consumer
13. Pulsar Virtual Summit Europe 2021
Massively Delayed Messages in Adv
Ledger-3 Ledger-5 Ledger-8
producer
Day index
Hour index
Minute index
WheelTim
er
1
2
3
4
5
6
7
…
consumer
Ledger-9 Ledger-11 Ledger-15
Origin topic
Delay topic
Origin
topic
14. Pulsar Virtual Summit Europe 2021
100 Billion + msg/day
70+ Clusters
166w+ Max Qps
Current status of internal operations
TDXA, a framework solution for transaction control. Ensuring high consistency of transactions, high fault tolerance, high scalability.
TDMQ,based on Pulsar,distributed Message Queue. EnablesTDXA to deal with messaging with high consistency and availability, and makes convergence of various exceptions with the state table