Scaling DB Access for Billions of Queries Per Day @PayPal

Scaling DB Access for 100s of billions of
queries per day
1

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
paypal-scale-db/

Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

TLDR - Hera
Open Source
High Efficiency
Reliable Access
to data stores
2
Out Of DB Connections?
• Scaling
• Multiplexing
• Sharding
• Resiliency
• Manageability
• Our switch to Go Lang

About Petrica
• Works @ PayPal
• developing the database proxy.
• database backed publish-subscriber messaging system
• PayPal Here backend
• Golang contributor
• Previously @
• Worked on distributed systems, embedded system, desktop apps
©2019 PayPal Inc. Confidential and proprietary. 3

About Kenneth
• Works @ PayPal
• paypal.de
• PayPal Verisign Two Factor Authentication
• Coded shared memory for sharding
• Migrated to Go (hera)
• Previously @
• Visualization of Genomic Metabolism, Artificial Chemistry, and EPICS
Monitoring

About PayPal – 2018 facts
100+
Currencies
200+
Markets
267M
Active Customer Accounts
9.9B
Payments Transactions
$578B
Total Payments Volume
Up 27% YoY*
Total Payment Volume
2,700
Applications
4,500
Engineers
17,000
Releases

PayPal OLTP Environment scale
2000+
Database Instances
100+ Billion
Calls/day
3000+
Database Nodes
74 PB
Total Storage
300+
Schemas
10 +
Availability Zones
200,000 +
Application Servers
1+ Billion
Calls/hour

8
Key Take-aways
• Massive scale
• Complexity
• Growth

Micro serv ice
connect
request
request
request
disconnect
request
response
Typical Database Access by Application

Micro serv ice
connect
request
request
request
disconnect
request
response
100 ms
1 ms
1 ms
1 ms
103 ms

Micro serv ice
connect
request
request
request
request
response
100 ms
~0 ms
1 ms
1 ms
~3 ms
Po o l
1 ms
Connection is pre-established, cached and re-used for each request
release
lease

From Few to Thousands Microservices
Micro serv ice
Configure the pool cache large
enough for availability

Micro serv ice
Micro serv ice
Add more service when scaling up

Micro serv ice
Micro serv ice
Micro serv ice

Micro serv ice
Micro serv ice
Micro serv ice
Micro serv ice

Micro serv ice
Micro serv ice
Micro serv ice
Micro serv ice
Micro serv ice
Hit a connection limit

17
Solutions?
• Lower the connection pool
size in each microservice
node
• Improve microservice code
to use the connection more
efficiently
• Buy better database
hardware

Partitioned Connection Pool
Micro serv ice
Micro serv ice
Request
Request
While servicing requests, the
microservices use only some
DB connections

Micro serv ice
Micro serv ice
Request
Request
DB connections
In some node all the
connections can be temporary
busy

Micro serv ice
DB connections
In some node all the
connections can be temporary
busy
Client requests will incur
latency though the database
has enough capacity
Micro serv ice
Request
Request

21
Solutions?
Scale with shared pool !!!

S h a r e d c o n n e c t i o n p o o l
Micro serv ice
Micro serv ice
Request
Request
Shared pool

Serv ice
Service request
Service
response
SQL request
SQL request
SQL request
SQL request
DB connection
Database connection is
allocated but mostly idle
Typical service request

Serv ice
Service request
Service
response
SQL request
SQL request
SQL request
SQL request
SQL request
SQL request
SQL request
DB connection
Serving other requests when client seems idle?

Multiplexing
Micro serv ice
Micro serv ice
Request
Request
Shared Po o l
Services perceive that they have a database connection

Hera – Highly Efficient Reliable Access
Micro serv ice
Micro serv ice
Request
Request
Hera
Shared connection pool multiplexer
to data stores

HERA
Microservice-
First
DB proxy
28
Principles and constraints
• Correctness
• Resiliency, -ilities,
performance
• Minimal configuration
• Thin client
• Minimal change to wire
protocol

29
Let’s talk about other features!

Scale
Features
30
• Multiplexing
• Read write split
• Sharding

Read-Write Split
RW
RW RW
RW
Oracle RAC - Real
Application Cluster
Each database is exchanging
lock information

Read-Write Split
RW RW
Oracle RAC - Real
Application Cluster
R
R R
Hera No de
Separate write queries
and direct to one node
Reduces traffic between
DB nodes

Sharding
33
• Client vs Server
• Finding the Shard
• Sample Query Run
• Convert Legacy App
• Move Scuttle Bin’s Data

Sharding: Client vs Server
C++ Client
C++ Lib
Database (shard 0) Database (shard 1) Database (shard 2)

Sharding: Same Logic in Java Client
C++ Client
C++ Lib
Java Client
Java Lib

Sharding: Duplicate Logic in Go Client
C++ Client
C++ Lib
Java Client Go Client
Java Lib Go Lib

Sharding: Copy Logic in Python
C++ Client
C++ Lib
Java Lib Go Lib
Python Client
Python Lib

Sharding: Node.js Client Too
©2019 PayPal Inc. Confidential and proprietary.
38
Node.js ClientC++ Client
C++ Lib
Database (shard 1) Database (shard 2)
Java Lib Go Lib
Python Client
Python Lib Node.js Lib
Database (shard 0)

Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Node.js Client
Sharding: Keep Logic in Server
© 2019 PayPal Inc. Confidential and proprietary. 39
Hera Multiplexer
C++ Client
C++ Lib
Hera Java Lib Hera Go Lib
Python Client
Python Lib Node.js Lib

Sharding: Implementation
1. Shard Key: account_id=2000
…
Shard 0 Shard 1 Shard M
Hash function (shard key value) %K
Scuttle bin to logical shard mapping
Logical to physical shard mapping
Shard Key
DB Shard
0
DB Shard
N
…
…
…

…
Shard Key
DB Shard
0
DB Shard
N
2. Murmur3Hash(2000)%1024=280
…
…
…

…
Shard Key
DB Shard
0
DB Shard
N
2. Murmur3Hash(2000)%1024=280
3. Select shard_id from
hera_shard_map where
scuttle_id = 280;
shard_id
---------
1
…
…
…

…
Shard Key
DB Shard
0
DB Shard
N
2. Murmur3Hash(2000)%1024=280
3. Select shard_id from
hera_shard_map where scuttle_id =
280;
shard_id
---------
1
4. TWO_TASK_1 = LOAN_SH1
TWO_TASK_1=‘tcp(loan-sh1:
3306)/loan’
…
…
…

Sharding: Converting Application Query
- Shard Key: account_id
- select * from loan, appfile where loan.id = ? and appfile.loan_id = loan.id

Sharding Compatible Query
- select * from loan, appfile where loan.id = ? and appfile.loan_id = loan.id
and loan.account_id = ? and appfile.account_id = loan.account_id

Sharding: Hera JDBC SQL Rewrite
- select * from loan, appfile where loan.id = :loan_id and appfile.loan_id =
loan.id and loan.account_id = :account_id and appfile.account_id =
loan.account_id
- Hera can now pick out the shard key name and value

Sharding: Legacy App Conversion
• All queries directed to shard 0
• Logs queries that don’t bind to shard
key
…
Shard 0
Shard Key
DB Shard
0
DB Shard
N
…
…

Sharding: Legacy App Conversion
• Whitelist sends one value to a
specific shard
• Limits risk of failures to 1 value
• Hera uses Shard 1, but same DB
• Fast rollback on errors
• Repeat for larger sets
…
Shard 0
Hash function (shard key value)
%K
Scuttle bin to logical shard
mapping
Shard
Key
DB Shard
0
DB Shard
N
Whitelist
Shard 1
…
…

Sharding: Physical Whitelist
• Validates permissions and data
copy
…
Shard 0
%K
mapping
Shard
Key
DB Shard
0
DB Shard
N
Whitelist
Shard 1
…

Sharding: Logical Move for One Scuttle Bin
• If successful, do a physical data
move next
…
Shard 0
%K
mapping
Shard
Key
DB Shard
0
DB Shard
N
Whitelist
Shard 1
…
…

Sharding: Moving Scuttle Bin
• Start data copy
• Typically, tables are partitioned by
scuttle bin
…
Shard Key
DB Shard
0
DB Shard
N
…
…
…

• Start data copy
scuttle bin
• Block writes
…
Shard Key
DB Shard
0
DB Shard
N
…
…

• Start data copy
scuttle bin
• Block writes
• Data fully copied
…
Shard Key
DB Shard
0
DB Shard
N
…
…
…

• Start data copy
scuttle bin
• Block writes
• Data fully copied
• Update map
• Use scuttle bin in new location
…
Shard Key
DB Shard
0
DB Shard
N
…
…
…

55
Sharding: Take Away
• Queries must bind shard key & value
• Manage risk converting legacy code
• Controlled data redistribution

Resiliency
Principles
56
• Deal with exceptional
cases
▪ Surge in load
▪ Slow SQL requests
▪ Issues with the DB

Resiliency
Principles
57
• Deal with exceptional
cases
▪ Surge in load
▪ Slow SQL requests
▪ Issues with the DB
• Limit or avoid impact
• Self heal

Resiliency
Features
58
• Bouncer
• Surge Queue
• Slow Query Eviction
• Read Replica
• Transparent Application
Failover

Resiliency: Oversubscription - Bouncer
Serv ice 1
Hera No de
Serv ice 2
Serv ice 3
Serv ice 4
Service 4 connection is
tcp accepted and then
closed without SSL.

Resiliency: Oversubscription – Surge Queue
Serv ice 1
Hera No de
Serv ice 2
Serv ice 3
Service 3 can make a
request
It waits--queued

Serv ice 1
Hera No de
Serv ice 2
Serv ice 3
Generally, 1ms later,
Service 3 gets run.

Serv ice 1
Hera No de
Serv ice 2
Serv ice 3
Waiting more than 1s gives
an error to Service 3.

Resiliency: Slow Query Eviction
Serv ice 1
Hera No de
Serv ice 2
Serv ice 3
1s
Slow Query
When Hera is overloaded

Resiliency: Slow Query Eviction
Serv ice 1
Hera No de
Serv ice 2
Serv ice 3
1s
Slow Query
Hundred of queries can run
once slow query is
removed.

Monitoring Usage
Serv ice 1
Hera No de
Serv ice 2
Serv ice 3
Each Hera Node
Free DB
Connections
(acpt)
Time

Resiliency: Eviction – Actual event
Each Hera Node
Free DB
Connections
(acpt)
Time
Happened few years back, when automatic eviction went unnoticed.
Evicting expensive query makes space for many typical queries.

Read Replicas
MUX R1 Pool
Po o l 1
R1 Pool
R1 Pool
MUX R1 Pool
R1 Pool
R2 Pool
Po o l 2
Hera
Application
Read Replica 1
Read Replica 2
Write DB
F a i l o v e r
m o d u l e
B u s i n e s s
l o g i c
R1 & R2: different data replication lag

Transparent Application Failover – Read Replica Retry
MUX
R1 Pool
R1 Pool
R1 Pool
R1 Pool
R1 Pool
R2 Pool
Po o l
Hera
Application
B u s i n e s s
l o g i c
Read Replica 1
Read Replica 2
Write DB
• Build failover on server side
• Also failover when query is slow

Manageability
69
• Maintenance without customer
impact
• Recommend clients recycle
often
• Oracle RAC maintenance

Recycle Connections– Node Going Away
Hera No de
Serv ice 2
Serv ice 3
Hera No de
Hera No de
TCP Load
Balancer

Recycle Connections for Manageability
Hera No de
Serv ice 2
Serv ice 3
Hera No de
Hera No de
TCP Load
Balancer

Recycle Connections– Node Returns
Hera No de
Serv ice 2
Serv ice 3
Hera No de
TCP Load
Balancer
Hera No de

DB Maintenance
RW 2
Oracle RAC - Real
Application Cluster
3
4 1
Hera No de
• Preparing Oracle RAC node 3
maintenance
• DBAs remove node 3 from Oracle
configs
• Insert into hera_maint (inst_id,
status, status_time, module,
machine) values (3, ‘F’, [unix
epoch]+2, [hera pool name], [host]

DB Maintenance
RW 2
Oracle RAC -Real
Application Cluster
3
4 1
Hera No de
• After [unix epoch] +2
Take Aways
• Keep controls near those who need
it
• Avoid being unnecessarily involved

- Written in C++, asynchronous
code
- Scalable
- Efficient
- Complex
Previous version
Transition to Go

code
- Scalable
- Efficient
- Complex
Previous version
Transition to Go
New changes
- New functional requirements
- New non functional requirements
(efficiency)

code
- Scalable
- Efficient
- Complex
Previous version
Transition to Go
New changes
- New functional requirements
- New non functional requirements
(efficiency)
- Go programing is synchronous
- Scale efficiently out of the box
- Does it meet our requirements?
- Is stop-the-world garbage collection time an issue?
How about writing in Go?

Proof-of-
concept
1st Release:
No Sharding
Full feature
with C++
dependencies
Full feature
all Go
Going with Go
Transition to Go

- Running in production for 1 year
- Latency and CPU met the requirements
- No clients change
- Had some issues and learnings
Mission Accomplished

func routine() {
time.Sleep(time.Second * 10)
}
func main() {
routines := 10000
wg.Add(routines)
for i := 0; i < routines; i++ {
go func() {
routine(i)
wg.Done()
}()
}
wg.Wait()
}
Spawn 10000 go-routines sleep for
10 seconds to simulate
processing
Running on my 2 cores test
machine
How many OS threads I expect at
runtime?
Learning Example

func routine() {
time.Sleep(time.Second * 10)
}
func main() {
routines := 10000
wg.Add(routines)
go func() {
routine(i)
wg.Done()
}()
}
wg.Wait()
}
10 seconds to simulate processing
machine
Learning Example
$> ./mytest &
[1] 12345
$> ps –L –C mytest –no-headers | wc –l
6
$>

func routine() {
//time.Sleep(time.Second * 10)
syscall.Select(0, nil, nil, nil,
&syscall.Timeval{Sec: 10})
}
func main() {
routines := 10000
wg.Add(routines)
go func() {
routine(i)
wg.Done()
}()
}
wg.Wait()
}
Learning Example
10 seconds to simulate
processing
machine
Using system call to sleep how
many OS threads I expect at
runtime?

func routine() {
//time.Sleep(time.Second * 10)
syscall.Select(0, nil, nil, nil,
&syscall.Timeval{Sec: 10})
}
func main() {
routines := 10000
wg.Add(routines)
go func() {
routine(i)
wg.Done()
}()
}
wg.Wait()
}
Learning Example
10 seconds to simulate processing
machine
$ ./mytest
runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion
runtime stack:
….

. . . “A Go program creates a new thread only when a goroutine is
ready to run but all the existing threads are blocked in system calls,
cgo calls, or are locked to other goroutines due to use of
runtime.LockOSThread”
from “runtime/debug” func SetMaxThreads()
Learning Example

86
Key Take Away • System calls lock the OS
thread. Ex: os.File read
• When using C/C++ library,
be aware of the systems
calls (mutexes, file reads,
socket reads, sleep)
• Golang locks, socket reads,
sleep are not locking the OS
thread
• In your load tests, verify the
number of OS threads

Open source
github.com/paypal/hera
✓ Code, documentation, examples
✓ Standard clients
• Java: JDBC driver
• Golang: database/sql driver
Google group heradatalink
Internal Mirror
Build
Deploy
PayPal Config

Acknowledgements
Kamlakar
Mukundan
Liana
Sid
Yaping
Shuping
Stephanie
Varun
Qing
Chun
Dwain
Paresh
Srini
Dheeraj
Pradeep
Pramod
Samrat
Akash
Ashwin
and countless others

Next Steps
Need more database connections?
Hera can help scale
• Multiplexing
• Sharding
• Resiliency
• Manageability
Open Source
Google Groups – heradatalink
Our Future
• Scaling Tools
• More Clients

github.com/paypal/hera Google group: heradatalink

Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
paypal-scale-db/

Scaling DB Access for Billions of Queries Per Day @PayPal

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Mehr von C4Media

Mehr von C4Media (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Scaling DB Access for Billions of Queries Per Day @PayPal