Slides for CXCIV Moscow ACM SIGMOD Chapter workshop

1. DATABASE MANAGEMENT
SYSTEMS BENCHMARKING
overview
and
trends

2. ГОНКИ TPC-C
296 results
Simfoware, Oracle DB 7–8, DB2/400 и UDB, Informix, MS SQL Server, Sybase ASE …

3. TPC-C, APRIL, 2017
SQL
Anywhere

4. AGENDA
What happened with TPC-C?
•TPC: early history
•TPC-A/B, -С, -H, -E
•TPC: obsolete and new
•Publication issues
The New Wave of benchmarks
•MapReduce (Hadoop)
•Graph benchmarks
•Meta-universal, atomic
«Pocket tools» to run
Input/Output
Outcomes

5. TP1
Mid 1970s,
IBM benchmark
Bank
transactions
processing
Idefix: 100 tps
(1973, “bank
with 1.000
branches and
10.000 tellers”)
Batch mode
without
networking
no delay for
teller response
Early 1980s:
fantastic
victorious reports
with 10 ktps
$$bln market…
… focus on fast-
growing OLTP
…however,
customers could
not achieved yet
1 ktps …

6. DAVIT DEVITT:
WISCONSIN BENCHMARK
Alternative, more
strictly determined
benchmark
Aimed to terminate
“benchmarking
wars”, but lit the
war even stronger!
Weak results of
Oracle DB caused
‘DeWitt clause’
DBMS license
prohibits publishing
benchmarks

7. JIM GREY: DEBITCREDIT
Counterpoise for Wisconsin benchmark,
based on TP1 subject area
Requires total systems cost, including
equipment, licenses, 5 years of support
Specified as textual functional
requirements, without code requirements
or code examples
Introduced scaling rules by users and table
sizes
Postulated a response time limit : 95 % of
transactions should complete in 1s

8. PRINCIPAL: RESPONSE TIME
BOUND
Изображение: ©Dell, 2013

9. TPC.ORG
Benchmarking wars continued: how to validated result?
Independent, non-profit organization need…
1988: Transaction Processing Performance Council
Omri Serlin and 8 [consonant] vendors
Actian Cisco Cloudera Dell DataCore Fujitsu HPE Hitachi Huawei IBM
Inspur Intel Lenovo Microsoft Oracle Pivotal Red Hat SAP Teradata VMWare
Members (2017):

10. TPC-A AND TPC-B: PROCESS
READ 100 bytes from TTY (AID, TID, BID, DELTA)
BEGIN TRANSACTION
UPDATE ACCOUNT WHERE ACCOUNT_ID = AID:
READ ACCOUNT_BALANCE FROM ACCOUNT
SET ACCOUNT_BALANCE = ACCOUNT_BALANCE + DELTA
WRITE ACCOUNT_BALANCE TO ACCOUNT
WRITE TO HISTORY:
AID, TID, BID, DELTA, TIME_STAMP
UPDATE TELLER WHERE TELLER_ID = TID:
SET TELLER_BALANCE = TELLER_BALANCE + DELTA
WRITE TELLER_BALANCE TO TELLER
UPDATE BRANCH WHERE BRANCH_ID = BID:
SET BRANCH_BALANCE = BRANCH_BALANCE + DELTA
WRITE BRANCH_BALANCE TO BRANCH
COMMIT TRANSACTION
WRITE 200 bytes to TTY (AID, TID, BID, DELTA)
Based on TP1 – retail banking transactions

11. TPC-A AND TPC-B: MODEL
BRANCH
B
ACCOUNT
B*100K
100K
HISTORY
B*2.6M
TELLER
B*1010
10s cycle for each
terminal
1 transaction per
second for each
branch
Response for 90%
of transactions
not less than 2s
Average
transactions per
second in 15 min

12. TPC-A AND TPC-B:
DIFFERENCES
TPC-A
Terminals
User response
delay
TPC-B
Server-only
benchmark
Reduced history
(30 days)

13. TPC-A AND TPC-B: CRITICISM
1995: considered unreliable
Last result (DEC, 1994)
3700 tpsA 4800 $/tpsA
First result (HP, 1990)
38.2 tpsA 29200 $/tpsA
TP1 legacy
Too simple to avoid “adjustments” Implausible dispersion of results

14. TPC-C: COMPLICATION
Response thresholds for 90% of transactions
Less than 5s for interactive operation Less than 20s for batch
More variances…
9 tables
Inserts, updates, deletes,
operation cancellations
Access by primary and
secondary keys
5 типов транзакций
NEW-ORDER
•new customer
order
PAYMENT
•payment event
•customer balance
refresh
DELIVERY
•delivery orer
•(batch)
ORDER-STATUS
•checking the status
of the last customer
order
STOCK-LEVEL
•checking the stock
level at the
warehouse
45% 43% 4% 4% 4%

15. TPC-C DATA MODEL
WAREHOUSE
W
DISTRICT
W*10
10
CUSTOMER
W*30K
3K
HISTORY
W*30K+
1+
ITEM
100K (fix)
STOCK
W*100K100K W
ORDER
W*30K+1+
ORDER-LINE
W*300K+
10–15
NEW-ORDER
W*5K0–1

16. TPC-C: SCALING
Each new warehouse →
+10 districts, + 100 items, +300k customers
Maximum 1.2 tpmC per terminal
10 terminals per warehouse
Scaling factor: warehouse (W)

17. TPC-C:
CLUSTERED AND NON-CLUSTERED
Clustered >1 nodes
Shared
nothing (fully
federated)
Sharded
Shared storage
(RAC)
Non-clustered 1 node

18. TPC-C: METRICS
tpmC
Transactions per
minute
tpmC / $
Transaction cost
Hardware
acquisition cost
Software and
services cost for
3 years
W / ktpmC
Energy
consumption per
1000 transactions
per minute

19. TPC-C INTERPRETATION
By Alan Parker (Alan Parker. Tuning databases on Solaris platform. Prentice-Hall, 2002)
•…not only order entrytpmC × 2
•…if transaction monitor does not usedtpmC / 2
•…if Oracle Forms-like client usedtpmC / 3
•…if lightweight client forms used (like curses-based)tpmC × 2 / 3
•…if SQL has not been tunedtpmC / 2
• …if there are heavy reports and batchestpmC / 2

20. COULD WE CHEAT TPC-C?
mount -t tmpfs -o size=2048g tmpfs /u01/tablespaces
CREATE UNLOGGED TABLE …
_ALLOW_RESETLOGS_CORRUPTION = TRUE
_IN_MEMORY_UNDO = TRUE
_DB_BLOCK_HASH_LATCHES = 32768
…

21. RESPONSE FROM COUNCIL
A l l A C I D a s p e c t s c h e c k s s t i f f l y i n t e r p r e t e d i n s p e c i f i c a t i o n
Single node reboot failure
Synchronous commit on
two nodes
Redo logs and recovering Reboot and recovering
Single media failure protection
Synchronous commit on
two separately powered media
Write-ahead logs on separately powered media
Commit = written on durable media

22. TPC-C: CRITICISM OF 1990S
Even wholesale providers works somewhat differently!
Order: some
unsuccessful wildcard
searches
Printing report after
each entry
(probably, 3 times)
Balance refresh for each
operation is impossible in
a heavy load environment
→ insert + batch calc
Trivial logic
No declarative constraints No trigger logic
Non-typical loads
What about reporting? Decision support systems?

23. TPC: OBSOLETED
TPC-D 1995–1999
First attempt
for OLAP
benchmark
TPC-R 2001–2005 Reporting
TPC-W 2001–2005
Online web-
commerce

24. TPC-H
1999 :
OLAP vs OLTP
“Ad-hoc
decision
support”
Instead of
TPC-D
(considered
irrelevant)
“Weight
categories”
100 GB
300 GB
1 TB
3 TB
10 TB
30 TB
100 TB
Parallel load
22 kinds of
complex
queries
2 kinds of
DWH
refresh

25. TPC-H: MODEL
1 SF ~ 1 GB
Declarative
constraints
Not star schema

26. TPC-H Q1
SELECT
l_returnflag, l_linestatus,
sum(l_quantity) as sum_qty,
sum(l_extendedprice) as sum_base_price,
sum(l_extendedprice*(1-l_discount)) as sum_disc_price,
sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge,
avg(l_quantity) as avg_qty,
avg(l_extendedprice) as avg_price,
avg(l_discount) as avg_disc,
count(*) as count_order
FROM lineitem
WHERE l_shipdate <= date '1998-12-01' - interval Δ day (3)
GROUP BY l_returnflag, l_linestatus
ORDER BY l_returnflag, l_linestatus
Δ = random [60…120]
“ F u n c t i o n a l Q u e r y D e f i n i t i o n ”

27. TPC-H: RESULTS, 2017
Tricky to apply
for non-relational DBMS
Variants to
apply for
MOLAP
(MDX)
Reports
with
application
for
Apache Hive
Exasol
MS SQL
Server
Oracle
Database
Actian
Vector
1 – 10 results for each
“weight category”

28. TPC-E: “PRAGMATIC OLTP”
Still OLTP, but more “hybrid”
Declarative constraints
No transaction monitors
More reads
More kinds of load

29. TPC-E:
MORE COMPLEX WORKFLOW

30. TPC-E: MODEL
Image: ©Transaction Processing Council, 2009

31. 78 results
All non-clustered, all on MS SQL Server for Windows Server x64

32. 11 KTPSE, $144/TPSE (1,6 MIO)
Изображение: ©Lenovo, © TPC, 2015

33. TPC-ABCEH

34. BRAND NEW: TPC-X И TPCX-X
TPC-DI
ETL workloads
TPC_DI_RPS
0 results
TPC-DS
Decision support
systems “on Big
Data solutions,
such as RDBMS as
well as
Hadoop/Spark
based systems”
QphDS@Size
0 results
TPCx-BB
Express benchmark
for “Hadoop-
based Big Data
Systems” adopted
from BigBench
BBpm@Size
0 results
TPCx-HS
Express benchmark
for HDFS-
compatible
systems, adopted
from TeraSort
HSph@Size
From 1 to 4 results
in different
“weight
categories”

35. PUBLISHING?
Publishing on TPC.org
Academic or research paper, could not be applicable for
marketing purposes
With definite clause that outcome is not comparable with
TPC.org results
With permission of TPC.org
Publishing TPC benchmarks is prohibited,
except the following cases:
Required audit from sizing.com

36. DEWITT CLAUSE IN 2017?
MSFT
EULA
•“You may not disclose the results of any benchmark test
… without Microsoft’s prior written approval”
OTN
Lic.
•“You may not disclose results of any Program benchmark
tests without Oracle’s prior consent”
IBM
IPLA
•“Licensee may disclose the results of any benchmark test of
the Program or its subcomponents to any third party
provided that Licensee, if … A) … B) … C)…”

37. NEW BENCHMARKS FOR
NEW WORKLOADS
2010s
2000s
1990s
1980s
OLTP
ROLAP ROLAP
In-memory
OLAP
TCP-DS
ROLAP TCP-H
Graph
queries
LinkBench
MapReduce HiBench
OLTP
OLTP[E] OLTP[E] TPC-E
OLTP
Atomic
access
YCSB
TPCx-HS
TPCx-BB

38. PRINCIPAL BENCHMARKS
OF “NEW WAVE”
Terrasort Benchmark
BigBench
Intel HiBench
Yahoo! Cloud Services
Benchmark
Linkbench
• Adopted in TPCx-HS
• Adopted in TPCx-BB
• Map-Reduce series
• Group of NoSQL
etalon runs
• Graph-like workload on
RDBMS and HBase

39. INTEL HIBENCH ДЛЯ
HADOOP
Image:Intel,2013

40. YAHOO! CLOUD SERVICES
BENCHMARK
New tool for “benchmark marketing”
Used by researchers for wide comparisons
(V. Abramova et al. Experimental Evaluation of NoSQL Databases // IJDMS Vol.6, №3, 2014)
Cassandra HBase
Elastic
search
MongoDB
Oracle
NoSQL
OrientDB Redis Scalaris Tarantool Voldemort
Bombardment from one load station
Atomic operation instead of transaction
(probably, read a few records)

41. YCSB: WORKLOADS

42. LINKBENCH
Facebook workload
•Early graph benchmarks were
graph-analysis oriented
•Real workload of the Internet
Giant
•Transactions (MVCC)
Методологично
•Statistic laws for data
generageion
•Measures: average response time
for each type of workload
•Avg from 99-percentile
•Documents in seconds for write
•Queries per second for read
MySQL
(InnoDB vs
TokuDB)
HBase
MongoDB /
TokuMX
OrientDB
T h e Q u e s t i o n o f Fa c e b o o k

43. POCKET TOOLS
TO MEASURE
Possible?
Correct?
Representative?
Comparable?
Repeatable?

44. PGBENCH
TPC-B
Standard
part of
PostgreSQL
distributions
One simple
command to
run
Standard de facto for PostgreSQL
internal investigations, such as:
XFS or ext4?
Tables on SSD, indexes
on HDD?
Block size: 4K or 8K?

45. SYSBENCH
Widely used for internal comparisons in MySQL, MariaDB,
and its forks communities
Sui generis tests, not similar to standard benchmarks
MySQL utility (by Alexei Kopytov)
fileio cpu memory threads mutex oltp

46. TPCE-MYSQL
TPC-E loader for MySQL by Percona

47. HAMMERDB
Free Java GUI program for
TPC-C (?) and TPC-H (?) with support a range of DBMSs
Oracle
Database
Microsoft
SQL Server
IBM DB2 TimesTen MySQL
MariaDB
Postgre
SQL
Postgres
Plus AS
Greenplum
DB
Redis
Amazon
Aurora
(MySQL)
Amazon
Redshift
(ParAccel)
Trafodion
SQL on
Hadoop

48. HAMMERDB:
ONE-CLICK TPC-C?

49. HAMMERDB:
TPC, BUT NOT QUITE…
Modern benchmarks were not been implemented
TPC-E TPC-DS
TPC-H
Single load station
TPC-C
Not emulated full environment, no
transaction monitor software
Load station in role of single
transaction monitor

50. HAMMERDB: POPULARITY
Used by
hardware and
software
vendors
Indirectly,
in form of a
“blog of our
tech guy”
Hub with
benchmark
results
Section
«Performance
Data»

51. SWINGBENCH
TPC-C-like workload for Oracle Database and TimesTen
+ Idiosyncratic OLTP benchmark with a lot of PL/SQL
Oracle DB-specific tools for monitoring and analysis (AWR, etc.)
Coordinator support
GUI and command line
Only internal comparability
Non-representative in RAC environments

52. DELL BENCHMARK FACTORY
FOR DATABASES
Oracle
Database
MySQL
MS SQL
Server
SQLite
SQL
Anywhere
Commercial tool (Quest Software legacy) TPC-C
TPC-D
TPC-E
TPC-H
ASP3AP
Supports a number of load stations
(Windows)

53. OSDLDBT.SOURCEFORGE.NET
•TPC-WDBT-1
•TPC-CDBT-2
•TPC-HDBT-3
•TPC-AppDBT-4
•TPC-EDBT-5
While the inspiration for these workloads
are the TPC-<x…>, workloads are entirely
different and results obtained from them
should not and can not be compared to
TPC results.
The use of any supplied results of these
tests for commercial purposes is expressly
prohibited.
MySQL PostgreSQL
…расширяемо

54. OLTPBENCH
github.com/oltpbenchmark/oltpbench
Java tool for command line
Supports any JDBC-enabled RDBMS
Special version for Hstore (VoltDB )
TPC-C Wikipedia
Synthetic
Resource
Stresser
Twitter Epinions.com
TATP AuctionMark SEATS YCSB
JPAB
(Hibernate)
CH-
benCHmark
Voter
SIBench
(Snapshot
Isolation)
SmallBank LinkBench

55. TPC TOOLS
Tools by Transaction
Processing
Council
C sources files
Not exists for TPC-C:
just sample in spec
“Do it
yourself”:
all connectivity
and other stuff

56. TPC-* BY EXAMPLE
По «отчётам о полном раскрытии информации» на TPC.org

57. RPE2
SAP SD
2-Tier
TPC-C
TPC-HSPEC
jbb2005
SPEC
CPU2006
Supercomposite indicator by Gartner (Ideas)
RPE2-ERP
RPE2-Java
RPE2-OLTP
RPE2-Compute
Intensive

58. BENCHWARE
Swiss kinfe by Manfred Drozd
Peakmarks Benchware
OraCPU
PL/SQL op
•[ops]
PL/SQL alg
•[ops]
OraSRV
In-memory
SQL
•[ms]
•[dbps]
•[tps]
•[rps]
OraSTO
SeqIO
•[GBps]
•[iops]
RandIO
•[GBps]
•[iops]
OraOLTP
OLTP Select
•[rps]
•[tps]
OLTP Update
•[rps]
•[tps]
OraLoad
TransLoad
•[rps]
•[tps]
BulkLoad
•[rps]
•[tps]
OraAgg
OraAgg &
Rep
•[rps]
•[tps]
Only for Oracle Database, only PL/SQL and SYS.V_$%

59. DATABASE MACHINES?
Pre-configured
appliances for databases
Should be precisely
measured and
benchmarked?

60. TERADATA
Latest Teradata publications with TPC-H:
Licensed by
«internal Qph» –
tPerf [Traditional Performance]

61. EXADATA
tps, Qph – not published
“Passport metrics” (X6-8)
෍ 𝑉 × IOPS ≈ Const

62. IBM PURE DATA FOR
OPERATIONAL ANALYTICS
Qph not published…
“Passport metrics” about input/output

63. “SQL IOPS”
Measuring SQL IOPS for another DBMS?
Statistical views (…IO_STATS…)
IOPS from Oracle Database side
Orion
(Oracle IO Numbers)
SLOB
(recommended by EMC,
Flashgrid)
Benchware
(?)
DBMS_RESOURCE_MANAGER
.CALIBRATE_IO

64. ATOMIZATION OF
AGGREGATE METRICS
tpmC
QpmH
SQL IOPS
SQL
bandwidth

65. METRICS ATOMIZATION:
PRO
Independent from predefined
models and schemes
Accepted by database
appliance vendors
Pocket tools availability
Representative for wide class
of DBMS and DBMS-like
systems
• Reprehensive not only for
3NF, snowflakes, stars
• Included in passport
metrics
• Running from DB side
• Could be interested for
NoSQL

66. METRICS ATOMIZATION:
…ET CONTRA
Methodology
Different results on
the same
environment from
CALIBRATE_IO
and Benchware
Not clear
application
sense
The same IO
operations in
different DBMSs
produces different
transaction counts
Tool not exists
for most of
DBMS
Found just for Oracle
DB, IBM DB2, MS SQL

67. RUNNING REAL
WORKLOAD AND
APPLICATIONS
BENCHMARKS
Most
practical
approach?

68. WORKLOAD REPLAY AS
TRUSTED EXPERIENCE
For systems with full API access
(usually JSON via HTTP)
Logging and
autocapture
Replay with
“sleeps”
Splitting workload
patterns
(user types)
Data scaling?
Wokrload emulators
JMeter LoadRunner …
DB-side tools
Oracle Real Application Testing
(Database Replay)
MS SQL Server Distributed Replay

69. BOOTSTRAPPING TRICKS
How to
rollout
data?
Clonning
Repeated
with random
shift?
Mixing with
real data
(open data)
Impact to
analytics
Predictable
query results
Low
selectivity
Impact to
OLTP
Keys,
indexes…

70. PACKAGED APPLICATIONS
1С benchmark series
Metrics: count of concurrent users with acceptable response time
Microsoft Dynamics AX Application Benchmark Toolkit
Oracle E-Business Suite Standard Benchamrks
Order-to-Cash OLTP Payroll …
SAPS [SAP Application Performance Standard] with SD module

71. QUESTIONS FOR
DISCUSSION
•Could them reach popularity like TPC-C и TPC-H?
•Megaconfigurations problem
TPC-E и TPC-DS
•Running TPC-B is simple
•Running others is tricky or not fully compliant
Pocket tool
•Acceptance, adoption, standardization, interpretation
•Other metrics of such kind?
SQL IOPS & Bandwidth
•Atomic (YCSB-type)
•Graph (LinkBench-type)
Standardization and generalization of benchmarks for new workloads
•New types of applications: portals, groupware, …
•New database architectures: sharding, in-memory databases, in-memory data grids
Future of benchmarks on real up-to-date workload

72. THANK YOU! mailto:anikolaenko@ibs.ru
mailto:anikolaenko@acm.org
Cover image: Paul Hudson, CC-SA