1. Big Data For OLTP
Handling Massive Writes
Liran Zelkha, Tona Consulting
Liran.zelkha@gmail.com

2. Intro
• Israel’s Big Data Meetup
• By Developers, For Developers
• We talk code, architecture, solutions
– No products
– No sales pitches
• Suggest ideas for next meetups
• Suggest yourselves as speakers for next meetups

3. About Me
• Liran Zelkha, from Tona Consulting
• Formerly co-founder at ScaleBase
– NewSQL solution for scaling RDBMS
• Works (allot) with applications that need to
scale, both reads and writes

4. Agenda
• Terminology
• Massive Reads vs. Massive Writes
• Solutions
– RDBMS
– NoSQL
– Code

5. TERMINOLOGY

6. Terminology
• OLTP
– a class of systems that facilitate and manage
transaction-oriented applications, typically for data
entry and retrieval transaction processing. The term is
somewhat ambiguous; some understand a
"transaction" in the context of computer or database
transactions, while others (such as the Transaction
Processing Performance Council) define it in terms of
business or commercial transactions. OLTP has also
been used to refer to processing in which the system
responds immediately to user requests. An automatic
teller machine (ATM) for a bank is an example of a
commercial transaction processing application.
http://en.wikipedia.org/wiki/Online_transaction_processing

7. Terminology
• OLAP
– s an approach to swiftly answer multi-dimensional
analytical (MDA) queries. OLAP is part of the broader
category of business intelligence, which also
encompasses relational reporting and data mining.
Typical applications of OLAP include business
reporting for sales, marketing, management
reporting, business process management (BPM),
budgeting and forecasting, financial reporting and
similar areas, with new applications coming up, such
as agriculture. The term OLAP was created as a slight
modification of the traditional database term OLTP
(Online Transaction Processing).
http://en.wikipedia.org/wiki/Online_analytical_processing

8. MASSIVE READS VS. MASSIVE
WRITES

9. Reads vs. Writes
Reads Writes
• Caching helps • Caching sucks
• No need for availability • Availability is a must (?)
• No transactions • Transactions are a must (?)
• No locking • Locking is a must (?)

10. Massive Reads Solutions
• Memory, Memory, Memory
– For Caching, Caching, Caching
• Column stores (?)

11. Caching
• Tons of solutions.
• To name a few:
– java.util.Map
– Hazelcast
– Infinispan
– Coherence

12. When Caching Consider
• Time To Live
• Memory Leaks
• Updates (?)
• What is cached and where

13. Column Store Databases
• Store data in columns vs. rows
• Faster reads, slower writes
• Compression helps store more data in less disk
space

14. Massive Write Solutions
• Memory?
– Only with fast networks
• Fast disks

15. Memory
• Memory can fail, machines can fail
• Distributed memory
• Size of memory
• See Nati Shalom’s points at
http://natishalom.typepad.com/nati_shaloms
_blog/2010/03/memory-is-the-new-disk-for-
the-enterprise.html

16. Memory Is The New Disk

17. Fast Disks
• When massive writes translates to massive
disk writes
– Fast disks are a must
• Can offer
– HA
– Scalability
– Low latency

18. 2 Words On Storage Technologies
• RAID
• SSD
• SAN
• NAS

19. Example
• NMS System
• Each device interaction is built from 5-10
request/response pair
• Each request causes up to 3 database insert/updates
– And multiple reads
• Support up to 5M devices
• Technology stack
– Jboss 6
– EJB3, JPA
– Oracle Database

20. JDBC Profiling – Standard Disks

21. JDBC Profiling – Fast SAN
• Spec
– HP P2000 with 16 drives configured as:
• 14 300G 10K SAS Drives in RAID 10 (Data)
• 2 300G 10K SAS Drives in RAID 1 (Redo)
• Write-back is enabled
– Sisk enclosure is connected via FC Switch and 8GB
Qlogic HBA on the server side.

22. JDBC Profiling – Fast SAN

23. RDBMS

24. MySQL Scaling Options
• Tuning
• Hardware upscale
• Partitioning
• New MySQL distribution
• Read/Write Split

25. Database Tuning
• There are many ways to tune your database
• Allot of data online, check out this post
– http://forge.mysql.com/wiki/Top10SQLPerforman
ceTips

26. Database Tuning – Some Examples
• innodb_buffer_pool_size
– Holds the data and indexes of tables in memory.
– Bigger buffer results in faster row lookups.
– The bigger the better.
– Default – 8M
• Query Cache
– Keeps the result of queries in memory until they are invalidated by
writes.
– query_cache_size
• total size of memory available to query caching
– query_cache_limit
• the maximum number of kilobytes one query may be in order to be cached.
– query_cache_size = 128MB
– query_cache_limit = 4MB

27. Database Tuning – Pros and Cons
Pros Cons
May result in major Doesn’t scale. No matter how well
performance improvements the tuning is performed, it will
reach a limit defined by machine
capabilities.
Doesn’t require application
changes

28. SQL Tuning
• If you write lousy SQL code, you’ll get lousy
performance
– Java gurus are not SQL gurus
– Your ORM code does not know how to write good
SQL code
• What will happen when executing
– SELECT * FROM
• Tuning your SQL commands is tedious but very
rewarding

29. SQL Tuning – Some Examples
• Here are just some examples:
– Use EXPLAIN to profile the query execution plan
– Use DISTINCT – not GROUP BY
– Don’t use an indexed column with a function
– Try not to use a non deterministic functions in
where clause

30. SQL Tuning – Pros and Cons
Pros Cons
May result in major Requires code modifications.
performance improvements
Doesn’t scale. No matter how
well the tuning is performed, it
will reach a limit defined by
machine capabilities.

31. Scaling Up Hardware
• Usually DB gets the strongest servers
• However – there is a limit to how much performance
gains you can get from increasing hardware
• Some data:
http://www.mysqlperformanceblog.com/2011/01/26/modeling-innodb-scalability-on-
multi-core-servers/

32. Scaling Up Hardware – Pros and Cons
Pros Cons
May result in major performance improvements Scaling is limited
Transparent Might be expensive
Easy

33. SSD
• Solid State Drive
– Better latency and access time than regular HDD
– Cost more per GB (but prices are dropping)
• Vadim Tkachenko from Percona gave a great
lecture on SSD at MySQL Conf 2011
– (see slides at
http://en.oreilly.com/mysql2011/public/schedule/det
ail/17117)
– Claims you can expect up to X7 performance from SSD

34. SSD – Pros and Cons
Pros Cons
May result in major performance improvements Expensive
Transparent Still limited scalability

35. Partitioning
• Partitioning was introduced to MySQL at
version 5.1.
• It is a way to split tables across multiple files, a
technique that proved very useful for
improving database performance.
• Benefits:
– Helps fit indexes in RAM
– Faster query/insert
– Instant delete

36. Partitioning Performance
• See excellent presentation by Giuseppe Maxia
from 2010
– http://www.slideshare.net/datacharmer/partition
s-performance-with-mysql-51-and-55
Engine 6 month range query
InnoDB 4min 30s
MyISAM 25.03s
InnoDB partitions 13.19s
MyISAM partiotions 4.45s

37. Partitioning
Pros Cons
May result in major performance MySQL server itself introduces limits. User
improvements concurrency, transaction chains, isolation,
are still bottlenecked by the single MySQL
that owns all partitions
Mostly transparent to the application

38. New MySQL Distributions
• There are many MySQL drop-in replacements
• Are MySQL, but tuned differently, different
extensions
• Leading examples
– PerconaServer
– MariaDB

39. New MySQL Distributions – Pros and
Cons
Pros Cons
Provide performance Still limited scalability
improvements
Transparent

40. Other Storage Engines
• InnoDB better than MyISAM
– Oh really?
– As always, it depends.
– InnoDB will cause less corruptions, and is probably
better for most high traffic applications.
• MEMORY can be great
– However, no persistency

41. Read/Write Splitting
• Write to MySQL master, read from 1 (or more)
slaves
• Excellent read scaling
• Many issues:
– Since replication is a-synchronous – read might
not be up to date
– Transactions create stickiness
– Code changes

42. Read/Write Splitting – Pros and Cons
Pros Cons
Provides performance Requires code changes
improvements
Scale out the database Good for scaling reads, not writes
Since replication is asynchronous, some reads
might get data that is not up to date.

43. Sharding
DB1
App
DB2

44. MySQL + NoSQL - HandlerSocket
• Fascinating post -
http://yoshinorimatsunobu.blogspot.com/2010/1
0/using-mysql-as-nosql-story-for.html
• MySQL spends huge amount of time on SQL
statement parsing
• Using InnoDB API directly
• MySQL Plugin
• Comes builtin with Percona Server 5.5

45. HandlerSocket - Architecture

46. Code Sample
#!/usr/bin/perl
use strict;
use warnings;
use Net::HandlerSocket;
#1. establishing a connection
my $args = { host => 'ip_to_remote_host', port => 9998 };
my $hs = new Net::HandlerSocket($args);
#2. initializing an index so that we can use in main logics.
# MySQL tables will be opened here (if not opened)
my $res = $hs->open_index(0, 'test', 'user', 'PRIMARY',
'user_name,user_email,created');
die $hs->get_error() if $res != 0;

47. Code Sample – Cont’
#3. main logic
#fetching rows by id
#execute_single (index id, cond, cond value, max rows, offset)
$res = $hs->execute_single(0, '=', [ '101' ], 1, 0);
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
for (my $row = 0; $row < 1; ++$row) {
my $user_name= $res->[$row + 0];
my $user_email= $res->[$row + 1];
my $created= $res->[$row + 2];
print "$user_namet$user_emailt$createdn";
}
#4. closing the connection
$hs->close();

48. Bashing Some NewSQL Solutions
• Xeround
– Limited Database size
– Only on the cloud
• VoltDB
– Rewrite your entire app to use stored procedures
• NimbusDB
– Still in Beta
• Clustrix
– Insanely expensive
– NoSQL that looks like MySQL
• Schooner
– Fast MySQL on SSD
• And no word on ScaleBase

49. NOSQL

50. NoSQL Is Here To Stay

51. NoSQL
• A term used to designate databases which
differ from classic relational databases in
some way. These data stores may not require
fixed table schemas, and usually
avoid join operations and typically scale
horizontally. Academics and papers typically
refer to these databases as structured storage,
a term which would include classic relational
databases as a subset.
http://en.wikipedia.org/wiki/NoSQL

52. NoSQL Types
• Key/Value
– A big hash table
– Examples: Voldemort, Amazon Dynamo
• Big Table
– Big table, column families
– Examples: Hbase, Cassandra
• Document based
– Collections of collections
– Examples: CouchDB, MongoDB
• Graph databases
– Based on graph theory
– Examples: Neo4J
• Each solves a different problem

53. NO-SQL
http://browsertoolkit.com/fault-tolerance.png

54. MongoDB
• I use the slides of Roger Bodamer from 10gen
• Find them here:
– http://assets.en.oreilly.com/1/event/61/Building%
20Web%20Applications%20with%20MongoDB%2
0Presentation.ppt
• In my book – Mongo doesn’t fit the massive
write story.

55. MongoDB
• Document Oriented Database
– Data is stored in documents, not tables / relations
• MongoDB is Implemented in C++ for best performance
• Platforms 32/64 bit Windows Linux, Mac OS-X, FreeBSD, Solaris
• Language drivers for:
– Ruby / Ruby-on-Rails
– Java
– C#
– JavaScript
– C / C++
– Erlang Python, Perl others..... and much more ! ..

56. Design
• Want to build an app where users can check in
to a location
• Leave notes or comments about that location
• Iterative Approach:
– Decide requirements
– Design documents
– Rinse, repeat :-)

57. Requirements
• Locations
– Need to store locations (Offices, Restaurants etc)
• Want to be able to store name, address and tags
• Maybe User Generated Content, i.e. tips / small notes ?
– Want to be able to find other locations nearby

58. Requirements
• Locations
– Need to store locations (Offices, Restaurants etc)
• Want to be able to store name, address and tags
• Maybe User Generated Content, i.e. tips / small notes ?
– Want to be able to find other locations nearby
• Checkins
– User should be able to ‘check in’ to a location
– Want to be able to generate statistics

59. Terminology
RDBMS Mongo
Table, View Collection
Row(s) JSON Document
Index Index
Join Embedded Document
Partition Shard
Partition Key Shard Key

60. Collections
loc1, loc2, loc3 User1, User2
Location
Users
s

61. JSON Sample Doc
{ _id : ObjectId("4c4ba5c0672c685e5e8aabf3"),
author : "roger",
date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)",
text : ”MongoSF",
tags : [ ”San Francisco", ”MongoDB" ] }
Notes:
- _id is unique, but can be anything you’d like

62. BSON
• JSON has powerful, but limited set of datatypes
– Mongo extends datypes with Date, Int types, Id, …
• MongoDB stores data in BSON
• BSON is a binary representation of JSON
– Optimized for performance and navigational abilities
– Also compression
– See bsonspec.org

63. Locations v1
location1= {
name: "10gen East Coast”,
address: ”134 5th Avenue 3rd Floor”,
city: "New York”,
zip: "10011”
}

64. Places v1
location1= {
name: "10gen East Coast”,
address: ”134 5th Avenue 3rd Floor”,
city: "New York”,
zip: "10011”
}
db.locations.find({zip:”10011”}).limit(10)

65. Places v2
location1 = {
name: "10gen East Coast”,
address: "17 West 18th Street 8th Floor”,
city: "New York”,
zip: "10011”,
tags: [“business”, “mongodb”]
}

66. Places v2
location1 = {
name: "10gen East Coast”,
address: "17 West 18th Street 8th Floor”,
city: "New York”,
zip: "10011”,
tags: [“business”, “mongodb”]
}
db.locations.find({zip:”10011”, tags:”business”})

67. Places v3
location1 = {
name: "10gen East Coast”,
address: "17 West 18th Street 8th Floor”,
city: "New York”,
zip: "10011”,
tags: [“business”, “mongodb”],
latlong: [40.0,72.0]
}

68. Places v3
location1 = {
name: "10gen East Coast”,
address: "17 West 18th Street 8th Floor”,
city: "New York”,
zip: "10011”,
tags: [“business”, “cool place”],
latlong: [40.0,72.0]
}
db.locations.ensureIndex({latlong:”2d”})

69. Places v3
location1 = {
name: "10gen HQ”,
address: "17 West 18th Street 8th Floor”,
city: "New York”,
zip: "10011”,
tags: [“business”, “cool place”],
latlong: [40.0,72.0]
}
db.locations.ensureIndex({latlong:”2d”})
db.locations.find({latlong:{$near:[40,70]}})

70. Places v4
location1 = {
name: "10gen HQ”,
address: "17 West 18th Street 8th Floor”,
city: "New York”,
zip: "10011”,
latlong: [40.0,72.0],
tags: [“business”, “cool place”],
tips: [
{user:"nosh", time:6/26/2010, tip:"stop by for
office hours on Wednesdays from 4-6pm"},
{.....},
]
}

71. Querying your Places
Creating your indexes
db.locations.ensureIndex({tags:1})
db.locations.ensureIndex({name:1})
db.locations.ensureIndex({latlong:”2d”})
Finding places:
db.locations.find({latlong:{$near:[40,70]}})
With regular expressions:
db.locations.find({name: /^typeaheadstring/)
By tag:
db.locations.find({tags: “business”})

72. Inserting and updating locations
Initial data load:
db.locations.insert(place1)
Using update to Add tips:
db.locations.update({name:"10gen HQ"},
{$push :{tips:
{user:"nosh", time:6/26/2010,
tip:"stop by for office hours on
Wednesdays from 4-6"}}}}

73. Requirements
• Locations
– Need to store locations (Offices, Restaurants etc)
• Want to be able to store name, address and tags
• Maybe User Generated Content, i.e. tips / small notes ?
– Want to be able to find other locations nearby
• Checkins
– User should be able to ‘check in’ to a location
– Want to be able to generate statistics

74. Users
user1 = {
name: “nosh”
email: “nosh@10gen.com”,
.
.
.
checkins: [{ location: “10gen HQ”,
ts: 9/20/2010 10:12:00,
…},
…
]
}

75. Simple Stats
db.users.find({„checkins.location‟: “10gen HQ”)
db.checkins.find({„checkins.location‟: “10gen HQ”})
.sort({ts:-1}).limit(10)
db.checkins.find({„checkins.location‟: “10gen HQ”,
ts: {$gt: midnight}}).count()

76. Alternative
user1 = {
name: “nosh”
email: “nosh@10gen.com”,
.
.
.
checkins: [4b97e62bf1d8c7152c9ccb74, 5a20e62bf1d8c736ab]
}
checkins [] = ObjectId reference to locations collection

77. User Check in
Check-in = 2 ops
read location to obtain location id
Update ($push) location id to user object
Queries: find all locations where a user checked in:
checkin_array = db.users.find({..},
{checkins:true}).checkins
db.location.find({_id:{$in: checkin_array}})

78. Unsharded Deployment
•Configure as a replica set for
Primary
automated failover
•Async replication between nodes
Secondary •Add more secondaries to scale reads
Secondary

79. Sharded Deployment
MongoS
confi
g
Primary
Secondary
•Autosharding distributes data among two or more replica sets
•Mongo Config Server(s) handles distribution & balancing
•Transparent to applications

80. Cassandra
• Slides used from eben hewitt
• See original slides here:
– http://assets.en.oreilly.com/1/event/51/Scaling%2
0Web%20Applications%20with%20Cassandra%20
Presentation.ppt

81. cassandra properties
• tuneably consistent
• very fast writes
• highly available
• fault tolerant
• linear, elastic scalability
• decentralized/symmetric
• ~12 client languages
– Thrift RPC API
• ~automatic provisioning of new nodes
• 0(1) dht
• big data

82. write op

83. Staged Event-Driven Architecture
• A general-purpose framework for high
concurrency & load conditioning
• Decomposes applications into stages
separated by queues
• Adopt a structured approach to event-driven
concurrency

84. instrumentation

85. data replication
• configurable replication factor
• replica placement strategy
rack unaware  Simple Strategy
rack aware  Old Network Topology Strategy
data center shard  Network Topology Strategy

86. partitioner smack-down
Random Preserving Order Preserving
• system will use MD5(key) to • key distribution determined
distribute data across nodes by token
• even distribution of keys • lexicographical ordering
from one CF across • required for range queries
ranges/nodes – scan over rows like cursor in
index
• can specify the token for
this node to use
• ‘scrabble’ distribution

87. agenda
• context
• features
• data model
• api

88. structure
keyspace
column family
settings
(eg,
partitioner) settings (eg,
column
comparator,
type [Std]) name value clock

89. keyspace
• ~= database
• typically one per application
• some settings are configurable only per
keyspace

90. column family
• group records of similar kind
• not same kind, because CFs are sparse tables
• ex:
– User
– Address
– Tweet
– PointOfInterest
– HotelRoom

91. think of cassandra as
row-oriented
• each row is uniquely identifiable by key
• rows group columns and super columns

92. column family
key nickname=
user=eben The
123 Situation
key icon= n=
user=alison
456 42

93. json-like notation
User {
123 : { email: alison@foo.com,
icon: },
456 : { email: eben@bar.com,
location: The Danger Zone}
}

94. example
$cassandra –f
$bin/cassandra-cli
cassandra> connect localhost/9160
cassandra> set
Keyspace1.Standard1[‘eben’][‘age’]=‘29’
cassandra> set
Keyspace1.Standard1[‘eben’][‘email’]=‘e@e.com’
cassandra> get Keyspace1.Standard1[‘eben'][‘age']
=> (column=6e616d65, value=39,
timestamp=1282170655390000)

95. a column has 3 parts
1. name
– byte[]
– determines sort order
– used in queries
– indexed
2. value
– byte[]
– you don’t query on column values
3. timestamp
– long (clock)
– last write wins conflict resolution

96. column comparators
• byte
• utf8
• long
• timeuuid
• lexicaluuid
• <pluggable>
– ex: lat/long

97. super column
super columns group columns under a common name

98. super column family
<<SCF>>PointOfInterest
<<SC>>Central <<SC>>
Park Empire State Bldg
10017 desc=Fun to desc=Great
phone=212.
walk in. view from
555.11212
102nd floor!
<<SC>>
85255 Phoenix
Zoo

99. super column family
super column family
PointOfInterest {
key: 85255 { column
Phoenix Zoo { phone: 480-555-5555, desc: They have animals here. },
Spring Training { phone: 623-333-3333, desc: Fun for baseball fans. },
}, //end phx
key
super column
key: 10019 { flexible schema
Central Park { desc: Walk around. It's pretty.} ,
s
Empire State Building { phone: 212-777-7777,
desc: Great view from 102nd floor. }
} //end nyc
}

100. about super column families
• sub-column names in a SCF are not indexed
– top level columns (SCF Name) are always indexed
• often used for denormalizing data from
standard CFs

101. slice predicate
• data structure describing columns to return
– SliceRange
• start column name
• finish column name (can be empty to stop on count)
• reverse
• count (like LIMIT)

102. • get() : Column
– get the Col or SC at given ColPath
read api
COSC cosc = client.get(key, path, CL);
• get_slice() : List<ColumnOrSuperColumn>
– get Cols in one row, specified by SlicePredicate:
List<ColumnOrSuperColumn> results =
client.get_slice(key, parent, predicate, CL);
• multiget_slice() : Map<key, List<CoSC>>
– get slices for list of keys, based on SlicePredicate
Map<byte[],List<ColumnOrSuperColumn>> results =
client.multiget_slice(rowKeys, parent, predicate, CL);
• get_range_slices() : List<KeySlice>
– returns multiple Cols according to a range
– range is startkey, endkey, starttoken, endtoken:
List<KeySlice> slices = client.get_range_slices(
parent, predicate, keyRange, CL);

103. client.insert(userKeyBytes, parent, write api
new Column(“band".getBytes(UTF8),
“Funkadelic".getBytes(), clock), CL);
batch_mutate
– void batch_mutate(
map<byte[], map<String, List<Mutation>>> , CL)
remove
– void remove(byte[],
ColumnPath column_path, Clock, CL)

104. //create param
batch_mutate
Map<byte[], Map<String, List<Mutation>>> mutationMap =
new HashMap<byte[], Map<String, List<Mutation>>>();
//create Cols for Muts
Column nameCol = new Column("name".getBytes(UTF8),
“Funkadelic”.getBytes("UTF-8"), new Clock(System.nanoTime()););
Mutation nameMut = new Mutation();
nameMut.column_or_supercolumn = nameCosc; //also phone, etc
Map<String, List<Mutation>> muts = new HashMap<String, List<Mutation>>();
List<Mutation> cols = new ArrayList<Mutation>();
cols.add(nameMut);
cols.add(phoneMut);
muts.put(CF, cols);
//outer map key is a row key; inner map key is the CF name
mutationMap.put(rowKey.getBytes(), muts);
//send to server
client.batch_mutate(mutationMap, CL);

105. what about…
SELECT WHERE
ORDER BY
JOIN ON
GROUP

106. rdbms: domain-based model
what answers do I have?
cassandra: query-based model
what questions do I have?

107. SELECT WHERE
cassandra is an index factory
<<cf>>USER
Key: UserID
Cols: username, email, birth date, city, state
How to support this query?
SELECT * FROM User WHERE city = ‘Scottsdale’
Create a new CF called UserCity:
<<cf>>USERCITY
Key: city
Cols: IDs of the users in that city.
Also uses the Valueless Column pattern

108. SELECT WHERE pt 2
• Use an aggregate key
state:city: { user1, user2}
• Get rows between AZ: & AZ;
for all Arizona users
• Get rows between AZ:Scottsdale &
AZ:Scottsdale1
for all Scottsdale users

109. ORDER BY
Columns Rows
are sorted according to are placed according to their Partitioner:
CompareWith or
CompareSubcolumnsWith
•Random: MD5 of key
•Order-Preserving: actual key
are sorted by key, regardless of partitioner

110. rdbms

111. cassandra

112. When To Use NoSQL
• No schema – No SQL
– Don’t do this KV DB design
• Terrible performance, impossible to maintain
• No persistency – No SQL
– Heck – use a distributed cache for that
• Low write latency
– And fast storage is not an option
• Simple queries
– You can always ETL to a DB later
• Cool factor
– Good luck with that

113. Real Life NoSQL Usages
• MongoDB is great for CMS
– Try MTV…
• Cassandra is great for low latency writes
• Use the right tool for the job – or you’ll get
worse performance than a DB
• Expect a very high learning curve to
implement

114. SpringData
• In your face – there are frameworks that use
NoSQL

115. And To Finish
• A long time ago people told me Java doesn’t
perform nor scale…

116. CODE

117. ORM
• Check the queries created
– For instance, in Hibernate
• setLast,setFirst – kill your performance
• Lazy fetching
• N+1
• Use batch updates

118. Hibernate With Batch Updates
session.doWork(new Work() {
public void execute(Connection connection) throws SQLException {
PreparedStatement stmt = connection.prepareStatement(“insert into test
values(?,?)");
stmt.setLong(1, id);
stmt.setLong(2, transId);
call.execute();
}
});

119. JDBC Code
• Always use prepared statements
• Use your Database performance extensions
– For instance, massive inserts for Column Store DBs

120. Transactions
• One transaction per hit. At most.
• Reads can hurt your writes

121. Database Tuning
• We talked about it, but:
– Indexes – good for read, bad for write
– Multi column indexes – good only if query reads using
the same order of columns
– Indexes and views
– EXPLAIN is your friend
• Keep your DB small
– BI against read-replica
– Delete history
– Ensure you only save what you actually need

122. Final Note
• NoSQL, SQL – it doesn’t matter if your code
sucks!

123. 2 Words On Cloud
• Storage sucks
• Network sucks
• So
– Cache locally
– Write a-sync
– Write to files, not DB
– Don’t use RDS
– Check Cloud providers that offer SSD