4. The Seven Stages Of Grief Scaling
1. Shock and Denial
2. Pain and Guilt
3. Anger and
Bargaining
4. Depression &
Reflection
5. The Upward Turn
6. Reconstruction
7. Acceptance and
Hope
5. The Seven Stages Of Grief Scaling
1. Shock and Denial
2. Pain and Guilt
3. Anger and
Bargaining
4. Depression &
Reflection
5. The Upward Turn
6. Reconstruction
7. Acceptance and
Hope
1. Monolithic Scaling
2. Hardware is
Expensive
3. If We Do It This
Way...
4. We Are So *%@#!&
5. Down To 150 Bugs!
6. Release Day
7. Beer & Therapy
(beerapy?)
6. The Problem
● The ability to efficiently backup and restore
● The amount of ram required to keep indexes
in memory
● Resource contention causing query planner
to make sub-optimal choices.
● Aged data extending query resources and
execution time
● Overlap in existing ID spaces
● No account crossover between shards. I.E.
Tii-UK and Tii require separate accounts.
7. Stage 2: Options
● Account based sharding
o Difficult to split account usage evenly across shards.
● Geographical based sharding
o Currently have one geographical shard (UK).
o Added deployment, poor resource utilization.
● Oracle RAC ($$$)
o Oracle OpenWorld is Sunday in SF. No bacon there.
● Horizontal sharding
o Move fast growing tables to separate physical hosts.
o Break relational constraints.
o Good path to a service oriented architecture node.
14. Stage 2: Options
Three Part Two Year Proposal: Short, Mid, and
Long term Goals.
Short: 3 Months
Query Partition and Refactor
Removal of ‘Leaf Service’: Marks
15. Stage 2: Options
Three Part Two Year Proposal: Short, Mid, and
Long term Goals.
Mid: 9 Months
ID Reconciliation Between Shards
Table Partitioning
16. Stage 2: Options
Three Part Two Year Proposal: Short, Mid, and
Long term Goals.
Long: 12 Months
Create DAL
Removal of Large Tables
Global Statistics and Reporting
17. Stage 2: Options
Short Term: 12 Months Later
I do not think it means what you think it
means.
21. Stage 3: Scoping The Solution - Database
Original: 236 tables
New main database (192 tables)
New marks database (40 tables)
22. Stage 3: Scoping The Solution - Code
Option 1 - Data Access Layer (DAL)
o Separate codebase encapsulating new set of tables
o Written in Golang, an HTTP based REST service
o Avoids carrying forward existing technical debt
o Requires detailed knowledge of existing product features
o Unit tests are very helpful, but coverage is never 100%
o 14 years of business logic (dark matter)
o In long lived web apps, tribal knowledge is authoritative
23. Stage 3: Scoping The Solution - Code
Option 2 - Add additional database handles to new db
o Perceived as a safer approach (deciding factor,
known risks).
o Requires paying interest on existing technical debt.
o Refactoring is less risky than rewriting.
o Take advantage of existing business logic and tribal
knowledge.
o Preserve sacred cows.
24. Stage 3: Scoping The Solution -
Hardware
"We can use smaller hardware because we are
splitting off part of the database"
➢ This is somewhat of a fallacy
➢ You might need smaller storage
➢ You might need slightly less CPU
➢ Stick with close to the same amount of RAM
25. Stage 4: Implementation - Rollback
S: “What if this fails?”
F: “We Rollback the code, restore the database,
and look for new jobs.”
26. Stage 4: Implementation - Rollback
Q: How do you bifurcate a database and
rollback without data loss?
A: Slony.
27. Stage 4: Implementation - Rollback
Timelines matter. Prepare in advance.
Split Replication Well In Advance.
Test Process, Then Test It Again.
28. Stage 4: Implementation - Archaeology
● What is this table? That service doesn’t exist
anymore?
○ Let’s Drop it!
● What’s that table? It’s an old version still in
use?
○ Let’s Drop it!
● What’s that one over there?
○ Let’s Drop it!
30. Stage 4: Implementation - Archaeology
● Fourteen years of application development.
● Five major codebases, dozens of support utilities.
● Hundreds of codepoints for database connections.
● A dozen different ORMs.
● Dynamically generated SQL joining tables.
● Technical debt (code with high maintenance costs).
● Best practices of 10 years ago are now liabilities.
31. How do you change all of
the electrical sockets in an
(old) office building?
Stage 4: Implementation - Archaeology
34. Stage 4: Implementation - Archaeology
James left 8 years ago. The elevator is in old building.
They tore down the old building to build a Target.
# this code is critical to our workflow, don’t remove it!!
# for details talk to jamesb <> who sits near the elevator
# $foo = $object->flocculate( key => $cfg->secret_key );
# return $foo;
return;
35. Stage 4: Implementation - Archaeology
Bob is still here though. Bob is a little particular
about his code though (we are all to some degree).
Now you’re in there meddling with Bob’s code. How
would you feel if you were Bob?
A little empathy goes a long way towards getting Bob
to help you get his code ported to the new dual
database schema.
36. Stage 4: Implementation - Queries
main database - marks database
SELECT count(m.*) FROM
gm3_mark m, gm3_qm_template qmt
WHERE m.read IN
(SELECT dgr.id FROM m_dg_read dgr
JOIN m_object_paper mop ON (mop.id =
dgr.source AND mop.owner = ?)
JOIN m_assignment ma ON (ma.id =
mop.assignment AND ma.class = ?) WHERE
reader = ?)
AND m.qm_template = qmt.id AND qmt.id = ?
37. Main Database - grab ids to pass to marks database.
SELECT p.id
FROM m_object_paper p
JOIN m_assignment a
ON a.id = p.assignment
WHERE a.class = ?
AND p.owner = ?
Stage 4: Implementation - Queries
38. Stage 4: Implementation - Queries
Marks database - pass former FK ids to an IN clause.
SELECT count(m.*)
FROM gm3_mark m
JOIN gm3_qm_template qmt
ON qmt.id = m.qm_template
JOIN m_dg_read dgr
ON dgr.id = m.read
WHERE dgr.source IN (?, ?, ?)
AND qmt.id = ? AND dgr.reader = ?
39. Stage 4: Implementation - Transactions
Single database transactions are easy.
eval {
$db->do(“INSERT INTO foo (name) VALUES
(‘bar’)”);
$id = $db->do(“SELECT CURRVAL(‘foo’)”);
$db->do(“INSERT INTO fee (foo_id) VALUES
($id)”);
};
if ($@) { # catch exception
$db->rollback; # roll transaction back
} else {
$db->commit; # commit transaction
}
40. Stage 4: Implementation - Transactions
Dual database transactions are harder.
eval {
# insert into foo in main db, grab last value
$main_db->do(“INSERT INTO foo VALUES (‘bar’)”);
$foo_id = $main_db->do(“SELECT CURRVAL(‘foo’)”);
# insert foo id into marks db, grab last value
$marks_db->do(“INSERT INTO fee VALUES ($id)”);
$fee_id = $main_db->do(“SELECT CURRVAL(‘fee’)”);
};
41. Stage 4: Implementation - Transactions
Roll back both handles on exception, commit both on
success.
if ($@) { # catch exception
$main_db->rollback; # roll main_db back
$marks_db->rollback; # roll marks_db back
} else {
$main_db->commit; # commit main_db
$marks_db->commit; # commit marks_db
}
42. Stage 4: Implementation - Transactions
What if the commit fails?
if ($@) { # catch exception
$main_db->rollback; # roll main_db back
$marks_db->rollback; # roll marks_db back
} else {
eval { $main_db->commit };
if ($@) {
$main_db->rollback;
$marks_db->rollback;
}
eval { $marks_db->commit }; ...
45. Stage 4: Implementation - Transactions
9 out of 10 users
prefer availability
So does customer support.
You can fix consistency.
46. Stage 4: Implementation - ORMs
ORMs are full of pain
● They hide away db connection details.
● They make it hard to break models apart.
● They make writing code easy…
● But debugging is much more difficult.
47. Stage 4: Implementation - ORMs
ORMs are full of pain
Back in my day we used SQL, and we liked it.
$classes = $c->classes->search( $select_hash, {
'+select' => 'source.id',
'+as' => 'src_id',
'join' => [ { 'user_rights_class' =>
{ 'user_role' => 'owner' } }, 'source' ],
'rows' => 200,
'page' => 1
} );
51. Stage 4: Implementation - Juggling
Main database - Marks database
I think he was talking to me.
52. Stage 4: Implementation - Config
● Main Database: One master, two slaves (2)
● Marks Database: One master, two slaves (2)
● ASP application: write user, read only user (2)
● Catalyst Application: write user, read only user (2)
● REST Application: write user, read only user (2)
● dev, qa, staging, production, sandbox, uk (6)
53. Stage 4: Implementation - Config
● Database hosts and users: 2*5 = 10
● Stages: 10 * 6 = 60
● Config managed in version control, no discovery.
● Config deployed via RPM with application.
● Get one wrong? Start all over again.
● Configuration is full of pain and suffering.
63. Stage 6: Cleanup
Patch Flavors:
How Did That Get there?
That’s a bug.
It worked fine in dev.
64. Stage 6: Cleanup
“Sometimes the query planner does dumb things”
o People forget why you embarked on this effort.
o People forget the successes and risk mitigation.
o People won’t forget the visceral reactions to
service degradations.
65. Stage 6: Cleanup
How to bring your site to a halt:
1.Start transaction to database 1
2.Start transaction to database 2
3.Wait for database 1 to finish