What are deadlocks and how to debug and fix them.

1. Intro to
Deadlocks
February 4th, 2013
Panupan Sriautharawong
Monday, February 11, 13

2. What are
Deadlocks?
A situation that occurs in concurrent systems.
Happens when two or more competing
actions get stuck because they’re waiting on
one another to finish.
A bug or design flaw in most cases.
Preventable!
Monday, February 11, 13

3. InnoDB
Storage Engine
Monday, February 11, 13

4. InnoDB
Storage Engine
Transactions
Are used to group of database operations
## Bank Transfer Example
BEGIN;
UPDATE account SET balance=balance+100 WHERE id=1;
UPDATE account SET balance=balance-100 WHERE id=2;
COMMIT;
Are atomic, meaning they can’t broken into
smaller pieces. Either everything goes
through or nothing does.
Monday, February 11, 13

5. InnoDB
Storage Engine
Shared vs Exclusive Locks
A lock must be acquired (either implicitly or
explicitly) before retrieving or modifying any data.
There are two types of locks:
- Shared (S) = Read only
- Exclusive (X) = Read/Write
Table vs Row-Level Locking
Properly defined indexes prevent too
many rows from unintentionally getting locked and
also ensures optimal performance.
Monday, February 11, 13

6. Deadlocked Doggies
Jake
Tony
Tony starts with 5 bones
Jake starts with 1 bone
Action #1: Jake steals a bone from Tony
Action #2: Tony, being a good doggie, gives
Jake a bone so they’ll both have 3
Monday, February 11, 13

7. Deadlocked Doggies
## INITIAL SETUP
CREATE TABLE `doggies` (
`id` INT NOT NULL AUTO_INCREMENT,
Jake
`name` VARCHAR(255) NOT NULL,
`bones` INT NOT NULL,
PRIMARY KEY (`id`) Tony
) ENGINE=InnoDB;
INSERT INTO doggies (id, name, bones) VALUES (1, "Tony", 5);
INSERT INTO doggies (id, name, bones) VALUES (2, "Jake", 1);
## ACTION #1
# Jake steals a bone
BEGIN;
UPDATE doggies SET bones=bones-1 WHERE id=1; # X lock acquired on 1
## ACTION #2
# Tony gives Jake a bone
BEGIN;
UPDATE doggies SET bones=bones+1 WHERE id=2; # X lock acquired on 2
UPDATE doggies SET bones=bones-1 WHERE id=1;! # LOCK WAIT!
COMMIT;
## ACTION #1 (continued)
UPDATE doggies SET bones=bones+1 WHERE id=2;! # DEADLOCK!
COMMIT;
Monday, February 11, 13

8. How to Debug’em
Inspecting MySQL Status
mysql> SHOW ENGINE INNODB STATUS;
mysql> SHOW PROCESSLIST;
mysql> EXPLAIN SELECT ...;
Grepping Rails Logs
$ cat log/production.log | grep -B 40 Deadlock
Monday, February 11, 13

9. ------------------------
How to Debug’em
LATEST DETECTED DEADLOCK
------------------------
130204 13:10:32
*** (1) TRANSACTION:
TRANSACTION 214FC, ACTIVE 12 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 376, 2 row lock(s), undo log entries 1
MySQL thread id 182, OS thread handle 0x111c8c000, query id 120976 localhost 127.0.0.1 root Updating
UPDATE doggies SET bones=bones-1 WHERE id=1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FC lock_mode X locks rec but not gap waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 4; hex 80000001; asc ;;
1: len 6; hex 0000000214fb; asc ;;
2: len 7; hex 7900000de72774; asc y 't;;
3: len 4; hex 546f6e79; asc Tony;;
4: len 4; hex 80000004; asc ;;
*** (2) TRANSACTION:
TRANSACTION 214FB, ACTIVE 44 sec starting index read
mysql tables in use 1, locked 1
3 lock struct(s), heap size 376, 2 row lock(s), undo log entries 1
MySQL thread id 184, OS thread handle 0x111e2b000, query id 120977 localhost 127.0.0.1 root Updating
## CONNECTION #1
# Jake steals a bone (cont..)
UPDATE doggies SET bones=bones+1 WHERE id=2
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FB lock_mode X locks rec but not gap
Record lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 4; hex 80000001; asc ;;
1: len 6; hex 0000000214fb; asc ;;
2: len 7; hex 7900000de72774; asc y 't;;
3: len 4; hex 546f6e79; asc Tony;;
4: len 4; hex 80000004; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FB lock_mode X locks rec but not gap waiting
Record lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 4; hex 80000002; asc ;;
1: len 6; hex 0000000214fc; asc ;;
2: len 7; hex 7a00000cf122d8; asc z " ;;
3: len 4; hex 4a616b65; asc Jake;;
4: len 4; hex 80000002; asc ;;
*** WE ROLL BACK TRANSACTION (2)
Monday, February 11, 13

10. Best Practices
Define proper indexes and use EXPLAIN SELECT to
verify. For example, make sure to define multi-column
indexes across [id, type] on our polymorphic models.
Most common method of deadlock exception handling
is to retry but this should only be used a last resort.
Easiest fix is to rearrange queries to make sure updates
occur before selects (not always possible).
You can also lock records manually at the beginning of a
transaction when you know they’ll need updating.
Monday, February 11, 13

11. Helpful Resources
http://dev.mysql.com/doc/refman/5.0/en/
innodb-lock-modes.html
http://rhnh.net/2010/07/01/debugging-
deadlocks-in-rails
http://www.chriscalender.com/?p=426
Monday, February 11, 13