MySQL Conference Presentation on the benefits and drawbacks on partitioning

1. Divide And Be Conquered? 04/23/2009 Brooks Johnson [email_address]

3. Often degrades performance

4. In theory could improve concurrency

5. First cover InnoDB

6. Then MyISAM

7. Finish up with admin improvements

8. No Partition - Classic Primary Key CREATE TABLE `test`.`SaleO` ( `orderId` int(11) NOT NULL, `customerId` int(11) NOT NULL, `productId` int(11) NOT NULL, `productBigId` int(11) NOT NULL, `unit` int(11) NOT NULL, `purchaseAmount` decimal(16,2) NOT NULL, `purchaseCost` decimal(16,2) NOT NULL, `purchaseDate` datetime NOT NULL, PRIMARY KEY (`orderId`), KEY `idx_sale_purchasedate` (`purchaseDate`), KEY `idx_sale_product` (`productId`), KEY `idx_sale_customer` (`customerId`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

9. Partitioning by Date (Month) CREATE TABLE `test`.`SaleP` ( `orderId` int(11) NOT NULL, `customerId` int(11) NOT NULL, `productId` int(11) NOT NULL, `productBigId` int(11) NOT NULL, `unit` int(11) NOT NULL, `purchaseAmount` decimal(16,2) NOT NULL, `purchaseCost` decimal(16,2) NOT NULL, `purchaseDate` datetime NOT NULL, PRIMARY KEY (`purchaseDate`,`orderId`), KEY `idx_sale_product` (`productId`), KEY `idx_sale_order` (`orderId`), KEY `idx_SaleP_orderId` (`orderId`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 PARTITION BY RANGE (to_days(purchaseDate)) (PARTITION p0 VALUES LESS THAN (730882) ENGINE = InnoDB, PARTITION p1 VALUES LESS THAN (730910) ENGINE = InnoDB, PARTITION p2 VALUES LESS THAN (730941) ENGINE = InnoDB, PARTITION p3 VALUES LESS THAN (730971) ENGINE = InnoDB, PARTITION p4 VALUES LESS THAN (731002) ENGINE = InnoDB, PARTITION p5 VALUES LESS THAN (731032) ENGINE = InnoDB, PARTITION p6 VALUES LESS THAN (731063) ENGINE = InnoDB, PARTITION p7 VALUES LESS THAN (731094) ENGINE = InnoDB, PARTITION p8 VALUES LESS THAN (731124) ENGINE = InnoDB, PARTITION p9 VALUES LESS THAN (731155) ENGINE = InnoDB, PARTITION p10 VALUES LESS THAN (731185) ENGINE = InnoDB, PARTITION p11 VALUES LESS THAN (731216) ENGINE = InnoDB, PARTITION p12 VALUES LESS THAN MAXVALUE ENGINE = InnoDB)

10. Partition by Order CREATE TABLE `test`.`SaleOPO` ( `orderId` int(11) NOT NULL, `customerId` int(11) NOT NULL, `productId` int(11) NOT NULL, `productBigId` int(11) NOT NULL, `unit` int(11) NOT NULL, `purchaseAmount` decimal(16,2) NOT NULL, `purchaseCost` decimal(16,2) NOT NULL, `purchaseDate` datetime NOT NULL, PRIMARY KEY (`orderId`), KEY `idx_sale_purchasedate` (`purchaseDate`), KEY `idx_sale_product` (`productId`), KEY `idx_sale_customer` (`customerId`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 PARTITION BY HASH (orderID) PARTITIONS 12

13. Performance improvement due to clustering by date, not partitioning

15. A bit longer than the non-partitioned table (34 seconds)

16. Partitioning might have caused a small overhead in this case

18. One process randomly selecting 10,000 orders

20. Over twice as long as the non-partitioned table (55 seconds)

21. Non-Partitioned Index

22. Partitioned Index

24. Partitioning might result in small overhead for each query

26. Non-partitioned – 173 seconds

27. Partitioned by date – 300 seconds

28. All the data was added to the last date partition

29. Partitioned by order – 237 seconds

30. Data was added to all 12 partitions

31. Partitioning “seemed” to add overhead

32. MyISAM without Partitioning CREATE TABLE `test`.`SaleI` ( `orderId` int(11) NOT NULL AUTO_INCREMENT, `customerId` int(11) NOT NULL, `productId` int(11) NOT NULL, `productBigId` int(11) NOT NULL, `unit` int(11) NOT NULL, `purchaseAmount` decimal(16,2) NOT NULL, `purchaseCost` decimal(16,2) NOT NULL, `purchaseDate` datetime NOT NULL, PRIMARY KEY (`orderId`), KEY `idx_sale_product` (`productId`), KEY `idx_sale_customer` (`customerId`), KEY `idx_SaleI_purchaseDate` (`purchaseDate`) ) ENGINE=MyISAM AUTO_INCREMENT=121900002 DEFAULT CHARSET=utf8;

33. MyISAM Partitioning by Date CREATE TABLE `test`.`SaleIP` ( `orderId` int(11) NOT NULL AUTO_INCREMENT, `customerId` int(11) NOT NULL, `productId` int(11) NOT NULL, `productBigId` int(11) NOT NULL, `unit` int(11) NOT NULL, `purchaseAmount` decimal(16,2) NOT NULL, `purchaseCost` decimal(16,2) NOT NULL, `purchaseDate` datetime NOT NULL, PRIMARY KEY (`purchaseDate`,`orderId`), KEY `idx_sale_order` (`orderId`), KEY `idx_sale_product` (`productId`), KEY `idx_saleIP_customer` (`customerId`) ) ENGINE=MyISAM AUTO_INCREMENT=122200002 DEFAULT CHARSET=utf8 PARTITION BY RANGE (to_days(purchaseDate)) (PARTITION p0 VALUES LESS THAN (730882) ENGINE = MyISAM, PARTITION p1 VALUES LESS THAN (730910) ENGINE = MyISAM, PARTITION p2 VALUES LESS THAN (730941) ENGINE = MyISAM, PARTITION p3 VALUES LESS THAN (730971) ENGINE = MyISAM, PARTITION p4 VALUES LESS THAN (731002) ENGINE = MyISAM, PARTITION p5 VALUES LESS THAN (731032) ENGINE = MyISAM, PARTITION p6 VALUES LESS THAN (731063) ENGINE = MyISAM, PARTITION p7 VALUES LESS THAN (731094) ENGINE = MyISAM, PARTITION p8 VALUES LESS THAN (731124) ENGINE = MyISAM, PARTITION p9 VALUES LESS THAN (731155) ENGINE = MyISAM, PARTITION p10 VALUES LESS THAN (731185) ENGINE = MyISAM, PARTITION p11 VALUES LESS THAN (731216) ENGINE = MyISAM, PARTITION p12 VALUES LESS THAN MAXVALUE ENGINE = MyISAM)

34. MyISAM Order Partitioned CREATE TABLE `test`.`SaleIPO` ( `orderId` int(11) NOT NULL AUTO_INCREMENT, `customerId` int(11) NOT NULL, `productId` int(11) NOT NULL, `productBigId` int(11) NOT NULL, `unit` int(11) NOT NULL, `purchaseAmount` decimal(16,2) NOT NULL, `purchaseCost` decimal(16,2) NOT NULL, `purchaseDate` datetime NOT NULL, PRIMARY KEY (`orderId`), KEY `idx_sale_purchaseDate` (`purchaseDate`), KEY `idx_sale_product` (`productId`), KEY `idx_sale_customer` (`customerId`) ) ENGINE=MyISAM AUTO_INCREMENT=120900003 DEFAULT CHARSET=utf8 PARTITION BY HASH (orderID) PARTITIONS 12

37. Real improvement

39. No difference from non-partitioned

41. One process selecting 10,000 random orders

43. A bit worse than non-partitioned, but not that bad

45. Worse than non-partitioned (96 non-partitioned)

46. Better than date partitioned (111 seconds)

47. Partitioning added overhead again

49. Non-partitioned table takes 702 seconds

50. Date partitioned table takes 47 seconds

51. All the data was added to the last date partition

52. Order partitioned table takes 974 seconds

53. The data was added to all 12 orderId partitions

54. Date partitioned table was the fastest by far

55. So, real concurrency improvement in this case

57. No longer use “OPTIMIZE TABLE SaleP”

58. Instead use “ALTER TABLE SaleP OPTIMIZE PARTITION p12”

59. Optimizing just the most recent partition can be over an order of magnitude faster than a full table optimization

60. One partition is far easier to fit into memory and much less data to sort

61. Dropping a partition is much faster than deleting rows

62. Partitioning or Disk Striping Partitioning on Different Disks

64. Can improve performance

65. Can degrade performance

66. Will improve administrative tasks

67. Performance depends on what is partitioned

68. Performance also depends on data distribution

69. Still more to learn

70. Any questions?