Sometimes your fastest queries can cause the most problems. I will take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby and Rails to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

1. @molly_struve

2. @molly_struve

3. @molly_struve

4. @molly_struve

5. @molly_struve

6. @molly_struve

7. @molly_struve

8. @molly_struve
Adding Indexes
Using SELECT statements
Batch Processing

9. @molly_struve
Elasticsearch::Transport::Errors::GatewayTimeout 504
{
"statusCode": 200,
"took": "100ms"
}

10. @molly_struve
Resque

11. @molly_struve

12. @molly_struve

13. @molly_struve

14. @molly_struve

15. @molly_struve
The average company has...
60 thousand
assets
24 million
vulnerabilities

16. @molly_struve
MySQL
Elasticsearch
Cluster

17. @molly_struve

18. @molly_struve
MySQL
Elasticsearch
Cluster
ActiveModelSerializers

19. @molly_struve
class Vulnerability < ActiveModel::Serializer
attributes :id, :client_id, :created_at, :updated_at,
:priority, :details, :notes, :asset_id,
:solution_id, :owner_id, :ticket_id
end

20. @molly_struve
200 MILLION

21. @molly_struve

22. @molly_struve

23. @molly_struve

24. @molly_struve
(1.6ms)
(0.9ms)
(4.1ms)(5.2ms)
(5.2ms)
(1.3ms)
(3.1ms)
(2.9ms)
(2.2ms)
(4.9ms)
(6.0ms)
(0.3ms)
(1.6ms)
(0.9ms)
(2.2ms)
(3.0ms)
(2.1ms)
(1.3ms)
(2.1ms)
(8.1ms)
(1.4ms)

25. @molly_struve
MySQL

26. @molly_struve

27. @molly_struve
class BulkVulnerabilityCache
attr_accessor :vulnerabilities, :client, :vulnerability_ids
def initialize(vulns, client)
self.vulnerabilities = vulns
self.vulnerability_ids = vulns.map(&:id)
self.client = client
end
# MySQL Lookups
end

28. @molly_struve
module Serializers
class Vulnerability
attr_accessor :vulnerability, :cache
def initialize(vuln, bulk_cache)
self.cache = bulk_cache
self.vulnerability = vuln
end
end
end
self.cache = bulk_cache

29. @molly_struve
class Vulnerability
has_many :custom_fields
end

30. @molly_struve
CustomField.where(:vulnerability_id => vuln.id)
cache.fetch('custom_fields', vuln.id)

31. @molly_struve
(pry)> vulns = Vulnerability.limit(300);
(pry)> Benchmark.realtime { vulns.each(&:serialize) }
=> 6.022452222998254
(pry)> Benchmark.realtime do
> BulkVulnerability.new(vulns, [], client).serialize
> end
=> 0.7267019419959979

32. @molly_struve
Individual Serialization:
Bulk Serialization:
2,100
7

33. @molly_struve
1k vulns 1k vulns 1k vulns

34. @molly_struve
1k vulns 1k vulns 1k vulns
7k 7

35. @molly_struve

36. @molly_struve

37. Process
in Bulk

38. @molly_struve

39. @molly_struve

40. @molly_struve
Elasticsearch
Cluster
+
Redis
MySQL
Vulnerabilities

41. @molly_struve
Redis.get
Client 1
Index
Client 2
Index
Client 3 & 4
Index

42. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

43. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

44. @molly_struve
(pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”)
DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234"
DEBUG -- : [Redis] call_time=1.07 ms
GET

45. @molly_struve

46. @molly_struve

47. @molly_struve
client_indexes = Hash.new do |h, client_id|
h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”)
end

48. @molly_struve
indexing_hashes = vuln_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{client_id}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end
client_indexes[hash[:client_id]],

49. @molly_struve
1 + 1 + 1
Client 1 Client 2 Client 3
1k 1k 1k

50. @molly_struve

51. @molly_struve
65% job
speed up

52. @molly_struve

53. @molly_struve

54. Process
in Bulk
Hash
Cache

55. @molly_struve
CLIENT 1
CLIENT 2
CLIENT 3

56. @molly_struve
Asset.using(:shard_1).find(1)

57. @molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}

58. @molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
sharding_config_hash[client_id]

59. @molly_struve

60. @molly_struve
20 bytes 1kb 13kb

61. @molly_struve

62. @molly_struve

63. @molly_struve
ActiveRecord::Base.connection

64. @molly_struve
(pry)> ActiveRecord::Base.connection
=> #<Octopus::Proxy:0x000055b38c697d10
@proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

65. @molly_struve

66. @molly_struve
module Octopus
class Proxy
attr_accessor :proxy_config
delegate :current_shard, :current_shard=,
:current_slave_group, :current_slave_group=,
:shard_names, :shards_for_group, :shards, :sharded,
:config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
end
end

67. @molly_struve

68. Process
in Bulk
Hash
Cache
Active
Record
Cache

69. @molly_struve

70. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

71. @molly_struve

72. @molly_struve
(pry)> User.where(:id => [])
User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0
=> []

73. @molly_struve

74. @molly_struve
return unless user_ids.any?
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

75. @molly_struve
(pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
(pry)> Benchmark.realtime do
> 10_000.times do
> next unless ids.any?
> User.where(:id => [])
> end
> end
=> 0.0006368421018123627

76. @molly_struve
(pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014

77. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

78. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end
users = User.where(:id => user_ids).active.short.single

79. @molly_struve
.none

80. @molly_struve
(pry)> User.where(:id => []).active.tall.single
User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND
`users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1
=> []
(pry)> User.none.active.tall.single
=> []

81. @molly_struve

82. @molly_struve

83. @molly_struve
pry(main)> Rails.logger.level = 0
$ redis-cli monitor > commands-redis-2018-10-01.txt
pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

84. @molly_struve

85. @molly_struve

86. @molly_struve

87. @molly_struve
(pry)> Report.zero_assets.count
=> 10805
(pry)> Report.active.count
=> 25842
(pry)> Report.average_asset_count
=> 1657
Investigating Existing Reports

88. @molly_struve

89. @molly_struve

90. @molly_struve
return if report.asset_count.zero?

91. @molly_struve

92. @molly_struve

93. Process
in Bulk
Hash
Cache
Database
Guards
Active
Record
Cache

94. @molly_struve
Resque Workers
Redis

95. @molly_struve
45 workers
45 workers
45 workers

96. @molly_struve
70 workers
70 workers
70 workers

97. @molly_struve

98. @molly_struve

99. @molly_struve

100. @molly_struve
48 MB
16 MB

101. @molly_struve
70 workers
100k
200k

102. @molly_struve

103. @molly_struve

104. @molly_struve
Resque
Throttling

105. @molly_struve
100k
200k

106. @molly_struve
48MB
16MB

107. @molly_struve

108. Process
in Bulk
Hash
Cache
Database
Guards
Remove
Datastore
Hits
Active
Record
Cache

109. @molly_struve

110. @molly_struve

111. @molly_struve

112. @molly_struve
https://www.linkedin.com/in/mollystruve/
https://github.com/mstruve
@molly_struve
molly.struve@gmail.com