The document discusses optimizing ORM framework performance configurations using a genetic algorithm. It finds that the default configurations are often significantly slower than optimal configurations. A genetic algorithm can guide the optimization of configurations across objectives like CPU usage, memory usage, and execution time. Experiments show the genetic algorithm finds configurations close to optimal in 5-20 minutes, much faster than trial-and-error. Developers should care about ORM performance configuration due to potential performance degradations from defaults.

1. Optimizing the Performance-RelatedOptimizing the Performance-Related
Configurations of ORM FrameworksConfigurations of ORM Frameworks
Ravjot Singh, Cor-Paul Bezemer,Weiyi Shang,Ahmed E. Hassan
Using a Multi-Objective Genetic AlgorithmUsing a Multi-Objective Genetic Algorithm

2. 2
Configuration is DIFFICULTConfiguration is DIFFICULT

3. 3
27% of customer issues are27% of customer issues are
due to misconfigurationdue to misconfiguration

4. 4
20% of misconfigurations cause20% of misconfigurations cause
performance degradationsperformance degradations

5. 5
A popular online store estimates a $1.6A popular online store estimates a $1.6
billion loss for a 1-second slowdownbillion loss for a 1-second slowdown

6. 6
Databases are at the coreDatabases are at the core
of many large systemsof many large systems

7. 7
Software developers knowSoftware developers know
how to think in objects...how to think in objects...

8. 8
But have less experienceBut have less experience
with databaseswith databases

9. 9
Object-relational mappingObject-relational mapping
(ORM) helps developers(ORM) helps developers
CODE

10. 10
Code without ORM isCode without ORM is
tedioustedious
public class Person {
// ...
public String getName(){
sql = “SELECT name FROM … WHERE …”;
ResultSet rs = stmt.executeQuery(sql);
while(rs.next())
return rs.getString(“name”);
}
}

11. 11
Code with ORM is cleanCode with ORM is clean
@Entity
public class Person {
@Id Integer getId() { ... }
public String getName(){
return this.name;
}
}

12. 12
Several popular ORMSeveral popular ORM
frameworks existframeworks exist

13. 13
ORM has many configurationORM has many configuration
optionsoptions

14. 14
ORM configuration is…ORM configuration is…
guess what?guess what?

15. 15
ORM configuration is…ORM configuration is…
guess what? Difficult!guess what? Difficult!

16. 16
The impact of one ORMThe impact of one ORM
configuration optionconfiguration option
hibernate.max_fetch_depth = {0|1|2|3|...}

17. 17
The impact of one ORMThe impact of one ORM
configuration optionconfiguration option
hibernate.max_fetch_depth = {0|1|2|3|...}
Optimal configuration depends on what you need!

18. 18
Should we care about ORMShould we care about ORM
performance configuration?performance configuration?
Analyze 11 boolean configuration options
hence 211
= 2048 configurations

19. 19
We compare the 'default' withWe compare the 'default' with
the optimal configurationthe optimal configuration
Default configuration
System Tests
Run tests
Default
execution
time

20. 20
We compare the 'default' withWe compare the 'default' with
the optimal configurationthe optimal configuration
Default configuration
System Tests
Run tests
Optimal configuration
System Tests
Default
execution
time
Optimal
execution
time
Run tests

21. 21
We compare the 'default' withWe compare the 'default' with
the optimal configurationthe optimal configuration
Default configuration
System Tests
Run tests
Optimal configuration
System Tests
Compare
Default
execution
time
Optimal
execution
time
Run tests

22. 22
Yes, we should care about ORMYes, we should care about ORM
performance configuration!performance configuration!
89% 96%
of the test cases are significantly
slower using default!

23. 23
How can we guide ORMHow can we guide ORM
performance configuration?performance configuration?

24. 24
Optimize ORM configurationOptimize ORM configuration
by trial-and-errorby trial-and-error
Randomly select
configuration A

25. 25
Optimize ORM configurationOptimize ORM configuration
by trial-and-errorby trial-and-error
Evaluate configuration A
on some workload W
Randomly select
configuration A

26. 26
Optimize ORM configurationOptimize ORM configuration
by trial-and-errorby trial-and-error
Evaluate configuration A
on some workload W
Compare execution time for configuration A
with execution time for current best configuration
Randomly select
configuration A

27. 27
Optimize ORM configurationOptimize ORM configuration
by trial-and-errorby trial-and-error
Evaluate configuration A
on some workload W
Compare execution time for configuration A
with execution time for current best configuration
Randomly select
configuration A
Use best configuration

28. 28
Optimize ORM configurationOptimize ORM configuration
by trial-and-errorby trial-and-error
Evaluate configuration A
on some workload W
Compare execution time for configuration A
with execution time for current best configuration
Randomly select
configuration A
Use best configuration

29. 29
An evolutionary/genetic algorithmAn evolutionary/genetic algorithm
guides this trial-and-error processguides this trial-and-error process

30. 30
Genetic algorithm conceptGenetic algorithm concept
Optimize population based on
one or more objectives
We start from the 'default'
configuration as supplied by
the developer

31. 31
Genetic algorithm conceptGenetic algorithm concept
1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1+

32. 32
Genetic algorithm conceptGenetic algorithm concept
1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1
1 1 0 0 0 0 0 1
+

33. 33
Genetic algorithm conceptGenetic algorithm concept
1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1
1 1 0 0 0 0 0 1
1 1 0 0 0 1 0 1
+

34. 34
Genetic algorithm conceptGenetic algorithm concept
1 1 0 0 0 0 0 0
1 0 0 1 0 0 0 1
1 1 0 0 0 1 0 1
Execution time = 50s
Execution time = 60s
Execution time = 30s

35. 35
Genetic algorithm conceptGenetic algorithm concept
1 1 0 0 0 0 0 0
1 0 0 1 0 0 0 1
1 1 0 0 0 1 0 1
Execution time = 50s
Execution time = 60s
Execution time = 30s

36. 36
Optimizing ORMOptimizing ORM
performance configurationperformance configuration
Objectives:
CPU
Memory
Execution time

37. 37
Optimizing ORMOptimizing ORM
performance configurationperformance configuration
Objectives:
CPU
Memory
Execution time
Calculate Δ with
default configuration

38. 38
Evaluation of our approachEvaluation of our approach
1. Closeness of configurations found by
our approach to the optimal
configuration
2. Speed with which we can find sub-
optimal configurations

39. 39
ExperimentsExperiments
Default configuration
Tests
Repeat 100 times
Optimal configuration

40. 40
Closeness of configurationsCloseness of configurations
to the optimalto the optimal
We rank all existing configurations based on
dominance
Configuration A dominates configuration B if:
1. B is not better than A for all objectives
2.A is better than B for at least one objective

41. 41
Ranking of configurationsRanking of configurations
Configuration CPU usage Memory
usage
Execution time
Default 0 0 0
A +50% +50% 0
B -80% -20% 0
C 0 -80% -20%

42. 42
Ranking of configurationsRanking of configurations
Configuration CPU usage Memory
usage
Execution time
Default 0 0 0
A +50% +50% 0
B -80% -20% 0
C 0 -80% -20%
Rank 1: B and C
Rank 2: default
Rank 3:A

43. 43
The genetic algorithm findsThe genetic algorithm finds
configurations close to the best rankconfigurations close to the best rank
BEST WORST

44. 44
Speed with whichSpeed with which
configurations are foundconfigurations are found
Depends on:
- Workload
- Application
- Data

45. 45
The genetic algorithm findsThe genetic algorithm finds
configurations fast (~5-20 minutes)configurations fast (~5-20 minutes)

46. 46
Yes, we should care about ORMYes, we should care about ORM
performance configuration!performance configuration!
89% 96%
of the test cases are significantly
slower using default!

47. 47

48. 48
How can we guide ORMHow can we guide ORM
performance configuration?performance configuration?

49. 49

50. 50
The genetic algorithm findsThe genetic algorithm finds
configurations close to the optimal rankconfigurations close to the optimal rank

51. 51

52. 52
The genetic algorithm findsThe genetic algorithm finds
configurations fast (~5-20 minutes)configurations fast (~5-20 minutes)

53. 53

54. 54
Cor-Paul Bezemer
bezemer@cs.queensu.ca
http://sailhome.cs.queensu.ca/~corpaul/

55. 55

56. 56
Studied configuration optionsStudied configuration options
order_updates
jdbc.batch_size
order_inserts
connection.release_mode
default_batch_fetch_size
jdbc.batch_versioned_data
max_fetch_depth
id.new_generator_mappings
jdbc.fetch_size
bytecode.use_reflection_optimizer
cache.use_second_level_cache

57. 57
Yes, we should care about ORMYes, we should care about ORM
performance configuration!performance configuration!