1.
Transparent Hugepages in RHEL 6 Raghu Udiyar 22 July 2011

2.
Transparent Huge Pages <ul><li>Written by Andrea Arcangeli (Red Hat kernel developer)

3.
Merged into upstream kernel 2.6.38

4.
Available in RHEL 6.0 on GA </li></ul>

5.
<ul><li>A page is the smallest unit of memory that can be manipulated

6.
Memory is divided into pages of equal size </li></ul>Transparent Huge Pages Pages Memory

7.
Page : Memory Management <ul><li>Page size varies with the architecture, x86 = 4KiB

8.
For other arch's : ia64 = 8KiB, ppc64 = 4,8 and 64KiB </li></ul>4KiB Page x86

9.
Page : Memory Management Physical Memory <ul><li>Aka : RAM – memory installed on the system e.g. 2GiB

10.
Programs never work directly with physical memory

11.
A physical page is called a Page Frame </li></ul>Virtual Memory <ul><li>Not real memory : 32bit – 4Gb, 64bit – more then 128T

12.
Programs always allocate virtual memory

13.
A virtual page is called simply a Page </li></ul>

14.
Page : Virtual and Physical <ul><li>Each page is mapped to a page frame using tables </li></ul>Physical Virtual Page Page Frame

15.
Page : Memory Management Process 2 Process 1

16.
Page : Memory Management <ul><li>Virtual to Physical address translations are stored in page tables

17.
These page tables are stored in memory

18.
Looking up these page tables is a processes called a

19.
Page Walk </li></ul>

20.
Page : Simplified view of the Page Table Virtual address <ul>Multiple tables are involved in a Page Walk </ul>

21.
Page : Memory Management <ul><li>Every memory access requires a Page Walk

22.
Hardware assisted but it is still expensive

23.
Typically, takes 10-100 cpu clock cycles

24.
The obvious optimization is to cache the results </li></ul>

25.
Page : TLB - Translation look-aside buffer TLB caches Virtual->Physical Translations <ul><ul><li>Check TLB first

26.
If TLB hit, return address

27.
If TLB miss, do a Page Walk and store translation in TLB </li></ul></ul><ul><ul><li>Using TLB, increases performance by as much as 15%

28.
Look up takes 0.5 – 1 clock cycl e </li></ul></ul>

29.
Page : TLB - Translation look-aside buffer <ul><li>CAM (content addressable memory) is used that is quite expensive to manufacture and have other limitations

30.
But they are very fast

31.
Typically, a TLB can only hold 8 to 1024 entries

32.
Plus, TLB cache has to be flushed on every context switch

33.
Therefore, TLB is a scarce resource and every attempt has to be made to maximize its usage </li></ul>

34.
Page : Large Memory Applications <ul><li>Applications touching greater then 10Gb

35.
4KiB page size produces a large overhead to manage these pages

36.
Example :

37.
17GiB Oracle SGA with 1000 Connection : </li><ul><li>~4.4 million pages

38.
Page table size : 16GiB </li></ul><li>TLB can only hold at most 1024 entries

39.
BIG performance penalty </li></ul>

40.
Page : Questions? <ul><li>Q1 : How can page walk work, if we don't know where the page tables are (as they are stored in memory) ?

41.
Ans : The address of the root table is stored in a special CR3 register. </li></ul>

42.
<ul>Huge Pages are pages of larger sizes e.g. on x86 it's 2MiB </ul>Transparent Huge Pages X 512 4 KiB 2 MiB

43.
Huge Pages : Benefits <ul><li>Now each TLB entry can effectively store 512 normal pages

44.
Therefore, Increases TLB hit ratio

45.
Decrease in number of pages to manage

46.
Previous Oracle example :

47.
Using huge pages : </li><ul><li>17GiB SGA uses 17*1024/2 = 8704 Huge Pages (compared to ~4.4 million! Normal pages)

48.
Page Table size? ~33MiB (1000 processes)

49.
(compared to 16GiB) </li></ul><li>Huge improvement! </li></ul>

50.
Huge Pages : Configuration <ul><li>Set “vm.nr_hugepages = 10” in /etc/sysctl.conf

51.
To verify :

52.
# cat /proc/meminfo | grep Huge

53.
HugePages_Total: 10

54.
HugePages_Free: 10

55.
HugePages_Rsvd: 0

56.
Hugepagesize: 2048 kB

57.
A reboot may be required if not enough contiguous physical pages are available.

58.
Refer to : https://access.redhat.com/kb/docs/DOC-56553 </li></ul>

59.
Huge Pages : Issues <ul><li>Huge pages have to be reserved beforehand

60.
Not available for normal applications

61.
Explicit application support is required

62.
Cannot be swapped to disk – locked in memory

63.
Only really used by large database applications, such as Oracle and Sybase </li></ul>

64.
Huge Pages : Questions? <ul><li>Why can't we have all normal pages as huge pages i.e. Increase from 4KiB to 2MiB

65.
Ans : This causes fragmentation. </li><ul><ul><li>Small page size causes large management overhead

66.
Large page size causes internal and external fragmentation </li></ul></ul><li>What we need is a best of both worlds </li></ul>

67.
<ul><li>This builds up on the limitations of normal Huge pages

68.
THP is enabled by default for all applications

69.
Transparent to the application – No configuration required

70.
All applications benefit from using Huge pages

71.
Up to 10% performance improvement

72.
Therefore, we successfully have a mixed or hybrid page size scenario </li></ul>Transparent Huge Pages

73.
THP: How does it work? <ul><li>Hugepages are allocated whenever possible

74.
Example : 2MiB memory requests

75.
THP are swappable, by breaking a hugepage to 4KiB pages

76.
A kernel thread Khugepaged will scan running processes and attempt to switch normal pages with Hugepages

77.
THP tries to maximizes Huge page use

78.
Kernel itself maps it's memory using hugepages </li></ul>

79.
THP: Work in Progress <ul><li>Only enabled for anonymous memory, for now

80.
Page cache and shared memory still in progress

81.
To check THP usage on RHEL 6 or F14 and above :

82.
# grep AnonHugePages /proc/meminfo

83.
AnonHugePages: 632832 kB </li></ul>

84.
THP : Questions? <ul><li>Refer for more details :

85.
https://access.redhat.com/kb/docs/DOC-49562 </li></ul>

86.
Thank You Raghu Udiyar 22 July 2011