In this presentation I talk about various topics related to Memory Management in SQL Server such as: 1. Memory Manager: Windows NT a. Virtual memory i. Address Space Layout ii. Virtual Memory Manager iii. 32-bit Virtual Addresses iv. Address Translation b. Memory Pool c. 4GT Tuning i. /3GB Switch (Two slides) ii. Effects of /3GB Tuning iii. /USERVA Switch d. PAE i. Using /3GB & PAE together e. AWE f. 32-bit vs 64-bit Virtual Memory 2. Memory Manager: SQLOS a. SQLOS i. Memory Management ii. Scheduling iii. Exception handling b. NUMA (Non-Uniform Memory Architecture) c. BP and MTL ? d. Memory Types e. Memory Pressure

1. Memory Management in SQL Server
Prashant Kumar
http://SQLactions.com

2. 1. Memory Manager: Windows NT
2. Memory Manager: SQLOS
Customer Service & Support

3. 1. Memory Manager:
Windows NT
• Virtual memory
o Address Space Layout
o Virtual Memory Manager
o 32-bit Virtual Addresses
o Address Translation
• Memory Pool
• 4GT Tuning
o /3GB Switch (Two slides)
o Effects of /3GB Tuning
o /USERVA Switch
• PAE
o Using /3GB & PAE together
• AWE
• 32-bit vs 64-bit Virtual Memory

4. Memory Manager
• As the size of applications and the operating systems
that run grow larger and larger, so do their demands on
memory.
• Consequently, all modern operating systems provide a
form of virtual memory to applications.
• Windows Server 2003 onwards, the applications ported
to it, eventually require even more memory than they
did on the last operating system on which they ran.
• Even applications being written exclusively for Windows
NT will be written with the future in mind and will no
doubt take advantage of all the memory that is
available to them.

5. Virtual Memory
• Address Space Layout
• Virtual-Memory Manager (VMM)
• 32-bit Virtual Addresses
• Address Translation

6. Address Space Layout
• Windows provides a page-based virtual memory management
scheme that allows applications to realize a 32-bit linear address
space for 4 GB of memory.
• Each application can address 2 GB of available memory, regardless
of how much physical memory actually exists.
• Windows employs the PC's hard disk as the memory-backing store,
and has a practical limit imposed only by the available disk space.

7. Virtual Memory Manager
• Enables Windows to provide virtually unlimited memory to the system.
• Is constructed upon a page-based memory management scheme that
divides all of memory into equal chunks called pages.
• Each page is 4096 bytes (4 KB) in size with no discrimination applied as to
how a page is used.
• Everything in Windows—code, data, resources, files, and dynamic
memory—is implemented using pages of physical memory.
• Employs the hard disk to store unutilized pages of memory in one or more
files called pagefiles.
• Pagefiles represent pages of data that are not currently being used, but
may be needed spontaneously at any time.
• Pagefiles in Windows are dynamic in size, allowing them to grow as the
demands for pages of memory grow.
• Is able to make pages of memory available to applications on demand
by swapping pages to and from pagefiles.
• Provides much more virtual memory than the available physical memory.

8. 32-bit Virtual Addresses
• Windows implement a virtual memory system based
on a flat (linear) 32-bit address space.
o 32 bits of address space translates into 4 GB of virtual memory.
• To Windows, the 32-bit virtual address is nothing more
than a placeholder of information used to find the
actual physical address.
o Windows separates each 32-bit virtual address into three groups:
• Two containing 10 bits and one containing 12 bits.
o Each group of bits is then used independently as an offset into a specific
page of memory.

9. Address Translation
• User applications reference
32-bit virtual addresses.
• Using data structures that the
memory manager creates
and maintains, the CPU
translates virtual addresses
into physical addresses.
• Virtual addresses are not
mapped directly to physical
ones.
• Instead, each virtual address
is associated with a system-
space structure called a
page table entry (PTE), which
contains the physical address
to which the virtual one is
mapped.

10. Memory Pools
• Paged Pool
o Is a region of virtual memory in
system space that can be paged in
and out of the system.
o Is accessible from any process
context.
o Device drivers that do not need to
access the memory from
DPC/dispatch level or above can
use paged pool.
• Nonpaged Pool
o Consists of ranges of system virtual
addresses that are guaranteed to
reside in physical memory at all
times.
o Can be accessed at any time (from
any IRQL level and from any
process context) without incurring a
page fault.
o Page faults cannot be satisfied at
DPC/dispatch level or above.

11. 4GT Tuning
• Operating systems based on Windows NT
technology provide applications with a 4 GB virtual
address space.
• The virtual address space is divided such that 2 GB is
available to the application and the other 2 GB is
available only to the system.
• The 4GT RAM Tuning feature:
o Increases the user-mode memory available to the application to 3 GB.
o Reduces the amount available to the system to 1 GB.
• The 4GB Tuning feature is enabled by adding /3GB
Switch to the Boot.ini file.

12. /3GB Switch
• /3GB is a switch used within the Boot.ini to:
o Increase the size of a user process address space from 2 GB to 3GB.
o Reduce the Kernel space from 2 GB to 1 GB.
• The application .exe must be flagged as a 3-GB-
aware application.

13. /3GB Switch

14. Effects of /3GB Tuning
• When you use the /3GB switch, the available
address space is reduced to only 1 GB of operating
system space.
• This 1 GB is reserved to load the operating system
kernel mode modules, all the page table entries,
and the pools.
• As a result, the operating system will have only a
fraction of kernel address space to store PTEs, Pool
Memory and System Cache.
• This could cause potential performance issues for
any kernel-mode component.

15. /USERVA Switch
• Windows 2003 Servers and
Windows XP SP1 onwards, a
new switch /USERVA was
incorporated to work in
conjunction with the /3GB
switch.
• You can use the /USERVA
switch for more precise tuning
of user and kernel virtual
memory space in the
Windows Server 2003 family.
• The use of the /USERVA switch
with the /3GB switch in the
Boot.ini file helps tune the
user-mode space to a value
between 2 GB and 3 GB.
• The difference is given back
to the kernel mode.

17. Summary of 4GT Tuning
• 4GT allows memory intensive Win32 applications to increase its virtual address space beyond
2GB
o Causes kernel virtual address space to be reduced to 1GB
• This feature is enabled by
o On pre-Vista OSs use /3GB switch in boot.ini
• If PAE mode is enabled, access to RAM is limited to 16GB
o On Vista and later OSs use “bcdedit/set IncreaseUserVa3072”
• If PAE mode is enabled, all of 64GB of RAM is accessible
• Only applications linked with /LARGEADDRESSAWARE flag can use this extra virtual address
space
o All other applications can only use 2GB
• /USERVA enables fine tuning of UVAS between 2GB and 3GB
o Used in tandem with the /3GB switch
• Reduces the following kernel virtual address ranges
o Maximum Paged Pool to 256MB
o Maximum Non-Pages Pool to 128MB
• Supported on the following Windows 2003 Server version of the OS
o Standard Edition, Enterprise Edition, DataCenterEdition

18. Physical Address Extension (PAE)
• Physical Address
Extension (PAE) allows
up to 64 GB of physical
memory.
• PAE maps up to 64 GB
of memory into a 32-bit
(4 GB) virtual address
space.
• PAE is supported in:
o Windows XP Professional
o Windows 2000 Advanced
Server and Datacenter Server
o Windows Server 2003, Enterprise
Edition and Datacenter Edition.

19. Using /3GB & /PAE
Together
• Using /3GB and/or /PAE changes the amount of
resources that are dedicated for various operations.
o Using /3GB reduces the amount of nonpaged pool from 256MB to 128MB.
• Using /3GB and /PAE together:
o Provides some customers the most optimum performance for their servers.
o Is required when using applications such as Exchange Server.

20. Address Windowing Extensions (AWE)
• Is a set of APIs that allows an
application to quickly
manipulate physical
memory greater than 4GB.
• Allows applications to
directly address huge
amounts of memory while
continuing to use 32-bit
pointers.
• Allows applications to have
data caches larger than
4GB.
• Uses physical nonpaged
memory and window views
of various portions of this
physical memory within a
32-bit virtual address space.

21. 32-bit vs. 64-bit Virtual
Memory

22. 2. Memory Manager: SQLOS
• NUMA (Non-Uniform Memory Architecture)
• BP and MTL ?
• Memory Types
• Memory Pressure

23. SQLOS
• Memory Management
• Scheduling
• Exception handling

24. Protocols
Query Processor
Database Query
Parser Optimizer SQL Manager
SQLOS API
Manager Executor
Transaction Services Access Methods
Lock Manager
Storage Engine Managers for:
Row Operations
External Components (CLR/MDAC)
Indexes
File Manager Utilities: Pages
Bulk Load DBCC Allocations
Buffer Manager Backup/Restore Versions
SQLOS API
Deadlock
SQLOS Hosting API
Lock
Monitor
Manager
Resource SQLOS I/O
Monitor Synchronization
Buffer
Lazy Pool
Writer Scheduling
Scheduler
Memory Manager
Monitor
= thread

25. “External” Components
• Three external components in SQL address space
o Linked servers
o COM objects (sp_OA*)
o Extended stored procs
• Linked Server and COM objects always use
MemToLeave. Xprocs may use either BP or MTL
depending upon the call to srv_alloc()

26. Memory Access)
Memory Memory
Local Local
Memory Memory
Access Access
CPU 0 CPU 1 CPU 2 CPU 3 CPU … CPU … CPU … CPU n

27. Non-Uniform Memory
Architecture (NUMA)
• Reduces load on memory placed by multiple high speed
CPUs
• Groups of CPUs organized into nodes
• Nodes have local memory, cache and I/O controllers
• Accessing memory in other nodes involves higher latency
• NUMA Ratio is ratio of local vs. remote memory access time
• Cache Coherency maintained by hardware
• BIOS Static Affinity Resource Table (SART) table describes
NUMA layout to Windows
• OS support for NUMA
• Schedules threads within same process on CPUs local to a
node
• Memory allocation requests from local node

28. What is Soft-NUMA
• Activates custom NUMA configuration on
top of any of hardware
• Registry settings control final SoftNUMA
configuration
• Provides greater performance, scalability,
and manageability on SMP as well as on
real NUMA hardware

29. Soft-NUMA Configuration Example
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
Server90NodeConfigurationNode0]
"CPUMask"=dword:0000000F
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
Server90NodeConfigurationNode1]
"CPUMask"=dword:00000030
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
Server90NodeConfigurationNode2]
"CPUMask"=dword:000000C0
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL
ServerMSSQL.1MSSQLServerSuperSocketNetLibTcpIPAll]
"TcpPort"="1433[0x3],2000[0x4]"
"TcpDynamicPorts"=""
"DisplayName"="Any IP Address"

30. BufferPool and MTL?
• How is Buffer Pool size calculated?
• What happens to the rest of the VAS?
• Roughly:
MTL = (MaxWorkerThreads * StackSize) +
DefautReservationSize
Bpool = Min(max server memory,Min(RAM,(VAS-MTL)))

31. Memory Types
• Internal Virtual Memory (VAS)
• External Virtual Memory(physical RAM +Swap)
• Internal Physical Memory(Memory Used by the
application)
• External Physical memory (PhysicalRAM)

32. Memory Pressure
• Internal VAS Pressure
• External VAS Pressure
• Internal Physical Memory Pressure
• External Physical memory Pressure

33. Internal Virtual Memory
Pressure
• Running low on VAS due to fragmentation (a lot of
VAS is available but in small blocks) and/or
consumption (direct allocations, DLLs loaded in
SQL Server VAS, high number of threads).
• SQL Server detects this condition and may release
reserved regions of VAS, reduce buffer pool commit
target, and start shrinking caches.

34. Internal Virtual Memory
Pressure
Troubleshooting
• Identify major memory consumers inside SQL Server.
• Verify server configuration.
• Further actions depend on the investigation: check
for workload; possible design issues; other resource
bottlenecks.

35. External Virtual Memory
Pressure
• Running low on space in the system page file(s). This
may cause the system to fail memory allocations, as
it is unable to page out currently allocated
memory. This condition may result in the whole
system responding very slowly or even bring it to a
halt.

37. External Virtual Memory
Pressure
Troubleshooting
• Increase swap file size.
• Check for major physical memory consumers
• Find major system memory consumers.
• Attempt to eliminate (if possible).

38. Internal Physical Memory
Pressure
• SQL Server detects high memory consumption
internally, causing redistribution of memory between
internal components.
• Internal memory pressure may be a result of:
• Responding to the external memory pressure (SQL Server sets
lower memory usage caps).
• Changed memory settings (e.g. ‘max server memory’).
• Changes in memory distribution of internal
components (due to high percentage of reserved
and stolen pages from the buffer pool).

39. Internal Physical Memory
Pressure
Troubleshooting
• Identify major memory consumers inside SQL Server.
• Verify server configuration.
• Further actions depend on the investigation: check
for workload; possible design issues; other resource
bottlenecks

40. External Physical Memory
Pressure
• Physical memory (RAM) running low. This causes the
system to trim working sets of currently running
processes, which may result in overall slowdown.
• SQL Server detects this condition and, depending
on the configuration, may reduce the commit
target of the buffer pool and start clearing internal
caches.

41. External Physical Memory
Pressure
Troubleshooting
• Find major system memory consumers.
• Attempt to eliminate (if possible).
• Check for adequate system RAM and consider
adding more RAM

42. Resources
• Blogs
o http://blogs.msdn.com/slavao
o http://blogs.msdn.com/psssql/archive/2007/10/18/do-i-have-to-assign-the-lock-privilege-for-local-
system.aspx
o http://blogs.msdn.com/slavao/archive/2005/02/19/376714.aspx
o http://blogs.msdn.com/slavao/archive/2005/08/18/453354.aspx
o http://blogs.msdn.com/slavao/articles/441058.aspx
o http://blogs.msdn.com/sqlprogrammability/
• External Links:
o http://lse.sourceforge.net/numa/faq
o http://www.modhul.com/2007/11/10/optimising-system-memory-for-sql-server-part-i/
o http://sqlserver.ro/blogs/cristians_blog/archive/2007/10/02/the-return-of-the-ring-buffers.aspx
o http://lse.sourceforge.net/numa/faq/system_descriptions.html
o http://lse.sourceforge.net/numa/faq/
• Books:
o Eldad Eilam: Reversing: Secrets of Reverse Engineering
o Ken Henderson: SQL Server 2005 Practical Troubleshooting The Database Engine
o Kalen Delaney: Inside Microsoft SQL Server 2005 The Storage Engine
• KBAs:
o http://support.microsoft.com/kb/918483
o http://support.microsoft.com//kb/271624
o http://support.microsoft.com//kb/907877

43. ?

44. Thank you!