Teradata Memory Management -- A balancing Act

1. Memory Management – A Balancing Act
Shaheryar Iqbal
Best viewed in Microsoft PowerPoint

2. Agenda
Node Memory
Between OS and FSG Cache
Memory Partitioning FSG Cache Percent
Free Memory
OS Managed Memory
FSG Cache.
PDE Managed Memory
Free memory availability
Monitoring memory
Memory allocation failures
Paging/Swapping.
Adjusting FSG Cache
Cache Effectiveness Rate
DBS Cache Threshold
Hash Join - HtMemAlloc
Redistribution Buffer
2 > 1 November 2012

3. Node Memory
4GB maximum is a limitation
Memory on hand of the current 32 bit OS.
64 bit Windows and LINUX
systems are shipping with
6GB (54XX) or 8GB (5500H/C).
# rallsh –sv memsize –k
How much mem on each Node
Node Memory is partitioned into
•OS Managed Memory and
•FSG Cache.
3 > 1 November 2012

4. Memory Partitioning between OS and FSG Cache
Free Memory
O.S. Overhead 100 MB FSG (File Segment Cache)
13 Vprocs @ 40 MB each
Remaining space 3476 MB available for FSG
1 PDE
2 PE
10 AMPs
13 Vprocs @ 40 MB
= 520 MB `
Total Free Memory
40 MB 40 MB 40 MB 40 MB 40 MB 40 MB 40 MB
= 100 + 520
= 620 MB PE-1 PE-2 AMP-1 AMP-2 AMP-3 AMP-4 ….. AMP-10
Vproc Vproc Vproc Vproc Vproc Vproc Vproc
FSG Cache
Total Memory 4096 MB
– 100 MB PDE (BaseVproc) – 40 MB
– 520 MB
FSG Cache 3476 MB Operating System – 100 MB
Ex. 4096 MB – 4 GB Memory
4 > 1 November 2012

5. What’s FSG CACHE PERCENT
FSG Cache An xctl DBS Control Performance field
Percent Reduces the amount of memory
to be used for FSG Cache.
The remaining memory is given
back to UNIX
5 > 1 November 2012

6. FSG Cache Percent (Contd)
Free Memory
• O.S. 100 MB
FSG Cache FSG (File Segment Cache)
• 13 Vprocs @ 40 MB
= 4096
– ( (10+2+1) * 40MB
= 520 MB) 80% of remaining space – 2780 MB available for FSG
• 20% of–remaining space
100 MB
= 3746 MB
returned = 696 MB example of
Let’s take an
Adjusted FSG Size = 80
FSG Cache Percent
Total Free i.e. the 3476 MB
Memory
= 80% x 20 % space will
= 100 + 520 + 696 back to OS.
= be returned
2780 MB
= 1316 MB 20% of remaining space – 696 MB returned back to O.S/Free Mem
`
Added to UNIX
= 20% x 3746 MB
FSG Cache
= 696 MB 40 MB 40 MB 40 MB 40 MB 40 MB 40 MB 40 MB
Total Memory 4096 MB
– 696 MB PE-1 PE-2 AMP-1 AMP-2 AMP-3 AMP-4 ….. AMP-10
– 100 MB Vproc Vproc Vproc Vproc Vproc Vproc Vproc
– 520 MB
FSG Cache = 2780 MB
PDE (BaseVproc) – 40 MB
Operating System – 100 MB
Ex. 4096 MB – 4 GB Memory
6 > 1 November 2012

7. What’s in Free Memory
Memory managed by the O.S.
Free Memory
TERADATA VPROCS
ADMINISTRATIVE PROGRAMS:
AMP worker tasks,
PE tasks –
Dictionary cache
Request-to-Steps
cache,
Vprocs Communication
kernel resources,
program text and data,
Redistribution Buffers,
Aggregation Buffers,
Hash Join Join Buffers,
other applications.
7 > 1 November 2012

8. What’s in FSGcache
FSG Cache
Memory managed by the PDE.
FSG Cache is used by:
FSG Cache is used for:
•AMPs
•Backup Activity
Used to access
database segments.
Permanent Data blocks
Full Cylinder read slots
(CI) for Permanent Data blocks
Spool CIs & Data blocks
Transient Journal data (TJs)
Permanent Journal data (PJs)
Sync Scan data blocks
Buddy Backup data blocks
Hash Join Spool Buffers
8 > 1 November 2012

9. Physical memory on a Teradata node
Memory managed by Memory managed by
O.S. PDE
9 > 1 November 2012

10. Memory Monitoring
Memory WHY
Monitoring
WHAT
HOW
Memory Depletions:
Common cause of
Performance degradation
MONITOR:
Free memory availability, Examine Resusage Data:
Memory allocation failures, With the same level of scrutiny
as CPU and I/O utilization.
Paging/Swapping.
Cache Effectiveness Rate
10 > 1 November 2012

11. Memory Availability
The low happened
dips show
Check ifDepletionswhen memory
hash join is frequency of
Given the enabled;
instances
during differentahours of
this depletion, it could
can potentially usecompletely
was low or high
amount of day,memory. of the
the OS all days First
be corrected by
depleted.
week with noFSGcache
adjusting particular
turn off hash join, see how to
pattern.value
lower
much memory can be
recovered.
11 > 1 November 2012

12. Memory Allocations Failures
Two cases when Mem
Alloc Failures occurs. How many and how
Case1:
Free Mem memory
frequently not
available. failures
allocations
occur
Which nodes or
Case2:
group of nodes are
Free Mem Available.
showing this pattern
12 > 1 November 2012

13. Memory Allocations Failures (Case1)
Mem Alloc Fails occurs
when FREE MEM
reaches to zero.
FREE MEM
zero at 14:10
Memory Allocation
Failures at 14:10
13 > 1 November 2012

14. Memory Allocations Failures (Case 2)
Check peaks in redistributed
blocks and correlate to
Memory allocation failures timeframes when large
occurring but MIN FREE MEM
Common cause of numbers of mem alloc failures
MEM has not reached Concurrent row
ALLOC fails is at 0 occurred.
during redistributions.
log interval.
FREE MEM never
Mem Alloc Fails reaches at zero.
at
Redistribution Peak
22:10
at 23:30
Mem Alloc Fails at
23:30
Then memory allocation
failures are coming from
some subset of memory
being used in the OS.
Redistribution Peak
at 22:10
14 > 1 November 2012

15. Cache Effectiveness Rate
Cache Effectiveness Cache utilization is a
Rate temporal cache hit rate
A 70% (time-based)
is Check Cache Hit ratio
Logical Reads – Physical Reads 70%
process. same
not the
of data, running level of
Logical Reads
with the same all day
scrutiny as CPU and
long. It is also spatial.
Cache Hit Ratio less than I/O utilization.
40% for a particular
workload
15 > 1 November 2012

16. Cache FSG Percent Tuning
Tuning memory generally All memory used inside
involves adjusting the FSGcache governed by a
FSG cache percent. thresh hold (e.g., datablock
caching) will also be
affected
Adjusting Cache
FSG Percent
This is why it is important Before tuning memory, it is
to measure cache important to see trends over
effectiveness rates before time e.g. take 30 days
and after making a sample
change.
16 > 1 November 2012

17. Summary of Steps in Cache FSG Percent tuning
Look for Low mem
1: Free Memory conditions, total
depletion
High number of I/Os,
2: Occurrence of high memory allocation
paging/swap I/Os failure
3: Check for use If OS memory is low/depleted
of memory intensive often, turn off hash join first
features as it does not require a
Target effectiveness
restart.
rates should be above
4: Check cache hits 50% (higher the better)
5: Check current Default is 80%
FSGcache setting
OnIfcoexistent withrate went
effectiveness
frequent aftermem
down low reducing
6: Lower FSGcache
conditions,it maycache size
fsgcache, the be
in 5-10% increments
necessaryincreased byto
can be to go down
7: Recheck dbs
65% at 4GB.
tuning DBSCACHETHR
cache hits
upward in dbscontrol.
8: Repeat steps
Sudden onset of 5-7 as needed
memory
depletion. 17 > 1 November 2012

18. DBSCacheThr - Understanding DBS Cache
FSG cache is the portion
DBS Cache of main memory
assumed available for
table data.
DBSCacheThr Case 1: SubTable/Table fits entirely of the
defines a percentage
DBScontrol tunable
FSG cache and helps control
parameter.
when data blocks are
Only permanent and
cached.
Case 2: SubTable/Table DBs can be
spool cannot fit
affected by this
parameter.
When a subtable fits The table is not eligible
entirely in this for synchronized full-file
percentage of memory, scan (sync scan).
its data blocks are
cached and aged out of fit
When the table cannot
memory normally.
in this percentage of If involved in full-file
memory, its data blocks scan, the table is also
are discarded from eligible for sync scan.
memory as soon as
possible.
18 > 1 November 2012

19. DBSCacheThr - Understanding DBS Cache
DBS Cache
Data in cache memory is
aged based on residency and
usage.
DBSCacheThr purpose
We want to hold all frequently
accessed reference tables in memory.
Range and Default Values
DBSCacheThr caching
technique can prevent a
large sequentially read or encourages smaller (e.g.
written table from pushing Since the large table ’ s data reference) tables to stay in
other data out of the blocks probably won ’ t be memory longer.
cache accessed again until they have
•Default value is 10 percent.
aged out of memory, •caching
Range of values is 0 - 100 percent.
them does little good and may
if the size of FSGcache cause other, more heavily-
changes, the size of DBS accessed blocks to age out
cache will change with it. prematurely.
19 > 1 November 2012

20. HTMemAlloc -Understanding Hash Join
What is it Hash Join
Hash Join vs Merge Join
Hash Join vs Product Join
Alternative join scheme
Memory Useage that performs better than
some cases of merge
Do not turn this feature on if Hash join eliminates the and product join.
sorting, and possible join
HJ builds an in-memory
redistribution or copying,
Hash Table using the
of the larger table.
The memory used when MJ require both left and smaller of the two join
hash join is enabled is right tables to be sorted relations.
from the available OS
Saves comparing every
memory
row in the right table with
every row in the left
table.
Right table rows are
The ResUsage data for Available free memory
compared to a single hash
the target system shows appears to be adequate
code in the indexed array.
a trend of low memory but page swaps are
conditions already frequent
present
20 > 1 November 2012

21. Hash Join Parameters
HTMemAllocBase = 10 MB
HTMemAlloc = 2% (Default) The default of 2% is chosen
based on having 50
concurrent hash joins
taking up a maximum 100%
of the HTMemAllocBase
Take an example; a system 10 MB of memory comes
size.
can have at max 25 concurrent from UNIX FreekB * 50 = 10 MB
200 Memory
users (e.g Throttling Applied)
2% applied to
Set HTMemAlloc = 4%
HTMemAllocBase Result when applied yields
4% of 10MB = 400 KB
a Hash Buffer size of 200
400Kb * 25 Users = 10 MB KB.
2% of 10MB = 200KBPartition size
New Hash
= 400KB (old 200KB)
More opportunity for queries of
having HJ instead of MJ or PJ.
Result :Faster executions
21 > 1 November 2012

22. Redistribution Buffer – Memory Requirement
Memory Requirement For Node Level Row Redistribution
Each AMP uses separate redistribution
buffer for each node in the system
Default Redistribution buffer size
= 32KB per target Node
Total Memory for 1 Sending AMP
= 32KB * number of Nodes in system
22 > 1 November 2012

23. Redistribution Buffer - Examples
Redistribution Configuration - 8 Nodes, 8 AMPs per node
Buffer Example 1
Single Node requirement - single user
= 32KB * 8
= 256 KB / Redistribution
Multi-user - 20 concurrent users
= 20 * 256 KB
= 5 MB (not a special problem)
Redistribution Configuration - 200 Nodes, 8 AMPs per node
Buffer Example 2
Single Node requirement - single user
= 32KB * 200
= 6400 KB (6 MB)
Multi-user - 20 users
= 20 * 6400 KB
= 128 MB
(far exceeding 80-100 MB per AMP)
23 > 1 November 2012

24. Redistribution Buffer - Recommendations
Symptoms: Excessive Paging/Swapping
RECOMMENDATIONS For Large Configurations:
Set Node Level Redistribution buffer size
smaller , e.g. 16 KB (Default is 32 KB)
Set FSG Cache Percent to less than 80%
24 > 1 November 2012

25. Feed back
Shaheryar-
This is an excellent presentation.
25 > 1 November 2012

26. Questions
The only bad question
is the question
never asked
26 > 1 November 2012