CNL malloc environment variables
rev. 7/10/2007

1. Overview

The CNL kernel provides some environment variables to control
how the system memory allocation routine "malloc" behaves.  These
variables are supported in the current CNL kernel.  They set the 
malloc tunable parameters provided by the mallopt system call.  
These tunable parameters are part of the general SVID/XPG interface, 
defined in 'malloc.h'.

2. Variables and Parameters

The environment variables are:
   MALLOC_TRIM_THRESHOLD_  
   MALLOC_TOP_PAD_
   MALLOC_MMAP_THRESHOLD_ 	
   MALLOC_MMAP_MAX_

Note the trailing underscores on the environment variable names.

These match to the following mallopt tunable parameters:
   M_TRIM_THRESHOLD
   M_TOP_PAD
   M_MMAP_THRESHOLD 	
   M_MMAP_MAX 

The two variables we have found most useful are MALLOC_MMAP_MAX_
and MALLOC_TRIM_THRESHOLD_ .

3. MALLOC_MMAP_MAX_

Setting MALLOC_MMAP_MAX_ limits the number of 'internal' mmap 
regions.  Setting this to 0 (from the default of 64) means that
instead of having 64 possible 'non' heap memory mapping regions, 
the program will not use any.  This eliminates the system calls 
to mmap/munmap.

Using MMAP regions turns out to be very costly compared to just using 
the heap.  They exist to allow a program to return unused memory 
back to the system more easily so it may be used by other processes
on the node.  This is most applicable and helpful on a SMP node 
architecture with multiple programs sharing a node.  Since XT systems 
do not run multiple applications on the same compute node, using MMAP 
regions is generally not helpful.  

If you have an MPI program where one process on a node uses a lot of 
memory, then frees it, then the other process on the same node uses 
a lot of memory later on, MMAP regions could help.  But memory usage 
is not typically asymmetric in this way.   An application that does
a lot of I/O near the end of its run, after freeing a lot of memory,
may also see a benefit from using MMAP regions.  But we have not 
encountered programs where this potential benefit outweighs the 
extra cost of doing mmaps and unmmaps during the life of the program.

We suggest setting :
   MALLOC_MMAP_MAX_ = 0
(this setting says not to use mmap regions)

4. MALLOC_TRIM_THRESHOLD_

Setting MALLOC_TRIM_THRESHOLD_ helps performance by reducing system 
time overhead by reducing the number of calls to sbrk/brk.
MALLOC_TRIM_THRESHOLD is the amount of free space at the top of the
heap after a free() that needs to exist before malloc will return 
the memory to the OS.  Returning memory to the OS is costly (as is 
obtaining it).  The default setting of 128 KBytes is much too low for 
a node with 4 GBytes of memory and one application.

We suggest setting :
   MALLOC_TRIM_THRESHOLD_ = 536870912  
(this setting is 0.5 GBytes)

If your code uses less then 0.5 GBytes, set the value to the amount the
code uses.  If your code needs lots of memory for I/O buffering, and 
it would free some memory if this value was set smaller, you may want 
to set this lower than our recommended 0.5 GBytes.  

5. MALLOC_TOP_PAD_

There is one more setting that may be useful for a few applications,
MALLOC_TOP_PAD_  .

This value controls how much additional ("pad") memory is requested
from the OS when a sbrk is done.  The default setting is 0.

If your code does a lot of 'small' memory allocs, this may help a
little.  As an example, if your code alloc's 4k pages worth of memory 
several times, and you set this MALLOC_TOP_PAD value to 2 MByte, this 
would force the first 4KB alloc to grab 2MB from the OS, and then the 
following 4KB allocs would use this 'pad' space and not go through the 
system overhead of an sbrk for each additional 4k.

We have not seen this value to have much effect on our test codes (as 
most high performance computing codes do not do this this type of 
small memory allocation, and the MALOC_TRIM_THRESHOLD tends to help 
for these cases anyway), either no gain or only a few seconds.  
However, this may help in some cases.

6. References

Addtional information on malloc tuning options can be found in 
various references.  Some ones that we have found useful are:

GNU Description of mallopt options:
http://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html

P. Ezolt  - USENIX paper on malloc tuning:
http://www.usenix.org/publications/library/proceedings/als01/full_papers/ezolt/ezolt.pdf

G. Insolvibile - Linux Journal article on malloc
(see the section "Controlling Allocation Strategy"):
http://www.linuxjournal.com/article/6390/print

Table of mallopt parameters and env variables:
http://www.linuxjournal.com/articles/lj/0109/6390/6390t1.html