[Spread-users] possible memory leak in spread daemon

[Jonathan Stanton]

This may be a memory leak, but Spread does have memory usage behavior that 
surprises people and sometimes looks like a leak but is not.

The two things to check first are:

1) Does it grow 'steadily' or does it jump up and then stay that way for 
while. 

2) Did you have peaks of traffic through spread (either in number of 
messages or the size of those messages) during that time.

(If it jumps and/or you had traffic increases then it is likely NOT 
evidence of a leak -- it might be, but spread would normally also behave 
that way. If it increases steadily or you are sure traffic load did not 
increase, then it is more likely to be a leak)

The reason for these questions is that Spread can look like it is 
leaking for two reasons:

1) Spread will temporarily use a lot of memory for messages buffers, and
then "free" the memory back to the libc malloc implementation by calling
'free()'. But most OS/libc's don't report that or actually return it to
the kernel. So in 'ps' or top commands the program looks like it is still
using the memory.

2) Spread has an internal memory management system that caches blocks of 
similary sized memory buffers to decrease the overhead of handling quickly 
varying message loads. These memory buffers can grow to substantial size 
if you have large numbers of clients, very fast message rates, lots of 
slow receivers, or other scenerios that require Spread to buffer lots of 
messages internally before it can deliver them to the receiver sets.

A year or so ago someone ran a memory leak tracking tool on Spread and we 
found (I think) one leak, and a bunch of false positives. I also built a 
custom memory leak tracking version of Spread's internal memory management 
code and used that for awhile. If you still see it growing I can pull out 
that code to try it. 

If you need to bound the memory usage tighter, recompiling with some 
smaller limits in the spread_params.h file will decrease the maximum size 
Spread can grow to by limiting how many messages it will buffer 
internally. This can increase the cases of clients getting cut off for 
reading too slow, or might limit the number of clients or daemons you can 
run.

Hope this helps some,

Jonathan

On Thu, Feb 05, 2004 at 04:35:52PM -0600, Greg Shebert wrote:
> hi folks
> 
> just wanted to report that, using the latest release of spread (not from
> CVS) on redhat 9 seems to leak memory...
> 
> i see my spread daemon grow from about 2 meg to about 50meg over the
> course of a few days...
> 
> any tips on how i might track down the problem?
> 
> -greg-
> 
> 
> 
> _______________________________________________
> Spread-users mailing list
> Spread-users at lists.spread.org
> http://lists.spread.org/mailman/listinfo/spread-users

-- 
-------------------------------------------------------
Jonathan R. Stanton         jonathan at cs.jhu.edu
Dept. of Computer Science   
Johns Hopkins University    
-------------------------------------------------------