[Spread-users] Memory leak? FD leak? Other?

[David Shaw]

On Fri, Aug 20, 2004 at 01:03:09PM -0400, John Schultz wrote:
> David Shaw wrote:
> 
> >I believe I have triggered both.  No memory is released by the daemon,
> >and some fds are left behind when using the sabuse program I sent.
> >
> >Currently, the sabuse program connects to spread and justs calls
> >SP_multicast over and over again.  If you modify the program to
> >SP_connect, send 500 messages, then SP_disconnect and then repeat the
> >whole connect-send-disconnect cycle over and over, you get different
> >behavior when killing sabuse: there are no fds left open in spread,
> >but the memory is still not returned.
> >
> >David
> >
> 
> Jonathan, doesn't Spread's memory subsystem keep memory it has allocated 
> around?  Furthermore, even if it releases that memory don't most 
> processes never release allocated memory pages back to the operating 
> system until they finish?  So if any process has a large spike in memory 
> usage, then their memory footprint as far as the OS is concerned never 
> shrinks?

This isn't true for all platforms.  With glibc (i.e. virtually all
Linux systems), if you free a memory block, glibc may shrink your
heap, giving memory back to the OS.  The OpenBSD libc does the same.
It all depends on a number of factors and platform-specific things,
and fragmentation can make this difficult, but glibc neatly hides all
this from the application.  Some memory allocators even do things like
mmap-ing /dev/zero and thus make it dead easy to release memory back
to the system.

My point is that in general, it's not a good idea to try and outsmart
the OS here.  When you want memory, malloc() it.  When you are done
with it, free() it.  Let the memory allocator do its job.

It's okay to keep some memory around if you are concerned about malloc
being too slow or other reasons, but then you really need to cap your
personal memory pool at some sane value so that a one-time spike in
memory usage doesn't result in your process permanently owning most of
the memory on the system (and thus preventing other processes from
doing something useful with it).

David