[Spread-users] Memory leak? FD leak? Other?

[Ryan Caudy]

Do any of the threads get disconnected?  Spread won't disconnect the
entire process (it doesn't know about processes), but it should
disconnect each of the individual worker threads once it manages to
accumulate MAX_SESSION_MESSAGES (1000 by default) sent by that thread
(since they are only sending to themselves, and not using
SELF_DISCARD).

What matters more for your purposes is that (a) the messages you are
sending are huge, and (b) Spread limits messages based on number of
messages, not total bytes.  So, what happens is that the daemon will
queue up to 500 messages (about 64 MB, in your case) that haven't been
sent, before it stops reading client messages.  After that, it will
have up to 8192 messages outstanding that can't yet be delivered for
whatever reason.  So, that's a lot of memory that it could be using in
a case like yours.  I'm not sure what the solution is, except changing
the parameters that dictate Spread's queueing behavior in these
abusive cases.

Maybe there's some way to make Spread more cautious, but I'm not sure
what to suggest.

Cheers,
Ryan

On Thu, 19 Aug 2004 12:07:26 -0400, David Shaw <dshaw at archivas.com> wrote:
> I've been seeing odd behavior with spread 3.17.2 recently.  Basically,
> the memory it uses grows steadily and never goes back down again.  To
> be sure, Spread does some memory management internally so it may not
> wish to give back memory when I expect it to, but the behavior I am
> seeing is pretty far out of line.
> 
> I've attached a simple program (sabuse) that spawns many threads, and
> each stuffs large messages into spread without reading them back.
> Obviously this is going to cause spread to grow since it must store
> the messages.  However, there is no limit on the growing - spread will
> happily grow until the system runs out of swap, rendering that machine
> useless.  Limiting the memory via ulimit does not work since spread
> will exit with a "Message_add_scat_element: Failed to allocate a new
> PACKET_BODY" if it cannot get enough memory.
> 
> Of course, under normal circumstances nobody would do such a thing.
> However, when I kill the sabuse program.  I would expect spread to
> give back some memory and it doesn't.  I know spread keeps some memory
> around for performance reasons, but at this point it owns most of the
> memory and swap on the system.  Similarly, I would expect spread to
> close all the file descriptors it has open to talk to the sabuse
> program and it doesn't (this is visible in /proc).
> 
> This behavior is on linux 2.4.25, and glibc 2.3.2.
> 
> David
> 
> 
> 


-- 
---------------------------------------------------------------------
Ryan W. Caudy
<rcaudy at gmail.com>
---------------------------------------------------------------------
Bloomberg L.P.
<rcaudy1 at bloomberg.net>
---------------------------------------------------------------------
[Alumnus]
<caudy at cnds.jhu.edu>         
Center for Networking and Distributed Systems
Department of Computer Science
Johns Hopkins University          
---------------------------------------------------------------------