[Spread-users] problem with spread/mod_log_spread/spreadlogd

[Theo Schlossnagle]

John Schultz wrote:

> Well -11 is CONNECTION_CLOSED, which just means the connection between 
> the client and daemon has been shut down.  The most common reason for 
> this is a flow control problem where msgs are being injected into the 
> system faster than readers can read them out.  At some point Spread 
> will kick the connection so that it doesn't run out of memory and kill 
> the daemon, thus losing all of its connections.
>
> I'm not familiar with mod_log_spread and I don't know if it performs 
> any kind of flow control.  If it doesn't and you are logging too fast 
> this could cause your clients to be repeatedly disconnected (assuming 
> they reconnect).

mod_log_spread does no flow control what-so-ever.  spreadlogd will read 
message from Spread as fast as it can write to disk.  So, the typical 
reason for this sort of behaviour is that you try to journal the logs 
from your entire cluster on an IDE system or some other slow storage 
facility.

The lack of flow control was a design decision in mod_log_spread.  In 
otder to have time-ordered, real-time logs, you either must have no flow 
control or you must allow the publishers to block.  In m_l_s, it was 
decided that under no circumstances should publishers block (as that 
would mean a slowdown in serving web pages).  If that approach doesn't 
"jive" with your idea of logging in a web cluster, then m_l_s isn't for you.

(The *you* above, is of course not John, but whomever is running m_l_s)

-- 
// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Ecelerity: Run with it. -- http://www.omniti.com/