[Spread-users] problem with spread/mod_log_spread/spreadlogd

[Scott Barvick]

John,

Has anyone given any thought to passing daemon or connection congestion
status back through through messages received by the clients?  I can
easily kill my Java client with too much traffic, but I might be able to
slow down the traffic if I knew I was starting to get into to trouble.  
Having a client close with an error is more disruptive in some cases
that cutting back on the data sent so it would be nice to have that
option.

Thanks,
Scott

On Tue, 2005-09-06 at 17:43, John Schultz wrote:
> FYI, Spread by default will buffer up to 1,000 msgs for a connection 
> before it will kick it.  The thing is that you can push Spread to pass 
> over 10,000 (small) msgs per second in a LAN.  So if you have some kind of 
> extreme msg spike and a reader can't keep up in that time frame, then I 
> believe they can be disconnected in less than a second!
> 
> I believe you can raise this limit by modifying spread_params.h
> 
> #define MAX_SESSION_MESSAGES 1000
> 
> try raising it by an order of magnitude
> 
> #define MAX_SESSION_MESSAGES 10000
> 
> Now the potentially negative side effect of this is that your spread 
> daemon will probably eat more memory on your system.  So you need to 
> ensure there is memory for it to grow because you don't want the daemon to 
> die due to low memory.
> 
> Good luck!
> 
> ---
> John Schultz
> Spread Concepts
> Phn: 443 838 2200
> 
> On Tue, 6 Sep 2005, Jeroen wrote:
> 
> > Theo Schlossnagle wrote:
> >
> >> John Schultz wrote:
> >> 
> >>> Well -11 is CONNECTION_CLOSED, which just means the connection between the 
> >>> client and daemon has been shut down.  The most common reason for this is 
> >>> a flow control problem where msgs are being injected into the system 
> >>> faster than readers can read them out.  At some point Spread will kick the 
> >>> connection so that it doesn't run out of memory and kill the daemon, thus 
> >>> losing all of its connections.
> >>> 
> >>> I'm not familiar with mod_log_spread and I don't know if it performs any 
> >>> kind of flow control.  If it doesn't and you are logging too fast this 
> >>> could cause your clients to be repeatedly disconnected (assuming they 
> >>> reconnect).
> >> 
> >> 
> >> mod_log_spread does no flow control what-so-ever.  spreadlogd will read 
> >> message from Spread as fast as it can write to disk.  So, the typical 
> >> reason for this sort of behaviour is that you try to journal the logs from 
> >> your entire cluster on an IDE system or some other slow storage facility.
> >> 
> >> The lack of flow control was a design decision in mod_log_spread.  In otder 
> >> to have time-ordered, real-time logs, you either must have no flow control 
> >> or you must allow the publishers to block.  In m_l_s, it was decided that 
> >> under no circumstances should publishers block (as that would mean a 
> >> slowdown in serving web pages).  If that approach doesn't "jive" with your 
> >> idea of logging in a web cluster, then m_l_s isn't for you.
> >> 
> >> (The *you* above, is of course not John, but whomever is running m_l_s)
> >> 
> > Thank you for your awnser!
> >
> > this is exactly what my thought were, but the problem is that the system that 
> > is writing the logs to disk is a scsi-320 system with hardware raid-1 :)
> >
> > it's just writing about 100mb of logs per hour, that doesn't look that much 
> > to me.
> >
> > Load is low on that server since it is a dedicated logserver, but I/O might 
> > be a problem (since that isn't always shown in pure load).
> > I'll hookup a raid-10 system with 15k rpm disks tomorrow and test further.
> > Another problem believing this conclusion is that we have another cluster 
> > that is performing great loging to a sata-raid array, writing almost 2 gigs 
> > of logs every hour, but that is another setup.
> >
> > Kind regards and please give your comments since this isn't solved yet,
> >
> > Jer
> >
> >
> > _______________________________________________
> > Spread-users mailing list
> > Spread-users at lists.spread.org
> > http://lists.spread.org/mailman/listinfo/spread-users
> >
> 
> _______________________________________________
> Spread-users mailing list
> Spread-users at lists.spread.org
> http://lists.spread.org/mailman/listinfo/spread-users