[Spread-users] Segments v/s Connection pooling

[George Schlossnagle]

On Feb 9, 2004, at 10:41 PM, Anurag Gupta wrote:

> Hi,
>
> We want to create an apache API (on the lines of mod_log_spread) for 
> some
> communication from webservers to a metering system. Do people have
> experience with large installations?
>
> We have ~75 machines in the cluster. Looking at the current code of
> mod_log_spread, a connection is opened for each yapache being spawned. 
> We
> have upto 200 child processes running at a given time. So, we need to 
> have
> upto 15K connections open to the spread daemon(s).
>
> Questions:
> 1) Is having a daemon at each server (in the same segment or different
> segments) a better option than having 1 or few daemons?  How should 15K
> clients be broken into different segments (1 daemon per segment). I 
> remember
> having read somewhere that more than 30 daemons in the same segment 
> may lead
> to problems (timeouts need to be increased etc.)

I think I've run the largest mod_log_spread instance around (130mm 
lines logged per day).  I used relatively small rings (<10 machines) 
and used a single log consolidator that belonged to all of the rings. 
That worked pretty well.  If I grew the rings too large I had stability 
issues, but the load on my machines was pretty high.  From what I've 
heard from Michael, you y! guys seem to manage better than I did.



>
> 2) Another thought in my team was opening a bunch of connections in the
> server_config stage and using some connection manager to use these
> connections. Has anybody done this in past or, more importantly, 
> decided
> against this? Of course, if we decide to have multiple spread daemons, 
> this
> may not be required.

A thought that's been thrown around was to use sysv message queues to 
push log lines into and then have a client daemon read off the queue 
and broadcast to spread.  That's never been implemented to my 
knowledge, but always sounded good.

George

// George Schlossnagle
// Principal, OmniTI Computer Consulting, Inc.
// (w) 410.872.4910x202
// (c) 240.460.5234