[Spread-users] spread multi-threaded

Theo Schlossnagle jesus at omniti.com
Wed Sep 10 14:40:03 EDT 2003


On Wednesday, Sep 10, 2003, at 14:09 US/Eastern, John Jolet wrote:
> I'm setting up a spread-based central logging server system and i've 
> got some logs that are incredibly high-activity.  on the central 
> server, i end up with what looks like the spread daemon taking up 99% 
> of one cpu, and none of the other.  can spread be made multi-threaded 
> to deal with a high volume of messages?  the eventual architecture 
> will be three spread servers, with hundreds of clients.  the clients 
> will not be running spread, but will attach to a virtual ip (shared 
> via whackamole) between the spread servers for availability.  right 
> now, there's just a single spread server, so he's not having to 
> forward the messages, and there's no token-passing issues, since he's 
> the only server.

I would use a multi-ring technique.

If you have 8 log generating (LG1 - LG8) servers and one log writing 
host (LW1) set up two Spread rings:

Ring one on port 4813 has LG1, LG2, LG3, LG4, LW1
Ring two on port 4913 has LG5, LG6, LG7, LG8, LW1

If you are using spreadlogd to write logs, it can listen to multple 
daemons.  So just point it at LW1:4813 and LW1:4913 and let'er rip.

This can increase throughput and reliability it extremely high-traffic 
logging environments.  I've seen sites have great success using 10 or 
more different rings where hundreds of web servers are involved.

// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Postal Engine -- http://www.postalengine.com/
// Ecelerity: fastest MTA on earth





More information about the Spread-users mailing list