[Spread-users] Spread bandwidth throttling

Jonathan Stanton jonathan at cnds.jhu.edu
Wed Nov 14 01:53:48 EST 2007


Sure. 

On Wed, Nov 14, 2007 at 05:19:17PM +1100, Sonam Chauhan wrote:
> Thanks a lot Jonathan. 
> 
> The application does not lend itself to simple bandwidth limiting - so I
> was hoping to limit bandwidth either in Spread or the network interface.

The key issue is that Spread provides strong guarantees like it doesn't lose your 
messages, so if they are arriving faster then either the network can forward them (if 
virtual network interfaces are used to limit) or then spread will process them (if it 
implemented limits itself) --- What should spread do then with the messages? It can't 
drop them or make the network run faster, so it buffers them. Even then 
if they keep coming Spread will eventually run out of memory.

Currently spread has a limit of 500 (defined as WATER_MARK in daemon/spread_params.h) 
messages that can be received from some client session but not yet sent on the network to 
the other daemons. If that many are received, it will 'block' client sessions by no longer 
reading from their sockets, the kernel buffers will store upto a TCP window size worth of 
messages from the client and then TCP will be blocked by it's own flow control and then 
the 'send' calls on the client side will start blocking because there is no room in the 
sender side TCP buffers.

All of that is a long way to say the sending clients will find their SP_multicast calls 
blocking once they are more then 500 messages ahead of the daemons. 

You can certainly raise that limit from 500 to a larger number, just change the WATER_MARK 
define and recompile the daemon, however that doesn't fix the ultimate problem, just 
delays it's onset for a longer time at the cost of the spread daemon using more memory.

> > 'fixed' bandwidth limit could have the problem that the clients
> actually 
> > send faster then that and so more and more messages have to be
> buffered up 
> > in the daemons -- eventually causing very high latency for message 
> > delivery and possible blocking of the sending clients (because the
> daemon 
> > won't buffer more messages for them)
> 
> Latency is no problem, but eventually blocking would be. Can message
> buffer limits be set in the daemons? Can they daemons be set to buffer
> messages, say, on disk, instead of in-memory? 

No, Spread does not use disk for anything (except for logging debug messsages if you 
choose to) It is not designed as a persistent store of messages, but as a fast, reliable 
and  strongly ordered   message bus. 

However a number of applications that use Spread do have persistence requirements and so
they use disk to store messages as well as (or before) sending them through Spread. 

Such a solution might work in your case, where your application sending 'results' updates
about the tests, could log the results to disk first, and then have a second process read
the records from disk and send them to Spread 'as fast as possible'. One of two things
would happen then:

1) The Spread network (with bandwith limited network) is fast enought for the incoming 
message stream: Then nothing would block and there would just be a small additional delay 
for writing the records to disk before sending them.
2) The spread network bandwidth is not suffient, so messages will be buffered, and then 
the sending clients will block. However the sending client in this case is just the second 
process who very simply reads from a log file and sends spread messages -- your actual 
logging program would just keep writing new records to the local disk log file and 
wouldn't block. 

So in either case your apps wouldn't block. Basically this solution uses a local disk as a 
spool for the records before sending them (like a printer spool). 

Maybe an architecture like that would solve the problem for you. I know such a log-sender 
process (read from disk log and send to spread) has been written many times before as part 
of support for distributed logging in clusters. I even think some students in our research 
lab wrote a version of it once.

Cheers,

Jonathan

> 
> -----Original Message-----
> From: Jonathan Stanton [mailto:jonathan at cnds.jhu.edu] 
> Sent: Wednesday, 14 November 2007 1:58 PM
> To: Sonam Chauhan
> Cc: Spread Users
> Subject: Re: [Spread-users] Spread bandwidth throttling
> 
> Spread doesn't have any bandwidth limiting built in, however the load it
> puts on the 
> network "in addition to the application messages your clients send" is
> quite small and 
> mainly involves a small token packet making rounds of all of the daemons
> (if idle it only 
> circles every second or so, if nodes are sending it circles more often)
> 
> So if you do bandwidth limiting at the application client level by
> limiting how many 
> messages you send, the actual network load should be close to what you
> send through the 
> clients.
> 
> Since Spread just disseminates the messages your applications generate,
> any 'fixed' 
> bandwidth limit could have the problem that the clients actually send
> faster then that and 
> so more and more messages have to be buffered up in the daemons --
> eventually causing 
> very high latency for message delivery and possible blocking of the
> sending clients 
> (because the daemon won't buffer more messages for them)
> 
> Your solution using virtual network interfaces which are bandwidth
> limited should work
> fine (noting, however, the observations in the previous paragraph).
> Spread can be
> configured to bind to specific interfaces for daemon-daemon traffic and
> client-daemon
> traffic. Check out the sample configs in sample.spread.conf in the
> source distribution. 
> 
> Cheers,
> 
> Jonathan
> 
> On Mon, Nov 12, 2007 at 10:42:54AM +1100, Sonam Chauhan wrote:
> > Hello! 
> > 
> >  
> > 
> > I am new to Spread. I am interested in using it to stream results in a
> > distributed load testing application where messages flow from the
> > testing nodes to a central log server. 
> > 
> >  
> > 
> > The issue is that the load testing itself consumes a lot of network
> > bandwidth. Does Spread or the Spread client have any bandwidth
> > throttling/ bandwidth-limiting features that can limit it's network
> > utilization? For instance, can I specify that Spread messages
> throughput
> > from a client not exceed 10 Mbps? 
> > 
> >  
> > 
> > If not, what would be the best way to achieve what I have in mind?
> Would
> > it be to setup virtual network interfaces limited to 10 Mbps and have
> > the Spread components bind to those interfaces? 
> > 
> >  
> > 
> > Kind regards,
> > 
> > Sonam Chauhan
> > 
> > -- 
> > 
> > Corporate Express Australia Ltd. 
> > 
> > Phone: +61-2-93350725, Email: sonam.chauhan at ce.com.au
> > 
> >  
> > 
> 
> > _______________________________________________
> > Spread-users mailing list
> > Spread-users at lists.spread.org
> > http://lists.spread.org/mailman/listinfo/spread-users
> 
> 
> -- 
> -------------------------------------------------------
> Jonathan R. Stanton         jonathan at cs.jhu.edu
> Dept. of Computer Science   
> Johns Hopkins University    
> -------------------------------------------------------

-- 
-------------------------------------------------------
Jonathan R. Stanton         jonathan at cs.jhu.edu
Dept. of Computer Science   
Johns Hopkins University    
-------------------------------------------------------




More information about the Spread-users mailing list