[Spread-users] changing TTL to use IP multicast in WAN

[John Lane Schultz]

Stanisław Trytek wrote:
> 
> As far as I am concerned publicly accesible version of spread uses 
> modified Ring protocol in WAN
> with one difference: instead of broadcast/multicast it uses point to 
> point udp protocol.
> It differs a lot from HOP described in article "A Low Latency, Loss 
> Tolerant Architecture and Protocol for Wide Area Group Communcation "
> There are no such wonderful mechanizms as in HOP.  

Yes, the paper to which you refer was done for an experimental version of Spread that never made it to production level code.  

Another system that you might want to take a look at which includes a lot of the overlay ideas from that experimental version (but has FAR weaker service semantics than Spread) is Spines.  http://www.spines.org/

> Thus each daemon 
> sends message to all daemons.
> Simply, the problem is with scalability. Number of messages sent by each 
> daemon is n*(m-1) where m is number of daemons.
> Having totally 10 daemons and one (or more) daemon working on lower 
> channel can be real problem.

Each daemon only sends a message to all daemons if you only have one daemon per WAN segment.  Each daemon sends one message to the leader of each segment who then locally broadcasts the message to all the daemons in that segment.

> Hopefully I am using spread in wide area network where it is possible to 
> use IP-Multicast.
> I have changed TTL option (and timeouts of course) and treat all 
> deameons as they work in one segment (in configuration file they all are 
> in one segment).
> It seems to work fine. The problem I have is same race conditions, the 
> token somehow overtakes messages and I causes retransmissions.
> But hope it is the issue of proper configuration of routers.

If you can get the IP multicast working properly across your WAN, then your approach could work.  

Another way you could get the token traffic to not overtake data traffic is to change the code so that the token goes on the same port as the data traffic.  This could increase your likelihood of losing tokens when routers start dropping your packets or when you start overflowing a receiver's buffers (a bad thing) but in a no loss situation it would get rid of the problem of your routers trying to be too smart.

> Have someone experience with such aproach? Are there any disadventages 
> or threats?

No, usually getting IP multicast to span any decent size network is quite difficult.  Usually on a WAN there are lots of intervening routers that you do not have control over and which will not propagate IP multicast trees.

The only other "threat" I can think of is that Spread was built with the assumption that multicasts would only be a single hop and so some other (e.g. - cost) assumptions in the code might be based on that.

---
John Schultz
Spread Concepts LLC
Phn: 443 838 2200 
Fax: 301 560 8875