[Spread-users] Optimizing inter-segment communication to reduce packets

[Jonathan Stanton]

On Sat, Feb 09, 2008 at 02:01:14PM -0800, Jim Kleckner wrote:
> >>- Having a reliable and complete view of all memberships might
> >>permit you to know definitively whether there are no members
> >>of a particular group in a given segment and allow you to
> >>elide sending non-membership messages to that segment.
> >>    
> >
> >Spread currently has a 2-level model internally -- groups are implemented 
> >at the
> >top level and the lower level message-sending and routing code doesn't 
> >know anything about them. To it there is only one group -- that of 
> >currently connected daemons. 
> 
> This seems to be the crux of the issue.
> The upper level might be able to compute a "list of segments" that have 
> members for groups.
> The lower level might ignore this based on the selected guarantee?
> 

The upper layer definitely knows which daemons have members of each group, and thus which 
segments. Except for the unreliable messages case (as you note below) we still have the 
limitation that the reliability and ordering protocols rely on the fact that every daemon 
will see every message (so each knows when there are gaps in delivery and when it has 
received all of the currently outstanding messages). So some header information about each 
message would still have to be sent (without the body). 


> >>- Would it simplify the optimization if we limited it to classes
> >>of message with lesser guarantees?  What level of guarantee
> >>causes the optimization to get very difficult?
> >>    
> >
> >Good question. This is possible. 
> >  
> 
> Is the UNRELIABLE_MESS case simple enough to implement?
> UNRELIABLE_MESS could handle some useful applications.

If unreliable is useful, then that is easier, as it provides no real guarantees -- 
it's just a 'best case' type of service.

We actually have usually implemented unreliable the same as reliable, because in the types 
of applications Spread is used for, there wasn't much advantage to having an 
unreliable service. 

> One could structure an application manually to organize groups with 
> different levels
> of guarantee and even send the same message twice, one unreliable and 
> the other reliable.

This type of separation can be done already if you know that certain groups will only 
involve a subset of the daemons by running multiple spread configurations with their own 
daemons. Each configuration only will have the daemons it needs running and thus messages 
will only reach those daemons that are part of the configuration the messages are sent to. 

This is the solution you mention below where an application connects to multiple spread 
networks/configurations. 

> Another question is whether one could compose this at a higher 
> application level.
> What barriers exist to a single program connecting to two spread networks?

None, they are independant. The idea of running multiple spread configurations has been 
proposed before (there should be some discussions in the archives) Mainly the reason has 
been to support more daemons then will work well in a single configuration (such as > 100 
daemons). 

> One could build an "active bridge" program that notices the memberships of
> groups on both sides, computes the group-to-network/segment mapping,
> and copies messages from one side to the other.  Each side works as today.

One of the key questions is if the set of group members that are located at certain 
daemons is static or dynamic? If it is static, then certain configurations can be assigned 
to different sets of groups (those that all use the same set of daemons) and then messages 
just need to be sent to the correct daemon configuration to reach only those daemons who 
have members. That's something that can be done with spread today.

However, if the mapping of group members to daemons is dynamic, then which configurations
each group is on would have to change during execution. That would not be possible with
spread without lots of changes. This idea has been explored before, you can find a 1996
paper on the idea of mapping dynamic sets of groups to sets of daemons (with a little
different terminology) if you google for "Dynamic Light-Weight Group Service". The reason
this hasn't been implemented (as far as I know) is that noone has really needed such
dynamic group mappings in an application that also needed strong semantics. If you don't
need the ordering/membership/reliability guarantees then most people just use IP or
application level multicast protocols that disseminate to different groups but arn't
strongly fault-tolerant. 

> 
> It doesn't address issues of guarantees at all though perhaps some coupling
> of the server logic for accepting the message as delivered could depend on
> the remote side being accepted.  This would probably be inefficient at best
> and get hairy for the more stringent guarantees.

Yes, in some cases it's not possible to bridge between multiple systems and still preserve 
the guarantees without adding new end-to-end protocols that remove any performance benefit 
from running multiple systems. 

Jonathan

-- 
-------------------------------------------------------
Jonathan R. Stanton         jonathan at cs.jhu.edu
Dept. of Computer Science   
Johns Hopkins University    
-------------------------------------------------------