[Spread-users] Re: FEC support, bw control, msg size, encryption and data resends

[Jonathan Stanton]

Comments are below.

On Wed, Aug 23, 2000 at 07:22:08PM +0200, Charl Matthee wrote:
> We provide a satellite card service for users. The satellite
> connection is fast with high latency (as these things are :). The
> user basically uses the satellite card to receive data and his
> normal connection (diginet, ISDN, modem, etc.) to send data.
> 
> What we wish to do is multicast data to these users via the satellite
> with no back channel (i.e. no connection to the internet). Because
> there is no back channel most reliable multicast implementation can't
> cope (they need the back channel to request packet resends).
> 
> FEC would basically make up for the fact that there is no back channel
> to request resends because the receiver would be able to reconstruct
> the packet that was sent. Something similar to reliable PGM.

OK. I think I see exactly what you are trying to do. You want to reliably
multicast through your satellite to lots of receivers(1000's++). Yes, I
agree that FEC could be really useful. I also have to admit that I'm not
sure that Spread will be very useful to you for that purpose. Spread is
designed for systems with no more then 100 daemons (the machines on the
multicast tree) with each daemon supporting a good number (1000+)
connections from end-users (point-to-point).

Since it also sounds like the only service you need is reliability, i.e.
you don't need ordering of the messages, strong membership services, or
highly interactive communication. Then you probably want a pure reliable
multicast package that only does that, but does it for very large numbers
of receivers and using FEC. 

I don't have one I can recommend off the top of my head, but I know a
reseacher in germany Marcus Hofmann <hofmann at acm.org> who has done some
work building rel-mcast systems and has (at least experimentally) built
one using FEC.

If I'm wrong about what services and scalability you need please tell me.
We have done some work exploring high-latency satellite connections for
wide area networks, it actually partially motivated my Ph.D work. But we
kept our focus on many-to-many communication as opposed to one-many
multicast.

> 
> > The maximum message size is essentially the largest 'single' object you
> > can send through spread and get the guarantees provided to a single
> > message (such as EVS, agreed ordering, etc). If you want to send 500Kb
> > say, you would send it as 5 separate spread messages and each one would
> > have whatever guarantees you requested, but it would be possible for one
> > message to arrive before a new member joined the group and another message
> > to arrive after the new member joined.
> 
> Shouldn't the segmentation of the object you wish to send be done as
> an abstraction to programmer/user via the API? I basically want to
> send data and have a guarantee on it's delivery. I don't want to worry
> about creating data segments that are of the right size. I just want
> to send the data and it must magically appear on the other side.

The API allows you to send any message size from 0 to 100kbytes. But for
larger ones you have to segment it. For reliable mcast it doesn't matter
as much, but think about sending state transfer messages which you send
with a global order of agreed (5 different senders are sending state
messages at the same time and you want all 5 of them to recieve the 5
messages (one from each) in the same order.) If the messages were too
large (and there is always a too large, even if the limit was 1 meg, or 10
megs) then the senders would each send 2 messages to make up thier state
and those two messages could end up not next to each other in the global
order (say that order was 1a 2a 3a 1b 4a 4b 3b 5a 2b 5b --where a and b
are teh two messages sent by each sender 1..5) You have to know that they
were really sent as two messages because only the application knows what
it's policy is for ordering them. It is even more important if a fault
occurs during the time the messages are being sent and say node 2 dies
just after message 4b in the above list. The application has to realize
that it will never get message 2b and deal with it.

Sorry if this is long winded or irrelevant :-) I just wanted to clarify
why there is a hard coded limit. We can't let the application set a larger
limit arbitrarily because the daemon code also knows about the size limit
(and it has to have some size limit or an application can block the entire
spread network from serving anyone by sending a 'never ending' message
which we cannot ignore because of the global ordering services that we
provide.

 >  
> > Because group communication is based on a datagram (as opposed to stream)
> > model, you have to pick some maximum and 100kb seemed a good tradeoff for
> > now. 
> 
> Sure. But I still feel this size should be abstracted via the API :)
> 
> Perhaps my understanding of how Spread works is a little (or quite a
> bit) flawed :) If so, please educate me.

I hope I helped a little bit. Some of the design choices made in Spread
come from wanting to provide More then just reliable multicast, however
sometimes all you want is reliable multicast and you can live with it's
limitations (weak fault tolerance, no membership, no ordering)

If you have more questions or any of this is unclear, please email me (or
the list).

Jonathan

> Ciao
> 
> Charl

-- 
-------------------------------------------------------
Jonathan R. Stanton         jonathan at cs.jhu.edu
Dept. of Computer Science   
Johns Hopkins University    
-------------------------------------------------------