[Spread-users] chunking messages?

Yair Amir yairamir at cnds.jhu.edu
Thu May 23 17:12:45 EDT 2002

Hi John,

I am not sure the proposed algorithm for large messages maintains the
Spread semantics in all cases.

For example, suppose you send a large Safe message. The first part
is received by everyone but then some people partition and don't receive
all of of the parts. As a result, they discard the message.
Meanwhile, some others get all the parts and deliver the message
before the transitional signal.

Nevertheless, some people might want to use it understanding the
limitations. Look at Spread's development page regarding how to
contribute. We could put that code there just like the current
multipath code by Marc Zyngier. http://www.spread.org/devel.html

          :) Yair.

p.s. I did not see a quick fix to the above problem.

John> Yair, Jonathan: Jacob and I wrote a Java subclass to SpreadConnection 
John> that transparently allows arbitrarily sized messages while maintaining 
John> Spread's safety properties. We'd like to include this in the Spread 
John> distribution if you agree. How does the IP work for code like this?

John> Once I hear from the Spread guys, I'll post the code to this list.

John> Quentin,

John> My company ran into a similiar problem as yours, so we wrote a small 
John> (Java) extension to SpreadConnection which transparently supports 
John> arbitrarily sized msgs (I think it is restricted by Java arrays to a max 
John> of 2^31 - 1 bytes). In your code just use MMSpreadConnection instead of 
John> SpreadConnection. Be sure all processes in the same group use one or the 
John> other, or you will be on the bullet train to Bugsville :)  If you aren't 
John> using Java, then you could use the code as a basis to write a C/C++, 
John> Perl, Python or whatever floats your boat, version.

John> This code is a recent (today) rewrite of a previous version and so it 
John> hasn't seen much testing yet (just what is in the main() fcn). I'll be 
John> happy to fix any problems you or we run into.

John> Here are the semantics:

John> Messages larger than 80K are broken into smaller msgs of 80K or less. 
John> The first msg is sent with the service type (FIFO, CAUSAL, etc.) 
John> requested by the user, the rest of the chunked msgs are sent in order as 
John> FIFO msgs until the entire data payload is sent. NOTE: No flow control 
John> is performed as the msg is sent so if you send REALLY big messages (10Mb 
John> or larger) then you will flood Spread and potentially cause disconnects 
John> of your receivers for not reading from Spread fast enough!

John> On the receiving end, msgs are put into a queue as they are received. If 
John>   a multi-msg is at the head of the queue then all other messages will 
John> be queued and delivered until after that multi-msg is received in its 
John> entirety and delivered.  If the sender of a multi-msg leaves before his 
John> entire msg is received then that MMsg is dropped.  If a receiver comes 
John> in during the middle of a multi-msg (doesn't get the first msg) it just 
John> drops all of those chunks. I'm pretty sure these heuristics maintain all 
John> of Spread's safety properties.

John> MMSpreadConnection is a transparent replacement for SpreadConnection, 
John> you don't have to do anything special, except be sure that all processes 
John> in a given group use MMSpreadConnection and that every process is a 
John> member of any group to which it sends.

John> PS - I used this class with 3.16.0 and haven't tried it with 3.16.2 yet, 
John> but I don't imagine any difficulties.

John> -- 
John> John Schultz
John> Co-Founder, Lead Engineer
John> D-Fusion, Inc. (http://www.d-fusion.net)
John> Phn: 443-838-2200 Fax: 707-885-1055

More information about the Spread-users mailing list