[Spread-users] chunking messages?

[John Schultz]

Guido van Rossum wrote:

 >>For example, suppose you send a large Safe message. The first part
 >>is received by everyone but then some people partition and don't receive
 >>all of of the parts. As a result, they discard the message.
 >>Meanwhile, some others get all the parts and deliver the message
 >>before the transitional signal.
 >>
 >
 > Wouldn't it be sufficient to send all parts as Safe messages?
 >
 > --Guido van Rossum (home page: http://www.python.org/~guido/)
 >
 >
 >
I was thinking along the same lines, but I think you would also need to 
fiddle with transitional signals and transitional sets. After I went 
through the thinking laid out below, it probably gets too ugly to try 
and overcome. Do the proposed heuristics maintain AGREED semantics properly?

Here's what I thought:

Let's first assume that all chunks of a message are sent with the same 
service as requested by the user for the big message.

Let's consider SAFE messages first:

** Scenario 1: A SAFE multi-msg that is delivered entirely inside one 
membership, potentially straddling a transitional signal at different 
members.

A receiver receives all of the chunks during a single membership before 
any transitional signal. This implies to that receiver that all members 
of his current membership will also deliver all of the chunks or crash.

Another member of the same membership receives all of the SAFE chunks 
but receives some (or all) of them after a transitional signal in the 
membership. This implies to him that all members of his survivors set of 
the following membership will deliver all of the chunks or crash. 
Therefore, he must deliver the big message after a transitional signal 
in the current membership. So he must generate a transitional signal 
before he delivers the multi-msg.

** Scenario 2: A SAFE multi-msg that is delivered across more than one 
membership, and the membership(s) subtract members from the membership. 
For example, the LEAVE membership of a member of the group is ordered in 
the middle of a multi-msg.

The current heuristics require us to deliver the multi-msg before we 
deliver the leave of the member. To make it correct we would have to 
transform the LEAVE into a NETWORK membership (with a surivors set equal 
to the new membership) and generate a transitional signal before 
delivering the multi-msg.

More complex scenarios of several subtractive memberships occurring 
during the course of delivering chunks of a multi-msg would have to be 
handled by computing the appropriate survivors set(s) as follows: 
initially it is the set of members of the membership in which the 
leading chunk of the msg was delivered. If any subtractive membership 
occurs before the multi-msg is completed one transitional signal would 
be generated and delivered before the multi-msg, and each leaving member 
would be subtracted from the survivors set. Then the first membership 
that follows the delivery of the multi-msg would be a NETWORK membership 
with the computed survivors set.

So you probably could maintain Spread's semantics, the code would just 
have to be much smarter. It would have to potentially track the status 
of each membership (several can be pending), compute transitional sets 
appropriately for each one and generate transitional signals appropriately.

-- 
John Schultz
Co-Founder, Lead Engineer
D-Fusion, Inc. (http://www.d-fusion.net)
Phn: 443-838-2200 Fax: 707-885-1055