[Spread-users] Big messages in Spread

[Jonathan Stanton]

On Fri, Jul 20, 2001 at 09:33:24PM +0100, Ben Laurie wrote:
> Jonathan Stanton wrote:
> > The problem is that not only this client has to store the message in
> > memory, but also all of the daemons and any other clients who are
> > receiving it. Also, since many messages may be in transit (or buffers) at
> > once, the actual memory usage is more like C * N * max_message_size, where
> > C is the number of clients on one machine and N is the number of buffered
> > messages. And one client's large messags interfere with the service given
> > to other clients who may only be using small messages.
> 
> Like I say, the application can decide how stupid that is.

I agree applications can choose to be stupid. Ok. 

However, one of the issues I tried to identify but I don't think I was
very clear about, is that there may be several DIFFERENT applications
using one set of Spread daemons. In which case I want to protect
applications from other applications. Now, maybe most current uses of
spread assume a complete daemon setup for each client, but I do not want
that to be a requirement.

At minimum you may have different priority
applications that make up the 'suite' of a bigger service (think an admin
control program that sends small messages and a user level file transfer
program that sends large files). They may be written by the same people
(in which case those people could choose to use small messages for the
file transfer even though they don't need to so their control messages are
still fast). 

But if everyone doesn't cooperate fully and understand these
issues, they will have problems -- if the max message size is fixed at a
'reasonable' number then they will have the problem of 'small' delays, but
everything will work. If they use arbitrarily large mesages, the problem
becomes 'long' blocked periods and things seem to not work.

I know neither is absolutely right or wrong, but a reasonable cap seems
better overall. 

The major issue though is NON-cooperative clients. So the daemon has to
arbitrate between them.

 > 
> > Also, most datagram protocols have a 'max message size' for practical
> > reasons, cause if you don't then what is a 'datagram'?
> 
> A datagram is not something Spread sends. The reason for the max message
> size in a datagram is because the underlying protocol is not reliable,
> so datagrams cannot be fragmented. Spread does not have that limitation.

Even unreliable datagrams can be fragmented, at the cost of a higher risk
of datagram loss. But I agree Spread can fragment (and does). 

> 
> > An issue that does not come up with 'unordered' delivery is that messages
> > that are ordered after a largemessage will be delayed until the large
> > message is fully received (which could be arbitrarily long if 'unlimited'
> > size messages are allowed). This is still true even with message size
> > limits (a 1 byte message is delayed by a 80Kbyte message in front of it)
> > but the size limit is chosen to 'work well in practice' (whatever that
> > means...since it depends on network technology, machine power, types of
> > messges, etc)
> 
> Once more, it is up to the application to decide how stupid this is (as
> an aside, is the ordering of a large message deferred for as long as
> possible?).

I'm not sure I get the question, but all messages are deferred for as
"SHORT" as possible (not as long). The ordering assigned to a message is
known at creation time (and on receive side is known as soon as the head
fragment is received). When the entire message is received, it is
delivered as soon as it can be 'correctly' (so if it is 'agreed' order and
some other message is still missing before it, we cannot deliver it until
that other message arrives)

 > 
> > > > Should that size depend on the way the daemon is compiled (i.e.
> > > > MAX_PACKET_SIZE)?
> > >
> > > Only insofar as it isn't possible to avoid.
> > 
> > Ok. I agree, but it probably isn't possible to avoid unless you want the
> > daemon to end up rejecting messages that the client thought were of ok
> > size because the daemon has a smaller limit.
> 
> Not clear why the daemon would have a limit (that could be smaller than
> the clients).

It is true the daemon could allow 'arbitrarily large' messages. We
currently implment it with a fixed size array of 'fragments' and each
fragment has a max size so the size of message a daemon can handle is
limited at compile time. A linked list representation of fragments would
not have this limitation, but would be more costly to process.

Jonathan
-- 
-------------------------------------------------------
Jonathan R. Stanton         jonathan at cs.jhu.edu
Dept. of Computer Science   
Johns Hopkins University    
-------------------------------------------------------