[Spread-users] Big messages in Spread

[Jacob Green]

We got around the big message problem (at least for the Java client
interface), by building a wrapper client layer around the java spread
interface.  Our messages are variables sizes and are often multiple
megabytes in size to as small as a few bytes.

The Java API stays the same, but the wrapper breaks up any size message into
sub messages, sends the sub messages, and then rebuilds the message on the
other side.  It works really well for FIFO and higher message types. Deals
with ordering and queuing of messages, and handles multiple simultaneous big
messages in transit from different senders.

I have the Java code if anyone wants it.

Jacob

> -----Original Message-----
> From: spread-users-admin at lists.spread.org
> [mailto:spread-users-admin at lists.spread.org]On Behalf Of Ben Laurie
> Sent: Friday, July 20, 2001 3:33 PM
> To: Jonathan Stanton
> Cc: Roel van der Goot; spread-users at lists.spread.org
> Subject: Re: [Spread-users] Big messages in Spread
>
>
> Jonathan Stanton wrote:
> >
> > On Fri, Jul 20, 2001 at 11:16:33AM -0600, Roel van der Goot wrote:
> > >
> > > Since we were encountering some issues with big messages, I found
> > > the following two things in Spread that should probably be changed.
> > > For completeness sake we are using fibrechannel datagrams that have a
> > > maximum of 964 bytes (MAX_PACKET_SIZE == 964):
> > >
> > >   - MAX_SCATTER_ELEMENTS is defined to be 100 in two files ("sp.h" and
> > >     "scatter.h").
> >
> > If your concern is that the same constant is defined twice, that is a
> > 'bug' although 'harmless' since they are the same number.
> Actually I think
> > the two uses of MAX_SCATTER_ELEMENTS are actually different and they do
> > not have to be the same number (but then they need to have different
> > names) The client interface could allow a different maximum number of
> > scatter elements in SP_scat_multicast and SP_scat_receive calls then the
> > daemon uses internally. The actual requirement is that the
> client does not
> > send a message larger to total size then then daemon can handle.
> >
> >  >
> > actually sp.c (from your second email)
> > >   - scatter.c uses 1440 as a constant instead of
> (MAX_PACKET_SIZE - 32)
> > >     where I don't know what the 32 represents. May be the 32 should
> > >     become a defined constant depending on its meaning.
> >
> > This is an oversight. 1440 should not be hardcoded in. The 32 bytes is
> > used to store the packet header. In the daemon code we do have a way to
> > get that number without hardcoding it. I'll see how the client
> should know
> > this.
> >
> > Here is the real question, which I would be interested in anyone's
> > opinions on.
> >
> > What should the maximum message size supported by a Spread
> > client library be?
>
> Ideally, as big as you can fit in memory (real or virtual). The
> application can decide how stupid it is to do that.
>
> > Should that size depend on the way the daemon is compiled (i.e.
> > MAX_PACKET_SIZE)?
>
> Only insofar as it isn't possible to avoid.
>
> Cheers,
>
> Ben.
>
> --
> http://www.apache-ssl.org/ben.html
>
> "There is no limit to what a man can do or how far he can go if he
> doesn't mind who gets the credit." - Robert Woodruff
>
>
> _______________________________________________
> spread-users mailing list
> spread-users at lists.spread.org
> http://lists.spread.org/mailman/listinfo/spread-users
>