[Spread-users] spread daemon as library?

[Ryan Caudy]

The main design consideration that makes the client-daemon
architecture in Spread a vital choice is performance/scalability of
network communication.

By having only a handful of daemons interacting on behalf of their
clients for the EVS algorithm, most systems can avoid a great deal of
redundant communication.

In your case, it seems like the benefit you're attempting to gain is
faster client-daemon communication than Unix domain sockets, and fewer
processes, etc.

In order to package the daemon as a library, for a general purpose
application the daemon software would need its own thread, and would
need to be scheduled at a fairly high priority to avoid potential
hiccups in the communication.  You could easily replace the
communication from the daemon code to the client library with an
in-memory queue protected by a mutex and condition variable.

If you intended to use the Spread toolkit's event-handling system, you
might be able to avoid some of the synchronization overhead, etc. 
However, you'd have to be careful to avoid potential conflicts with
the timing of Spread's behavior.

For either implementation, you could avoid some of the
packing/unpacking code for an added performance gain, but this would
require somewhat deeper knowledge of the Spread daemon's internals and
system guarantees.

To the best of my knowledge, this idea isn't entirely goofy... I think
many other GCS toolkits are implemented as libraries.  However, I'm
not particularly aware of the techniques employed.

I'd be careful to avoid investing a huge amount of time in this
effort.  In your specific case of one client per network node, you may
be able to improve communication latency by a only a very small
fraction when compared to the time for network transmission and any
artificial delays imposed while the daemon waits for delivery
guarantees to be met.  As I mentioned above, there's also a risk of
preventing the daemon from processing its scheduled or message events
in a timely fashion, potentially delaying message delivery or
triggering spurious membership events.  The last issue should only be
a concern with finely-tuned Spread network timeouts, or with
client-level code that holds resources for a long period of time.

Cheers,
Ryan

On 4/17/06, Kim Barrett <kab at irobot.com> wrote:
> We are developing a system that involves network communication among
> multiple processors, and in many ways spread looks like a good fit
> for us.
>
> However, one way in which it might not be ideal is the presumption of
> a spread daemon process which other processes on a given processor
> communicate with. It turns out that in our system the vast majority
> (probably all on some processors) of local client communication with
> the spread daemon on a given processor would be from a single process
> on that same processor. In the interest of reducing context switch
> and IPC overhead, we are starting to look at the possibility of
> making the daemon into a library that can be directly linked against
> by that one program of ours.
>
> It looks like the initialization of the daemon in main() in
> daemon/spread.c is pretty straightforward, and could be repackaged as
> a function to be called from our program. The hard part appears to be
> replacing the network plumbing between the client-side interface and
> the daemon's processing with a direct in-memory connection.
>
> Before possibly spending a lot of time going down this path, we're
> soliciting opinions from folks more familiar with spread as to
> whether this might actually be worthwhile or not, or whether it is
> perhaps just plain goofy to try this.
>
>
> _______________________________________________
> Spread-users mailing list
> Spread-users at lists.spread.org
> http://lists.spread.org/mailman/listinfo/spread-users
>