[Spread-users] Question about thread-safety

White Stuart - stwhit Stuart.White at acxiom.com
Fri Jun 13 15:20:14 EDT 2003

Hi John,

Thanks for your comments.  I am #defining _REENTRANT and linking
libtspread.so.  I am not closing/opening multiple connections, so I don't
think I'm running into your file descriptor-reuse issue.

I decided to try locking a mutex around all SP_* calls to try to resolve the
problem, but it occurs to be that I cannot, because calls to SP_receive()
will block until a message is available.  If I lock a mutex around all SP_*
calls, you can see that this could easily create a deadlock situation.  A
call to SP_receive will block (with the mutex locked) until a message
becomes available, but none ever will, because the "sender" thread attempts
to lock the same mutex (and blocks) before he calls SP_multicast().

Perhaps I'm experiencing one of the non-user errors you mentioned, such as
socket failure.  Bummer.

Are there any resources/example programs which demonstrate correct spread
usage in threaded C applications?


-----Original Message-----
From: John Schultz [mailto:jschultz at d-fusion.net]
Sent: Friday, June 13, 2003 1:31 PM
To: spread-users at lists.spread.org
Subject: Re: [Spread-users] Question about thread-safety

NOTE: if anyone knows how to instruct Unix/Linux systems not to reuse 
file descriptor IDs in a process, could you please email me or the list 
a good reference (with page #)? Thanks!

Hi Stuart,

What you are proposing is sound provided that you #define _REENTRANT 
when compiling and link with Spread's thread safe library libtspread.a.

Currently, you can get spurious errors from Spread for the following 
reason: whenever there is a non-user error (socket failure, etc.) on a 
mailbox/socket, the Spread user library immediately closes and 
invalidates the mailbox/socket and returns CONNECTION_CLOSED.

Any subsequent SP call's on that mailbox/socket will return 
ILLEGAL_SESSION.  So, if your sender thread gets a CONNECTION_CLOSED 
your recv'er thread would very likely get an ILLEGAL_SESSION (or maybe a 
CONNECTION_CLOSED), and vice versa.  Just treat any such ILLEGAL_SESSION 
error as if it were a CONNECTION_CLOSED error.

Personally, I think that the Spread library should be modified to record 
any such error and return it for all subsequent SP calls on that 
mailbox.  Furthermore, the mailbox/socket should be invalidated/closed 
only upon the user calling SP_disconnect on it.

If your program is opening and closing multiple Spread connections then 
there is also a OS file descriptor reuse race condition that could be 
causing problems.  This race condition is best explained by example:

Imagine you have a sender thread (x) and a receiver thread (y) for 
mailbox/socket A and another thread (z) which is going to call 
SP_connect to create a mailbox/socket B.  Just before x starts writing a 
msg on A, y receives an error on A and therefore immediately 
closes/invalidates it. Next, z successfully performs SP_connect and is 
assigned mailbox/socket B, which happens to have the same value as A due 
to the OS reusing file descriptor IDs. Finally, y happily (and 
successfully) writes its msg for A on B not realizing that it is 
actually writing to a different Spread connection!

This behavior is obviously not correct! I'm not sure if this race 
condition exists on Windows but it definitely exists in Unix/Linux. I 
don't know if this problem can be reliably handled on the daemon side 
and I doubt if currently the daemon even tries to detect it.

The only way I can think of to avoid this race condition with the 
current Spread library is to instruct your OS not to reuse file 
descriptors IDs (see NOTE above).

If the Spread library is modified as I suggested above, then I think the 
race condition could be avoided by synchronizing calls to SP_connect and 

John Schultz
Co-Founder, Lead Engineer
D-Fusion, Inc. (http://www.d-fusion.net)
Phn: 443-838-2200 Fax: 707-885-1055

White Stuart - stwhit wrote:

> Hello all,
> I'm a spread newbie - just downloaded a few days ago.  My question is
> regarding using Spread in a threaded C application.
> I would like to write an application that can simultaneously send and
> receive messages.  In my main thread, I spawn another thread (the "message
> handler" thread) and he loops forever calling SP_receive().  The main
> loops, sending his messages via SP_multicast().
> My question is this:  I'm getting spurrious failures, (getting errors like
> ILLEGAL_SESSION from SP_receive) and I'm wondering what synchronization is
> appropriate for using spread in a threaded application?  Do I need to make
> sure only 1 thread is calling an SP_* function at-a-time?
> Thanks much!
> ______________________________________
> Stuart White
> Acxiom Corporation
> AbiliTec Architecture
> Email: stuart.white at acxiom.com
> Phone: 501.329.5975

Spread-users mailing list
Spread-users at lists.spread.org

The information contained in this communication is
confidential, is intended only for the use of the recipient
named above, and may be legally privileged.
If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, 
distribution, or copying of this communication is strictly
If you have received this communication in error,
please re-send this communication to the sender and
delete the original message or any copy of it from your
computer system. Thank You.

More information about the Spread-users mailing list