[Spread-cvs] commit: r314 - trunk/docs
jonathan at spread.org
jonathan at spread.org
Tue Feb 21 00:26:22 EST 2006
Author: jonathan
Date: 2006-02-21 00:26:22 -0500 (Tue, 21 Feb 2006)
New Revision: 314
Added:
trunk/docs/Flush_or_SpreadAPI.txt
Modified:
trunk/docs/DynamicConfiguration.txt
trunk/docs/MultithreadedClients.txt
Log:
Add documentation comparing Flush Spread API and the Core Spread API
Reformat text docs to be line wrapped.
Modified: trunk/docs/DynamicConfiguration.txt
===================================================================
--- trunk/docs/DynamicConfiguration.txt 2006-02-21 03:48:05 UTC (rev 313)
+++ trunk/docs/DynamicConfiguration.txt 2006-02-21 05:26:22 UTC (rev 314)
@@ -1,79 +1,200 @@
Motivation:
-Spread prior to version 4 required a configuration file specifying all of the daemons who would ever be part of the spread configuration and to make changes to the set of potential daemons required that all of the daemons be shut down and restarted once the configuration file was edited to include the changes. This made adding servers to a pool or changing the name or IP of an existing server require a full restart of the system which can be disruptive.
+Spread prior to version 4 required a configuration file specifying all
+of the daemons who would ever be part of the spread configuration and
+to make changes to the set of potential daemons required that all of
+the daemons be shut down and restarted once the configuration file was
+edited to include the changes. This made adding servers to a pool or
+changing the name or IP of an existing server require a full restart
+of the system which can be disruptive.
Solution:
-Spread in version 4 can now handle changes to the spread.conf daemon configuration without shutting down and restarting all of the daemons. Daemons can be added to segments or removed without existing clients losing their spread connections. Some changes to the spread configuration (such as changing the name of an existing daemon) will cause the clients to receive a view change message showing a temporary change in the membership. This feature is called Dynamic Configuration.
+Spread in version 4 can now handle changes to the spread.conf daemon
+configuration without shutting down and restarting all of the
+daemons. Daemons can be added to segments or removed without existing
+clients losing their spread connections. Some changes to the spread
+configuration (such as changing the name of an existing daemon) will
+cause the clients to receive a view change message showing a temporary
+change in the membership. This feature is called Dynamic
+Configuration.
Equal Configuration Enforcement:
-The Spread system has always had a requirement that the daemon configurations specified in spread.conf at all of the node were identical. However earlier versions of Spread did not enforce that requirement automatically. Now Spread enforces that all participating daemons are using an identical configuration.
+The Spread system has always had a requirement that the daemon
+configurations specified in spread.conf at all of the node were
+identical. However earlier versions of Spread did not enforce that
+requirement automatically. Now Spread enforces that all participating
+daemons are using an identical configuration.
-Spread enforces this property by having all packets sent between daemons contain an identification code in the message header for the configuration that was active when they were sent. When a packet is received, if the packet configuration code does not match the current configuration code of the receiving daemon, then the packet is discarded. If one daemon switches configurations, it cannot communicate with any other daemon that still uses the old configuration. If the other daemons do not also switch, then they will be partitioned until the configuration error is repaired.
+Spread enforces this property by having all packets sent between
+daemons contain an identification code in the message header for the
+configuration that was active when they were sent. When a packet is
+received, if the packet configuration code does not match the current
+configuration code of the receiving daemon, then the packet is
+discarded. If one daemon switches configurations, it cannot
+communicate with any other daemon that still uses the old
+configuration. If the other daemons do not also switch, then they will
+be partitioned until the configuration error is repaired.
-The identification code is implemented as a 32 bit hash of the relevant sections of the Spread.conf file.
+The identification code is implemented as a 32 bit hash of the
+relevant sections of the Spread.conf file.
Dynamic Configuration Client Experience:
-When a new configuration is loaded, application groups may experience a membership change. The change may appear spurious, meaning no actual change in the list of members occurs, but a new group_id or view_id is generated and a membership message is received. The application may also see a membership change that shows all non-local clients (those connected to a different daemon) leaving the group and then a second membership messages showing all of the clients rejoining the group (possibly with some missing if they were connected to a daemon that is no longer valid in the new configuration file).
+When a new configuration is loaded, application groups may experience
+a membership change. The change may appear spurious, meaning no actual
+change in the list of members occurs, but a new group_id or view_id is
+generated and a membership message is received. The application may
+also see a membership change that shows all non-local clients (those
+connected to a different daemon) leaving the group and then a second
+membership messages showing all of the clients rejoining the group
+(possibly with some missing if they were connected to a daemon that is
+no longer valid in the new configuration file).
-No changes to any user-visible data structures or functions are necessary for this.
+No changes to any user-visible data structures or functions are
+necessary for this.
How to activate a Dynamic Configuration Change:
The process of executing a configuration changes is as follows:
-1) The admin runs the spmonitor program.
- It is necessary to do this first so that the spmonitor program reads in the list of possible daemons from the old spread.conf file. It needs to know what daemons were in the old configuration in order to send them the notification to reread the spread.conf file.
+1) The admin runs the spmonitor program. It is necessary to do this
+ first so that the spmonitor program reads in the list of possible
+ daemons from the old spread.conf file. It needs to know what
+ daemons were in the old configuration in order to send them the
+ notification to reread the spread.conf file.
-2) The admin updates the Spread.conf file on disk with the new contents. They then copy that updated file to all of the hosts running daemons who are in the old or new configuration.
+2) The admin updates the Spread.conf file on disk with the new
+ contents. They then copy that updated file to all of the hosts
+ running daemons who are in the old or new configuration.
-3) The admin chooses the spmonitor option "Reload Configuration".
- This causes a command packet to be sent by the spmonitor program to all daemons in the old configuration file. The command packet triggers the daemon to start the reload process. The internal details of the daemon algorithm are described below. At this point all of the daemons will switch to the new configuration.
+3) The admin chooses the spmonitor option "Reload Configuration".
+ This causes a command packet to be sent by the spmonitor program to
+ all daemons in the old configuration file. The command packet
+ triggers the daemon to start the reload process. The internal
+ details of the daemon algorithm are described below. At this point
+ all of the daemons will switch to the new configuration.
-4) Once the reload is done, all client applications may receive a membership change notification message.
- The message will be generated if any groups the client is in have changed the set of members (because a daemon is no longer in the membership and had clients in that group) or, for certain types of configuration changes, all groups with members connected to more then one daemon will receive a membership change showing all group members from remote daemons leaving the group, and then a second membership showing the group with all clients still connected to running daemons back together.
+4) Once the reload is done, all client applications may receive a
+ membership change notification message. The message will be
+ generated if any groups the client is in have changed the set of
+ members (because a daemon is no longer in the membership and had
+ clients in that group) or, for certain types of configuration
+ changes, all groups with members connected to more then one daemon
+ will receive a membership change showing all group members from
+ remote daemons leaving the group, and then a second membership
+ showing the group with all clients still connected to running
+ daemons back together.
Semantics of a Dynamic Configuration Change:
-For the basic case, the algorithm looks like a regular membership in which certain daemons have crashed or partitioned (those who are in the old configuration and not in the new configuration) and some daemons have started up or are remerging (those in the new configuration but not in the old).
+For the basic case, the algorithm looks like a regular membership in
+which certain daemons have crashed or partitioned (those who are in
+the old configuration and not in the new configuration) and some
+daemons have started up or are remerging (those in the new
+configuration but not in the old).
-Crashed/partitioned daemons do not have any group state we need to track since we will remove all of their group members from the current membership view. Therefore, we can just ignore them as in a normal membership. The new daemons that have been added to the configuration will merge any group state they have during the normal Extended Virtual Synchrony protocol as if they had only been partitioned away. Since group members are identified with the daemons they are connected to, there cannot be any inconsistent state between two daemons, as they will each represent their own state in the group membership protocol.
+Crashed/partitioned daemons do not have any group state we need to
+track since we will remove all of their group members from the current
+membership view. Therefore, we can just ignore them as in a normal
+membership. The new daemons that have been added to the configuration
+will merge any group state they have during the normal Extended
+Virtual Synchrony protocol as if they had only been partitioned
+away. Since group members are identified with the daemons they are
+connected to, there cannot be any inconsistent state between two
+daemons, as they will each represent their own state in the group
+membership protocol.
-A special case occurs if the same IP address is in two different configuration files with an inconsistency (different name, changed broadcast address, etc.) When that case is identified, a singleton partition is created. The partition will separate each daemon from all other daemons and cause their group state to shrink to only the locally connected client applications. Then, when the singleton partition is removed, they will merge with all of the other daemons. Since each daemon will only know its own locally connected client group members, merging the partitions will bring the system to a consistent state.
+A special case occurs if the same IP address is in two different
+configuration files with an inconsistency (different name, changed
+broadcast address, etc.) When that case is identified, a singleton
+partition is created. The partition will separate each daemon from all
+other daemons and cause their group state to shrink to only the
+locally connected client applications. Then, when the singleton
+partition is removed, they will merge with all of the other
+daemons. Since each daemon will only know its own locally connected
+client group members, merging the partitions will bring the system to
+a consistent state.
-Any daemon whose network specification has changed between the old and new configuration files will have to shut down and be restarted. This is because the Spread daemon does not re-bind to changed broadcast, multicast or IP unicast addresses after starting up. So changing the IP address of a machine will require the daemon on that machine to be shut down and then restarted with the new configuration, however other daemons who are active will not shut down and will continue to work over the configuration change. Only the machine with the changed network configuration will shut down.
+Any daemon whose network specification has changed between the old and
+new configuration files will have to shut down and be restarted. This
+is because the Spread daemon does not re-bind to changed broadcast,
+multicast or IP unicast addresses after starting up. So changing the
+IP address of a machine will require the daemon on that machine to be
+shut down and then restarted with the new configuration, however other
+daemons who are active will not shut down and will continue to work
+over the configuration change. Only the machine with the changed
+network configuration will shut down.
Daemon Algorithm Details:
-When a spmonitor command to reload configuration is received, the daemon executes the following steps. (See function Prot_handle_conf_reload() for the code).
+When a spmonitor command to reload configuration is received, the
+daemon executes the following steps. (See function
+Prot_handle_conf_reload() for the code).
-1) Call function in configuration code to load new configuration file and examine the type of changes to the configuration. If the configuration changes in the new file require a singleton partition (see below for details) then indicate that to protocol code.
+1) Call function in configuration code to load new configuration file
+ and examine the type of changes to the configuration. If the
+ configuration changes in the new file require a singleton partition
+ (see below for details) then indicate that to protocol code.
-2) Inform all subsystems in the daemon that a configuration change is in progress and any caches or data structures that are storing the current state of the daemon configuration should be reloaded/updated with the new information. The old configuration can still be accessed through a different name until the reload is complete. This is done by a calling functions called "*_signal_conf_reload()" where the * is replaced by the name of the subsystem. For example "Net_signal_conf_reload()" and "Memb_signal_conf_reload()".
+2) Inform all subsystems in the daemon that a configuration change is
+ in progress and any caches or data structures that are storing the
+ current state of the daemon configuration should be
+ reloaded/updated with the new information. The old configuration
+ can still be accessed through a different name until the reload is
+ complete. This is done by a calling functions called
+ "*_signal_conf_reload()" where the * is replaced by the name of the
+ subsystem. For example "Net_signal_conf_reload()" and
+ "Memb_signal_conf_reload()".
-3a) If no partition is needed, then initiate a membership change (with token_loss) and return. The membership protocol will run to completion as normal by probing for all active daemons and forming a new membership including any new daemons and removing any daemons who are no longer in the configuration file.
+3a) If no partition is needed, then initiate a membership change (with
+ token_loss) and return. The membership protocol will run to
+ completion as normal by probing for all active daemons and forming
+ a new membership including any new daemons and removing any daemons
+ who are no longer in the configuration file.
-3b) If a partition is needed, then a singleton partition is created, the Conf_reload_state variable is set so other parts of the daemon know a configuration reload is in progress, and a membership change is triggered (by a scheduled token_loss).
+3b) If a partition is needed, then a singleton partition is created,
+ the Conf_reload_state variable is set so other parts of the daemon
+ know a configuration reload is in progress, and a membership change
+ is triggered (by a scheduled token_loss).
-4) When membership completes in the Discard_packets() function, if a configuration reload state is active, then the partition is removed and a probe for new members (to find all of the now 'non-partitioned' daemons) is started. Once this second membership change takes place, then the configuration reload is complete.
+4) When membership completes in the Discard_packets() function, if a
+ configuration reload state is active, then the partition is removed
+ and a probe for new members (to find all of the now
+ 'non-partitioned' daemons) is started. Once this second membership
+ change takes place, then the configuration reload is complete.
-The test for whether a partition is needed for a new configuration is below. See function Conf_reload_initiate() in configuration.c for the implementation details.
+ The test for whether a partition is needed for a new configuration is
+ below. See function Conf_reload_initiate() in configuration.c for the
+ implementation details.
-a) If any of the following are true then the daemon process will quit as it should no longer be executing in the new configuration file.
+ a) If any of the following are true then the daemon process will quit
+ as it should no longer be executing in the new configuration file.
+
1) I am no longer in the configuration.
+
2) My IP/Name has changed.
+
3) My broadcast/netmask has changed.
-b) Otherwise examine all entries that are both in the new configuration and in the old configuration (matching based on IP address).
-If the same IP address is in both configurations then if any of the following differs between the old and new configuration, then a partition is required. If they are the same for ALL hosts that are in both files, then no partition is needed.
+ b) Otherwise examine all entries that are both in the new
+ configuration and in the old configuration (matching based on IP
+ address). If the same IP address is in both configurations then if
+ any of the following differs between the old and new configuration,
+ then a partition is required. If they are the same for ALL hosts that
+ are in both files, then no partition is needed.
+
1) Name.
+
2) Number of interfaces.
+
3) Broadcast address or netmask.
+
4) Any interface specifications have changed.
-Note that in the common cases of simply adding and removing daemons, no partition is needed.
+ Note that in the common cases of simply adding and removing daemons,
+ no partition is needed.
Added: trunk/docs/Flush_or_SpreadAPI.txt
===================================================================
--- trunk/docs/Flush_or_SpreadAPI.txt (rev 0)
+++ trunk/docs/Flush_or_SpreadAPI.txt 2006-02-21 05:26:22 UTC (rev 314)
@@ -0,0 +1,108 @@
+A Comparison of the Flush and Spread API's and Semantics
+--------------------------------------------------------
+
+The Flush Spread API is an extension to the Spread Wide Area Group
+Communication System. Flush Spread and Core Spread are extremely
+similiar group communication systems (GCSs), in the services that they
+provide, and in their general interface. Therefore, reading Spread's
+documentation before reading Flush Spread's documentation is highly
+recommended to get a general grasp of group communication and to fill
+in any holes in this documentation.
+
+There are two main differences between Spread and Flush Spread: (1)
+Spread provides extended virtual synchrony GCS semantics while Flush
+Spread provides view synchrony GCS semantics, and (2) Spread provides
+open-group semantics while Flush Spread only provides closed-group
+semantics.
+
+Extended Virtual Synchrony (EVS) vs. View Synchrony Semantics (VS)
+------------------------------------------------------------------
+
+The main difference between these two models is how they handle
+view/membership changes. In an EVS system, membership changes occur
+without a client's intervention, whereas in a VS system a client must
+give its permission before a new view/membership can be installed. Of
+course, in "real-life" a VS system cannot truly hold-off membership
+changes (as other clients can crash, network can partition, etc.), but
+what it can do is insulate a client from these changes and not allow
+new members (who either joined or merged) or their messages to be
+presented to the client without its permission.
+
+In a VS system, when an underlying membership change occurs (another
+member joins/leaves/disconnects, network parititions or merges) the
+GCS generates a flush request message indicating that the current view
+is out-of-date and requesting permission to install a new view. The
+client is still allowed to send messages at this point, and they may
+be able to receive messages from surviving members. When the client is
+ready, it flushes the group, which gives the GCS permission to install
+a new view. After flushing, a client is not allowed to send any
+messages to the flushed group until they receive the new view of the
+group.
+
+In a EVS system, when an underlying membership change occurs the GCS
+figures out the new membership and delivers a new view to the client
+in its normal stream of messages.
+
+In both systems, using the safety guarantees of the different message
+types and the virtual synchrony property (i.e. transitional sets) the
+client can infer some information about what other members have seen
+without further communication.
+
+Although the difference in these models seems slight, it can lead to
+quite astonishing differences in performance and ease of
+programming. In general, the EVS model is faster and scales better
+with respect to membership changes, especially for simple group
+changes such as joins and leaves. But this performance comes at a
+price, namely that the client has less control over what is
+transpiring and therefore in general it is harder to program
+algorithms under EVS than under VS. For example, under EVS the sender
+doesn't know which other clients might receive its messages until the
+message is delivered back to the sender. Also, membership changes can
+come fast and furious and your algorithm must be tolerant to every
+combination of such changes. Under VS, the client knows exactly which
+other clients might receive its message when it sends -- so the
+application doesn't have to worry about some arbitrary client
+receiving its messages and mis-interpreting them, for example. Also,
+membership changes only occur as fast as the applications want them
+to, although several underlying memberships may be collapsed into
+one. So they are generally easier to handle and track.
+
+Open-Group vs. Closed-Group Semantics
+-------------------------------------
+
+Open-group semantics allows clients to perform group specific calls,
+such as multicasts, without being a member of the group in
+question. Closed-group semantics requires a client to be a full
+fledged member of a group (see below) in order to perform group
+specific calls. In general, a closed-group GCS is much more picky than
+an open-group system about a client's state and the operations they
+perform with respect to any particular group.
+
+Under Spread, a client can call group specific calls such as join,
+leave, and *cast (multicast and its variants) pretty much whenever
+they want regardless of their membership in that group. Also, under
+Spread the guarantees of multicast messages span differerent
+groups. For example, if a sender sends two FIFO messages to two
+different groups and a receiver receives both messages, they will
+always receive the first and then the second FIFO message.
+
+Under Flush Spread, a client can make group specific calls such as
+join, leave, flush, and *cast only when in the proper membership state
+with respect to that group. For example, leave, flush, and *cast calls
+can only be made when the client is a full-fledged member of the
+group. A full-fledged member is a client that has joined a group,
+received a membership including them in the group and is not in the
+process of leaving the group. Join calls can only be made when the
+client is not a member of the group (either never joined it or whose
+most recent membership for that group was a self-leave) and not
+already in the process of joining that group.
+
+Also under Flush Spread, *cast and flush calls (i.e. FL_flush) are
+restricted depending on the group membership state. Flush calls can
+only be made ONCE per view/membership in response to receiving a flush
+request message. Once a client flushes a group they are not allowed to
+*cast to that group until they receive the next view/membership for
+that group. Flush Spread also does not support cross group message
+semantics. For example, if a sender sends two FIFO messages to two
+different groups and a receiver receives both messages, they may
+receive the first and then the second OR vice-versa.
Modified: trunk/docs/MultithreadedClients.txt
===================================================================
--- trunk/docs/MultithreadedClients.txt 2006-02-21 03:48:05 UTC (rev 313)
+++ trunk/docs/MultithreadedClients.txt 2006-02-21 05:26:22 UTC (rev 314)
@@ -1,33 +1,123 @@
-Spread specific help with writing Multi-Threaded and Multi-Process client applications
---------------------------------------------------------------------------------------
+ Spread specific help with writing
-The Spread system supports both single-threaded client applications and multi-threaded client applications. As much as possible it tries to provide both types with high-efficiency interfaces and API's. However, writing multi-threaded applications entails some unavoidable increase in complexity, both at the API level and with performance considerations and so this document attempts to explain how best to utilize Spread from multi-threaded applications.
+ Multi-Threaded and Multi-Process client applications
+--------------------------------------------------------------------
-Spread exports several different APIs for client applications. The core Spread API provides extended virtual synchrony semantics and is defined in the sp.h header and consists of functions beginning with the SP_ prefix. The Flush Spread API provides Virtual Synchrony semantics and is defined in the fl.h header and consists of functions beginning with the FL_ prefix. Both APIs can be used by multithreaded applications and are written to be 'thread-safe' (not taking into account some of the specific semantic limitations explained below).
+The Spread system supports both single-threaded client applications
+and multi-threaded client applications. As much as possible it tries
+to provide both types with high-efficiency interfaces and
+API's. However, writing multi-threaded applications entails some
+unavoidable increase in complexity, both at the API level and with
+performance considerations and so this document attempts to explain
+how best to utilize Spread from multi-threaded applications.
+Spread exports several different APIs for client applications. The
+core Spread API provides extended virtual synchrony semantics and is
+defined in the sp.h header and consists of functions beginning with
+the SP_ prefix. The Flush Spread API provides Virtual Synchrony
+semantics and is defined in the fl.h header and consists of functions
+beginning with the FL_ prefix. Both APIs can be used by multithreaded
+applications and are written to be 'thread-safe' (not taking into
+account some of the specific semantic limitations explained below).
+
Linking and Libraries:
-For applications that are single-threaded and do not need, or want, threaded locks around shared datastructures, a 'non-thread-safe' version of the core Spread API's is available in the libspread-core library. All multi-threaded applications should use either the main libspread library which contains all of the API's in their thread-safe form, or the libtspread-core library for only the core Spread API's in thread-safe form. (Note the "t" character in the library name)
+For applications that are single-threaded and do not need, or want,
+threaded locks around shared datastructures, a 'non-thread-safe'
+version of the core Spread API's is available in the libspread-core
+library. All multi-threaded applications should use either the main
+libspread library which contains all of the API's in their thread-safe
+form, or the libtspread-core library for only the core Spread API's in
+thread-safe form. (Note the "t" character in the library name)
-The rest of this document will assume you do want the multi-thread safe libraries and will use either the VS or EVS Spread API's.
+The rest of this document will assume you do want the multi-thread
+safe libraries and will use either the VS or EVS Spread API's.
-To link your application code with Spread you will need to compile your code with the platform appropriate _REENTRANT define enabled, and will need to link with the pthreads library on unix systems as well as the libspread or libtspread-core library. On Windows you need to link with the multi-thread safe windows system libraries just like any normal Windows threaded program.
+To link your application code with Spread you will need to compile
+your code with the platform appropriate _REENTRANT define enabled, and
+will need to link with the pthreads library on unix systems as well as
+the libspread or libtspread-core library. On Windows you need to link
+with the multi-thread safe windows system libraries just like any
+normal Windows threaded program.
Spread Locking Semantics:
-Multi-threaded programs can use separate threads to receive messages on different connections, send messages to connections, process application messages, etc. The Spread library's internal locking will protect the library data structures from concurrent access, will provide serialization of SP_receive family calls so only one thread receives each message, and will enforce that each message send occurs atomically on a connection. If more then one thread is sending or receiving from the same connection, then the ordering of messages sent or received can not be determined because it depends on the order in which the processes are granted the shared lock for the connection.
+Multi-threaded programs can use separate threads to receive messages
+on different connections, send messages to connections, process
+application messages, etc. The Spread library's internal locking will
+protect the library data structures from concurrent access, will
+provide serialization of SP_receive family calls so only one thread
+receives each message, and will enforce that each message send occurs
+atomically on a connection. If more then one thread is sending or
+receiving from the same connection, then the ordering of messages sent
+or received can not be determined because it depends on the order in
+which the processes are granted the shared lock for the connection.
-If several threads call SP_recv on the same connection, then they will all be blocked until messages arrive and then each one will read one message from the connection, the order in which they receive the messages is undetermined as they are all blocked on a shared lock. Thus normally if you care about the message ordering or safety semantics, only one thread will be assigned to call SP_receive for each connection. After it receives the messages, it can make an application level decision to dispatch other threads to process the message contents based on the type of message and what semantics are enforced.
+If several threads call SP_recv on the same connection, then they will
+all be blocked until messages arrive and then each one will read one
+message from the connection, the order in which they receive the
+messages is undetermined as they are all blocked on a shared
+lock. Thus normally if you care about the message ordering or safety
+semantics, only one thread will be assigned to call SP_receive for
+each connection. After it receives the messages, it can make an
+application level decision to dispatch other threads to process the
+message contents based on the type of message and what semantics are
+enforced.
-In the same way if multiple threads call SP_mcast functions on the same connection, each message will be sent out correctly, but the order in which they are sent is indeterminate.
+In the same way if multiple threads call SP_mcast functions on the
+same connection, each message will be sent out correctly, but the
+order in which they are sent is indeterminate.
Multiple Processes Semantics:
-Multiple processes, on the other hand, should NEVER call SP_receive on the same spread connection as the locking does not protect against accesses from different processes and thus can cause corrupted messages, deadlocks and other problems. What a program can do when using multiple processes is pass a spread connection from one process to another through a fork call, and as long as only one of the processes uses the connection, then the behavior will be correct. For example a parent process can pass the connection through a fork to a child and as long as only the parent or the child (but not both) use the connection it will work correctly, even if a parent is also multithreaded. Note, this does NOT allow a process to send the spread connection to another process through mechanisms such as Unix Domain Socket File Descriptor passing, as although the spread connection is a file descriptor and will correctly be passed, the library state of the connection that is stored in the libspread will not be shared with the receiving process, and thus the spread library will not be able to work with the received connection.
+Multiple processes, on the other hand, should NEVER call SP_receive on
+the same spread connection as the locking does not protect against
+accesses from different processes and thus can cause corrupted
+messages, deadlocks and other problems. What a program can do when
+using multiple processes is pass a spread connection from one process
+to another through a fork call, and as long as only one of the
+processes uses the connection, then the behavior will be correct. For
+example a parent process can pass the connection through a fork to a
+child and as long as only the parent or the child (but not both) use
+the connection it will work correctly, even if a parent is also
+multithreaded. Note, this does NOT allow a process to send the spread
+connection to another process through mechanisms such as Unix Domain
+Socket File Descriptor passing, as although the spread connection is a
+file descriptor and will correctly be passed, the library state of the
+connection that is stored in the libspread will not be shared with the
+receiving process, and thus the spread library will not be able to
+work with the received connection.
-When calling fork in a multi-threaded application using Spread, the Spread library uses an at_fork handler to release any locks that are held in the child process at the time of the fork. This is strictly speaking, unsafe, because a parent thread may have a Mutex lock on the mailbox because it is reading from the socket and once the locks in the child are reset, the child process could also attempt a read -- resulting in garbage in both processe. However -- We document and require that spread mailboxes cannot be read from different processes at the same time, so the application is responsible for not using them this way.
+When calling fork in a multi-threaded application using Spread, the
+Spread library uses an at_fork handler to release any locks that are
+held in the child process at the time of the fork. This is strictly
+speaking, unsafe, because a parent thread may have a Mutex lock on the
+mailbox because it is reading from the socket and once the locks in
+the child are reset, the child process could also attempt a read --
+resulting in garbage in both processe. However -- We document and
+require that spread mailboxes cannot be read from different processes
+at the same time, so the application is responsible for not using them
+this way.
-To enable a multi-threaded application to who is the parent of a forked child to cleanly remove their libspread state about the connection passed to the client, the function SP_kill has now been exposed in the Spread API. This function will close the Spread connection and remove any library state about the connection WITHOUT notifying the daemon about the close. If, as in the forked parent/child case, the client connection is still connected and open in a different process (such as the parent calling SP_kill after a fork and the child not closing it) then the daemon will not notice any change and will continue to communicate with the process who currently has the connection still open.
+To enable a multi-threaded application to who is the parent of a
+forked child to cleanly remove their libspread state about the
+connection passed to the client, the function SP_kill has now been
+exposed in the Spread API. This function will close the Spread
+connection and remove any library state about the connection WITHOUT
+notifying the daemon about the close. If, as in the forked
+parent/child case, the client connection is still connected and open
+in a different process (such as the parent calling SP_kill after a
+fork and the child not closing it) then the daemon will not notice any
+change and will continue to communicate with the process who currently
+has the connection still open.
-However, generally the SP_disconnect function should be used to disconnect from the daemon as it correctly and expediently notifies the daemon about the disconnection. If a client calls SP_kill instead of SP_disconnect when no other process has the client side mbox open (because of fork) then the network connection to the daemon will be closed and the daemon will detect the client as disconnected as soon as the network layer (TCP) registers a closed socket on the daemon side. Thus it will appear to the daemon as if a true network fault occurred or the client crashed as opposed to cleanly disconnecting.
+However, generally the SP_disconnect function should be used to
+disconnect from the daemon as it correctly and expediently notifies
+the daemon about the disconnection. If a client calls SP_kill instead
+of SP_disconnect when no other process has the client side mbox open
+(because of fork) then the network connection to the daemon will be
+closed and the daemon will detect the client as disconnected as soon
+as the network layer (TCP) registers a closed socket on the daemon
+side. Thus it will appear to the daemon as if a true network fault
+occurred or the client crashed as opposed to cleanly disconnecting.
More information about the Spread-cvs
mailing list