[Spread-cvs] commit: r300 - trunk/docs

[jonathan at spread.org]

Author: jonathan
Date: 2006-02-19 16:04:21 -0500 (Sun, 19 Feb 2006)
New Revision: 300

Added:
   trunk/docs/DynamicConfiguration.txt
Log:
Document how to use new Spread 4 dynamic configuration feature and how it works.


Added: trunk/docs/DynamicConfiguration.txt
===================================================================
--- trunk/docs/DynamicConfiguration.txt	                        (rev 0)
+++ trunk/docs/DynamicConfiguration.txt	2006-02-19 21:04:21 UTC (rev 300)
@@ -0,0 +1,79 @@
+Motivation:
+
+Spread prior to version 4 required a configuration file specifying all of the daemons who would ever be part of the spread configuration and to make changes to the set of potential daemons required that all of the daemons be shut down and restarted once the configuration file was edited to include the changes. This made adding servers to a pool or changing the name or IP of an existing server require a full restart of the system which can be disruptive. 
+
+Solution:
+
+Spread in version 4 can now handle changes to the spread.conf daemon configuration without shutting down and restarting all of the daemons. Daemons can be added to segments or removed without existing clients losing their spread connections. Some changes to the spread configuration (such as changing the name of an existing daemon) will cause the clients to receive a view change message showing a temporary change in the membership. This feature is called Dynamic Configuration. 
+
+Equal Configuration Enforcement:
+
+The Spread system has always had a requirement that the daemon configurations specified in spread.conf at all of the node were identical. However earlier versions of Spread did not enforce that requirement automatically. Now Spread enforces that all participating daemons are using an identical configuration. 
+
+Spread enforces this property by having all packets sent between daemons contain an identification code in the message header for the configuration that was active when they were sent. When a packet is received, if the packet configuration code does not match the current configuration code of the receiving daemon, then the packet is discarded. If one daemon switches configurations, it cannot communicate with any other daemon that still uses the old configuration. If the other daemons do not also switch, then they will be partitioned until the configuration error is repaired.
+
+The identification code is implemented as a 32 bit hash of the relevant sections of the Spread.conf file.
+
+Dynamic Configuration Client Experience:
+
+When a new configuration is loaded, application groups may experience a membership change. The change may appear spurious, meaning no actual change in the list of members occurs, but a new group_id or view_id is generated and a membership message is received. The application may also see a membership change that shows all non-local clients (those connected to a different daemon) leaving the group and then a second membership messages showing all of the clients rejoining the group (possibly with some missing if they were connected to a daemon that is no longer valid in the new configuration file).
+
+No changes to any user-visible data structures or functions are necessary for this. 
+
+
+How to activate a Dynamic Configuration Change:
+
+The process of executing a configuration changes is as follows:
+
+1) The admin runs the spmonitor program. 
+   It is necessary to do this first so that the spmonitor program reads in the list of possible daemons from the old spread.conf file. It needs to know what daemons were in the old configuration in order to send them the notification to reread the spread.conf file. 
+
+2) The admin updates the Spread.conf file on disk with the new contents. They then copy that updated file to all of the hosts running daemons who are in the old or new configuration.
+
+3) The admin chooses the spmonitor option "Reload Configuration". 
+   This causes a command packet to be sent by the spmonitor program to all daemons in the old configuration file. The command packet triggers the daemon to start the reload process. The internal details of the daemon algorithm are described below. At this point all of the daemons will switch to the new configuration. 
+
+4) Once the reload is done, all client applications may receive a membership change notification message. 
+   The message will be generated if any groups the client is in have changed the set of members (because a daemon is no longer in the membership and had clients in that group) or, for certain types of configuration changes, all groups with members connected to more then one daemon will receive a membership change showing all group members from remote daemons leaving the group, and then a second membership showing the group with all clients still connected to running daemons back together. 
+
+
+Semantics of a Dynamic Configuration Change:
+
+For the basic case, the algorithm looks like a regular membership in which certain daemons have crashed or partitioned (those who are in the old configuration and not in the new configuration) and some daemons have started up or are remerging (those in the new configuration but not in the old). 
+
+Crashed/partitioned daemons do not have any group state we need to track since we will remove all of their group members from the current membership view. Therefore, we can just ignore them as in a normal membership. The new daemons that have been added to the configuration will merge any group state they have during the normal Extended Virtual Synchrony protocol as if they had only been partitioned away. Since group members are identified with the daemons they are connected to, there cannot be any inconsistent state between two daemons, as they will each represent their own state in the group membership protocol.
+
+A special case occurs if the same IP address is in two different configuration files with an inconsistency (different name, changed broadcast address, etc.) When that case is identified, a singleton partition is created. The partition will separate each daemon from all other daemons and cause their group state to shrink to only the locally connected client applications. Then, when the singleton partition is removed, they will merge with all of the other daemons. Since each daemon will only know its own locally connected client group members, merging the partitions will bring the system to a consistent state.
+
+Any daemon whose network specification has changed between the old and new configuration files will have to shut down and be restarted. This is because the Spread daemon does not re-bind to changed broadcast, multicast or IP unicast addresses after starting up. So changing the IP address of a machine will require the daemon on that machine to be shut down and then restarted with the new configuration, however other daemons who are active will not shut down and will continue to work over the configuration change. Only the machine with the changed network configuration will shut down.
+
+Daemon Algorithm Details:
+
+When a spmonitor command to reload configuration is received, the daemon executes the following steps. (See function Prot_handle_conf_reload() for the code).
+
+1) Call function in configuration code to load new configuration file and examine the type of changes to the configuration. If the configuration changes in the new file require a singleton partition (see below for details) then indicate that to protocol code.
+
+2) Inform all subsystems in the daemon that a configuration change is in progress and any caches or data structures that are storing the current state of the daemon configuration should be reloaded/updated with the new information. The old configuration can still be accessed through a different name until the reload is complete. This is done by a calling functions called "*_signal_conf_reload()" where the * is replaced by the name of the subsystem. For example "Net_signal_conf_reload()" and "Memb_signal_conf_reload()".
+
+3a) If no partition is needed, then initiate a membership change (with token_loss) and return. The membership protocol will run to completion as normal by probing for all active daemons and forming a new membership including any new daemons and removing any daemons who are no longer in the configuration file.
+
+3b) If a partition is needed, then a singleton partition is created, the Conf_reload_state variable is set so other parts of the daemon know a configuration reload is in progress, and a membership change is triggered (by a scheduled token_loss).
+
+4) When membership completes in the Discard_packets() function, if a configuration reload state is active, then the partition is removed and a probe for new members (to find all of the now 'non-partitioned' daemons) is started. Once this second membership change takes place, then the configuration reload is complete. 
+
+The test for whether a partition is needed for a new configuration is below. See function Conf_reload_initiate() in configuration.c for the implementation details.
+
+a)  If any of the following are true then the daemon process will quit as it should no longer be executing in the new configuration file.
+	 1) I am no longer in the configuration.
+	 2) My IP/Name has changed.
+	 3) My broadcast/netmask has changed.
+
+b) Otherwise examine all entries that are both in the new configuration and in the old configuration (matching based on IP address).
+If the same IP address is in both configurations then if any of the following differs between the old and new configuration, then a partition is required. If they are the same for ALL hosts that are in both files, then no partition is needed. 
+	 1) Name.
+	 2) Number of interfaces.
+	 3) Broadcast address or netmask.
+	 4) Any interface specifications have changed.
+
+Note that in the common cases of simply adding and removing daemons, no partition is needed. 
+