[Spread-users] Partition problem

[Aswin Almeida]

Hello folks.

BBN Technologies is conducting experiments on the Spread and Secure Spread (layered architecture) for DARPA.

Recently, we experienced problems with a "partitioning issue" which can affect the measurement of join times.
Yair is aware of these problems via our BBN-JHU-SRI experiment mailing list.
I wanted to appeal to a _wider audience_ as well to see if anyone else has ideas.

The Network
------------------

Network contains enclaves (Spread Segments) in several locations around the United States.
We will refer to these segments by letter and dub them Site [A..F].

Site A is in Virginia
Site B is in Cambridge
Site C is in New York
Site D is in California
Site E is in Hawaii
Site F is in New Mexico

Background
-----------------

Four users from site A join group "test".  (The "initial set of users").
Four users from Site B join group "test".  (The "joining set of users").

Thus, only membership messages are involved, not regular messages between users.

What was observed?

The four users from site B will form their own group.  A partition occurs.  Approximately 30 seconds later a merge will take place after the groups discover each other.  With a larger initial set of users (e.g. 64 users) and a larger joining set of users (e.g. 64 users) this initially partitioned and then merged behavior does NOT occur.  Similar behavior was observed when the users are drawn from sites A and C.

More recently, we observed this same behavior when executing between sites B and C.  This link was thought to be good between sites B and C for Spread's purposes, according to the "send" and "receive" utilities.  Use of s.c and r.c is discussed in the next section.

Why is this of concern?
----------------------------------

Beyond being a mild curiousity, it could potentially affect data collection.  

Yair sat down with Sara and I in Virginia and suggested the use of the s.c and r.c (send and receive) utilities.  
Deploying "send" at Site A and "receive" at Site B, the three of us observed approximately 30% loss.
Using these utilities at Site B and Site C, the loss was alot lower (5-7%) and Yair said this was more acceptable.  

However, we are still seeing the partitioning behavior even between sites B and C, where the reported loss by s and r is markedly lower than 30%.

Investigation to Date
------------------------------

Initially there was thought that Site A, its hardware (Sidewinder NIC card which had perhaps stale errors on the console), or its connectivity to the outside world could have been the culprit for this partitioning issue.

Since Yair visited with Sara and I at Site A weeks ago, no errors have appeared on the console.
This site has a fractional T3 as its connection to the outside world.
Other sites such as B, C which look ok with the s.c and r.c utilities have exhibited this partitioning behavior as well.
In fact, site pairs which have high loss rates reported by s.c and r.c produce somewhat consistent join times across runs.

This supports the theory that the problem is not with any particular enclave, but perhaps with the underlying network itself.
Pairwise pings and traceroutes using external IPs for our VPN don't show a problem between the sites.

Maybe we are missing something and can use spmonitor and subsequently tweak flow control.  Or maybe there is something else we can try with the Spread.

What have we already tried?  Yair had suggested we try to tune the parameters in flow_control.c, thus Sara changed Window = 15 and Personal_window = 3 in FC_init.  Unfortunately this did not help.  We still are seeing this partition/merge behavior, which would skew join time measurements.

What we need
---------------------
When users at two different geographic locations join a single group, we'd like to not see a *partition* (and then many moments later) a merge, as this will skew our measured data for jointimes.  Avoiding the slow, eventual merge would help data collection, but it would be optimal to avoid this partitioning behavior altogether.

Thoughts, comments appreciated,

Aswin Almeida 
BBN Technologies