Jonathan, i'll try to give more color to your points below.<br><br>1) we were originally running only 3.17.4 daemons in the configuration below, but<br> b/c of the daemon going into a hung state, we changed it to the current <br> configuration. a configuration where we added a 3.17.3 daemon to each<br> segment in an attempt to see if the problem was version specific. We will <br> revert back to one version once our test completes. Note, all client <br> connections are made to the 3.17.4 daemons.<br><br>Spread_Segment 172.25.30.255:8006 {<br> as-ny-cbapps2 172.25.30.27<br>}<br>Spread_Segment 172.25.5.255:8006 {<br> as-phl-cbiis1 172.25.5.61<br>}<br><br>2) true, so the daemons are not stressed which makes it more confusing with<br> regards to how it can suddenly stop working since load does not appear to<br> be
the issue<br><br>3) So, to confirm, b/c most of the retrans are of type b, drops are happening<br> betweens Spread_Segments. So, how is information shared between <br> segments? it doesn't use the segment ip address (i.e. 172.25.5.255, <br> 172.25.30.255) right? So, what mechanism is used for segment<br> communication? And what protocal is used to sync the daemons in the<br> cluster?<br><br><br>we've confirmed the packet drop problem is not the network but instead<br>something related to two of the four servers used to host the daemons.<br><br>as for using the monitoring tool, once a daemon goes bad, in the past, if i <br>remember correctly, the daemon will not respond to the monitors status <br>request. but i will confirm and send you the result when a daemon goes bad.<br><br>b/c of you insight, I'm thinking the daemon freeze is a result of our drops. is there a configuration
we can use to bypass or reduce these drops to see if this coincides with improve daemon stability?<br><br><br>Thanks <br><br><br><br> <br><br><b><i>Jonathan Stanton <jonathan@cnds.jhu.edu></i></b> wrote:<blockquote class="replbq" style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;"> Hi,<br><br>The monitor output is very intersting. A few points that jump out at me <br>immediately.<br><br>1) You appear to be running some 3.17.3 daemons and some 3.17.4 daemons <br>(version listed in first line of monitor output) I'd recommend upgrading <br>all of them to 3.17.4 -- not only because of the bugfixes in the <br>release, but because we don't recommend running mixed groups of daemons <br>at different versions as changes between versions may break those <br>setups -- and we don't test for 'inter-version' compatibility. Sometimes <br>it works, but it often doesn't. <br><br>2) Your overall traffic levels are low (3000 packets in
74000 seconds), <br>so most of the time the daemon should be relatively idle (in cpu and <br>network activity)<br><br>3) A few nodes like as-phl-cbiis1 are generating a *lot* of<br>retransmissions. 1251 b retrans out of 3000 finaly delivered packets and<br>as-ny-cbapps2 with 600 / 3000. (b retrans means it a broadcast<br>retransmission because more then 1 of the daemons in the configuration<br>missed the packet, so we rebroadcast it). So about 1/2 of the time the<br>packet is lost and has to be resent. <br><br>So I would definitely look into the loss rates on the network and see if <br>they can be improved. Now packet loss should not cause a daemon to <br>'hang' or freeze excpet in certain very pathological states where the <br>same packets that a daemon needs to synchronize are continually being <br>lost but other packets are getting through (i.e. daemon retransmits and <br>it is lost again, but other packets are getting through -- because if <br>nothing gets through the
daemon will determine that a network fault has <br>occurred and will partition itself into a separate configuration.<br><br>If you are able to run the monitor command when one of the daemons is in <br>the 'hung' state that would be very helpful in determining what is <br>happening.<br><br>Cheers,<br><br>Jonathan<br><br>On Tue, Mar 11, 2008 at 03:04:11PM -0700, chanh hua wrote:<br>> john, <br>> <br>> when a daemon goes into a bad state, it doesn't just stutter, but stops working completely; however, other daemon in the cluster still functions correctly.<br>> <br>> i did some further testing and the massive drops only occur btw these two servers. if i used spsend and sprecv against any one of this server and another machine not defined in the segment, the drop comes down to the amount you specified. so the drops is somehow confined to this segment. will need to do some more digging.<br>> <br>> looking at sptmonitor output, the "retrans" to "sent pack"
seems high. do you think there is a network issue or configuration problem?<br>> <br>> <br>> <br>> ============================<br>> Status at as-phl-cbiis4 V 3.17. 3 (state 1, gstate 1) after 74354 seconds :<br>> Membership : 4 procs in 2 segments, leader is as-ny-cbapps2<br>> rounds : 42454 tok_hurry : 38124 memb change: 9<br>> sent pack: 6 recv pack : 3040 retrans : 0<br>> u retrans: 0 s retrans : 0 b retrans : 0<br>> My_aru : 2745 Aru : 2745 Highest seq: 2745<br>> Sessions : 0 Groups : 33 Window : 60<br>> Deliver M: 1595 Deliver Pk: 3048 Pers Window: 15<br>> Delta Mes: 1595 Delta Pack: 2745 Delta sec : 74354<br>> ==================================<br>> <br>> Monitor><br>> ============================<br>> Status at as-phl-cbiis1 V 3.17. 4
(state 1, gstate 1) after 73908 seconds :<br>> Membership : 4 procs in 2 segments, leader is as-ny-cbapps2<br>> rounds : 21796 tok_hurry : 37928 memb change: 4<br>> sent pack: 1708 recv pack : 1137 retrans : 1291<br>> u retrans: 37 s retrans : 0 b retrans : 1254<br>> My_aru : 2745 Aru : 2745 Highest seq: 2745<br>> Sessions : 9 Groups : 56 Window : 60<br>> Deliver M: 1595 Deliver Pk: 2944 Pers Window: 15<br>> Delta Mes: 0 Delta Pack: 0 Delta sec : -446<br>> ==================================<br>> <br>> Monitor><br>> ============================<br>> Status at as-ny-cbsql3 V 3.17. 3 (state 1, gstate 1) after 74106 seconds :<br>> Membership : 4 procs in 2 segments, leader is as-ny-cbapps2<br>> rounds : 42313 tok_hurry : 37997 memb change:
6<br>> sent pack: 5 recv pack : 3072 retrans : 0<br>> u retrans: 0 s retrans : 0 b retrans : 0<br>> My_aru : 2745 Aru : 2745 Highest seq: 2745<br>> Sessions : 0 Groups : 19 Window : 60<br>> Deliver M: 1595 Deliver Pk: 3040 Pers Window: 15<br>> Delta Mes: 0 Delta Pack: 0 Delta sec : 198<br>> ==================================<br>> <br>> Monitor><br>> ============================<br>> Status at as-ny-cbapps2 V 3.17. 4 (state 1, gstate 1) after 73982 seconds :<br>> Membership : 4 procs in 2 segments, leader is as-ny-cbapps2<br>> rounds : 21797 tok_hurry : 37970 memb change: 4<br>> sent pack: 1145 recv pack : 1714 retrans : 739<br>> u retrans: 109 s retrans : 37 b retrans : 593<br>> My_aru : 2745
Aru : 2745 Highest seq: 2745<br>> Sessions : 20 Groups : 56 Window : 60<br>> Deliver M: 1595 Deliver Pk: 3028 Pers Window: 15<br>> Delta Mes: 0 Delta Pack: 0 Delta sec : -124<br>> ==================================<br>> <br>> <br>> John Schultz <jschultz@spreadconcepts.com> wrote: On Tue, 11 Mar 2008, chanh hua wrote:<br>> <br>> > Bring me to my question, what network property is the misses data <br>> > suppose to tell us?<br>> <br>> The misses data tells you how many of the sent packets the receiver missed <br>> (i.e. - didn't receive). From your report it looks like the sender sent <br>> 10000 packets but the receiver only heard 1999 (10000 - 8001) of them <br>> before it got the last packet. If correct, then that would be about an <br>> 80% loss rate for your configuration. A typical loss rate for LAN <br>>
broad/multicast is well below 1%.<br>> <br>> > The explanation he gave for why we might have observed all<br>> > these misses was b/c the broadcast address used contains all<br>> > network machines(i.e. desktops, printers, etc...) and not<br>> > just servers and most of those machines ignore broadcast.<br>> > But since he doesn't know what these results mean, he can't<br>> > say for sure.<br>> <br>> He is correct that broadcast will bother (i.e. - potentially increase <br>> load) all the machines on the associated subnet. If you instead use <br>> multicast, then either your switch/router or you NICs should filter out <br>> the packets before an interrupt is generated on non-participating <br>> machines. Multicast is preferable, however, occasionally some switches <br>> and routers don't implement multicast well or their multicast is <br>> misconfigured. In such situations, broadcast sometimes works better
due <br>> to its simplicity.<br>> <br>> > If this is an issue, would using a multicast address be better? <br>> > However, when i used a multicast address for the test, i still saw a lot <br>> > of misses.<br>> <br>> Typically, broadcast should not increase loss versus multicast unless your <br>> switch/router is biased against broadcast somehow for some weird reason.<br>> <br>> > I talked to the network admin, and he's not seeing any drops<br>> > btw the servers on the segments. And he confirmed the<br>> > broadcast address i used was correct.<br>> <br>> Well, it definitely seems like something is wrong from your reports. Try <br>> using spmonitor to view the status of the daemons as they are running. <br>> Like I said, if you see their retrans counts going up by more than a <br>> couple a second, then something is probably wrong in your network.<br>> <br>> > would having a lot of drops lead
cause daemon to be unresponsive?<br>> <br>> Theoretically, it could. If the daemons got stuck in a loop of trying to <br>> establish a membership due to intermittent / flaky communications with <br>> other daemons, then the system would appear to freeze as the daemons stop <br>> processing client communications in this state. Usually, the "freeze"<br>> wouldn't persist forever but rather you would see lots of daemon <br>> membership changes and progress would stutter forward.<br>> <br>> Cheers!<br>> <br>> ---<br>> John Schultz<br>> Spread Concepts<br>> Phn: 443 838 2200<br>> <br>> <br>> <br>> ---------------------------------<br>> Never miss a thing. Make Yahoo your homepage.<br>> _______________________________________________<br>> Spread-users mailing list<br>> Spread-users@lists.spread.org<br>> http://lists.spread.org/mailman/listinfo/spread-users<br><br><br>--
<br>-------------------------------------------------------<br>Jonathan R. Stanton jonathan@cs.jhu.edu<br>Dept. of Computer Science <br>Johns Hopkins University <br>-------------------------------------------------------<br></jschultz@spreadconcepts.com></blockquote><br><p> 
<hr size=1>Never miss a thing. <a href="http://us.rd.yahoo.com/evt=51438/*http://www.yahoo.com/r/hs"> Make Yahoo your homepage.</a>