[Spread-users] Cluster locked up again

[Tom Mornini]

Well, making certain to drain the mailboxes didn't prevent Spread from 
hanging again.

I've already implemented the connect level group_membership change, but 
that code hasn't been pushed yet.

Here's an spmonitor dump. Does anything seem out of order? The huge 
retrans number on obi (the group leader) looks rather suspicious...

Question: Is there any problem with running a spread daemon on a system 
that nobody is connecting to? We run jabba as a hot spare, but nothing 
is actually running there. I notice that it's recv pack number is twice 
as high as the next highest...and that doesn't seem right!

============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 248709 seconds :
Membership  :  5  procs in 1 segments, leader is obi
rounds   : 76806053     tok_hurry :  353432     memb change:       1
sent pack:       2      recv pack :  752333     retrans    :     226
u retrans:       9      s retrans :     217     b retrans  :       0
My_aru   :  768294      Aru       :  766693     Highest seq:  768294
Sessions :       3      Groups    :       3     Window     :      60
Deliver M:  767663      Deliver Pk:  768294     Pers Window:      15
Delta Mes:       0      Delta Pack:       0     Delta sec  :      11
==================================

Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 248741 seconds :
Membership  :  5  procs in 1 segments, leader is obi
rounds   : 76810910     tok_hurry :  353473     memb change:       2
sent pack:  364288      recv pack :  387839     retrans    :     510
u retrans:      12      s retrans :     498     b retrans  :       0
My_aru   :  768294      Aru       :  766693     Highest seq:  768294
Sessions :      84      Groups    :       3     Window     :      60
Deliver M:  767737      Deliver Pk:  768384     Pers Window:      15
Delta Mes:      74      Delta Pack:       0     Delta sec  :      32
==================================

Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 248736 seconds :
Membership  :  5  procs in 1 segments, leader is obi
rounds   : 76810909     tok_hurry :  353473     memb change:       2
sent pack:  387600      recv pack :  365029     retrans    :       0
u retrans:       0      s retrans :       0     b retrans  :       0
My_aru   :  768294      Aru       :  766693     Highest seq:  768294
Sessions :      84      Groups    :       3     Window     :      60
Deliver M:  767737      Deliver Pk:  768384     Pers Window:      15
Delta Mes:       0      Delta Pack:       0     Delta sec  :      -5
==================================

Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 248742 seconds :
Membership  :  5  procs in 1 segments, leader is obi
rounds   : 76810909     tok_hurry :  353473     memb change:       2
sent pack:      18      recv pack : 1711493     retrans    :       4
u retrans:       4      s retrans :       0     b retrans  :       0
My_aru   :  768294      Aru       :  766693     Highest seq:  768294
Sessions :       0      Groups    :       3     Window     :      60
Deliver M:  767737      Deliver Pk:  768384     Pers Window:      15
Delta Mes:       0      Delta Pack:       0     Delta sec  :       6
==================================

Monitor>
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 248710 seconds :
Membership  :  5  procs in 1 segments, leader is obi
rounds   : 76806052     tok_hurry :  354419     memb change:       1
sent pack:       2      recv pack :  752558     retrans    :  206268
u retrans:  206268      s retrans :       0     b retrans  :       0
My_aru   :  768294      Aru       :  766693     Highest seq:  768294
Sessions :       3      Groups    :       3     Window     :      60
Deliver M:  767663      Deliver Pk:  768294     Pers Window:      15
Delta Mes:     -74      Delta Pack:       0     Delta sec  :     -32
==================================

--
-- Tom Mornini
-- eWingz Systems, Inc.
--
-- ICQ: 113526784, AOL: tmornini, Yahoo: tmornini, MSN: tmornini