[Spread-users] System hangs using spread for logging.
Jonathan Stanton
jonathan at cnds.jhu.edu
Wed Mar 20 15:58:54 EST 2002
On Tue, Mar 19, 2002 at 09:50:44PM +0000, Jacob Davies wrote:
>
> Hello there,
>
> Apache error). What happens is that after a while running like
> this -- has been up to two weeks, has been as short as a couple
> of hours -- the system hangs completely, presumably all processes
> waiting on being able to write more logging output to spread.
I took a look at both these traces and Tom's last ones he emailed in and I'm
still a bit puzzled. This does not appear to be a membersihp problem, as all
of the daemons are in a correct operational state. What is happening is that
the token is continuing to cycle, but both the ARU count is not going up and
no new messages are being sent.
Normally new messages are blocked in only two cases. First, during some parts
of membership, and second, when too many messages are outstanding in teh
system (a form of flow control). It appears that the second reason is the
cause, as in both traces the ARU is 1600 messages behind Highest_Seq which is
exactly the maximum gap allowed in Spread. (unless you recompiled with a
modifed spread_params.h file which I doubt).
So the core problem is that the ARU is not being increased even though
everyone has all of the messages (as evidenced by the my_aru counters on all
of them being equal and higher then ARU). I'll look into this and see if I can
see a reason this could occur.
>
> When we restart one spread daemon the system generally starts
> working again, with a flood of messages logged as soon as we
> restart (presumably the contents of various buffers that are
> waiting). Sometimes though we need to restart everything (spread
> and the applications) to get them to work again.
The restart triggers a membership change which somehow kicks the system out of
the non-aru updateing state. It is interesting that sometimes it requires more
processes to restart. In the traces you have emailed, did those fix themselves
with a single daemon restart? Or in either of these cases did you have to
restart everything?
>
> Previously we were asked for several cycles of spmonitor output
> from while the system was hung, so what follows is that -- the system
> was hung completely while I was running spmonitor, nobody was able
> to put anything into spread or get anything out.
Thanks. Yes, in this state no new messages or requests to join groups will be
accepted by the daemon.
>
> Hope this makes some sense -- I'm curious about what further
> debugging I can do to help here -- system call traces? netstat
> output?
This appears to be an internal daemon issue. After I look into the code more
I'll see if anything could be helpful.
Thanks for your help so far.
Jonathan
>
> =============
> Activate Status
> -------------
>
> Enter Proc Name: all
> Enter Proc Name:
> Monitor: send status query
>
> Monitor>
> ============================
> Status at obi V 3.16. 1 (state 1, gstate 1) after 156137 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37014803 tok_hurry : 143898 memb change: 1
> sent pack: 2 recv pack : 240617 retrans : 113917
> u retrans: 113876 s retrans : 41 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: 286934 Delta Pack: 285524 Delta sec : 156137
> ==================================
>
> Monitor>
> ============================
> Status at boba V 3.16. 1 (state 1, gstate 1) after 156136 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37014804 tok_hurry : 143200 memb change: 1
> sent pack: 2 recv pack : 240518 retrans : 141
> u retrans: 0 s retrans : 141 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: 0 Delta Pack: 0 Delta sec : -1
> ==================================
>
> Monitor>
> ============================
> Status at lando V 3.16. 1 (state 1, gstate 1) after 156170 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37020613 tok_hurry : 143252 memb change: 3
> sent pack: 119740 recv pack : 120850 retrans : 178
> u retrans: 32 s retrans : 146 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 91 Delta Pack: 0 Delta sec : 34
> ==================================
>
> Monitor>
> ============================
> Status at greedo V 3.16. 1 (state 1, gstate 1) after 156159 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37019431 tok_hurry : 143241 memb change: 2
> sent pack: 120557 recv pack : 187466 retrans : 104
> u retrans: 0 s retrans : 104 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
> Delta Mes: -20 Delta Pack: 0 Delta sec : -11
> ==================================
>
> Monitor>
> ============================
> Status at jabba V 3.16. 1 (state 1, gstate 1) after 156171 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37020610 tok_hurry : 143252 memb change: 3
> sent pack: 3 recv pack : 595373 retrans : 67407
> u retrans: 67407 s retrans : 0 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 0 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 20 Delta Pack: 0 Delta sec : 12
> ==================================
>
> Monitor> Monitor: send status query
>
> ============================
> Status at obi V 3.16. 1 (state 1, gstate 1) after 156147 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37025760 tok_hurry : 143898 memb change: 1
> sent pack: 2 recv pack : 240617 retrans : 113917
> u retrans: 113876 s retrans : 41 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -91 Delta Pack: 0 Delta sec : -24
> ==================================
>
> Monitor>
> ============================
> Status at boba V 3.16. 1 (state 1, gstate 1) after 156146 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37025761 tok_hurry : 143200 memb change: 1
> sent pack: 2 recv pack : 240518 retrans : 141
> u retrans: 0 s retrans : 141 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: 0 Delta Pack: 0 Delta sec : -1
> ==================================
>
> Monitor>
> ============================
> Status at lando V 3.16. 1 (state 1, gstate 1) after 156180 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37031570 tok_hurry : 143252 memb change: 3
> sent pack: 119740 recv pack : 120850 retrans : 178
> u retrans: 32 s retrans : 146 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 91 Delta Pack: 0 Delta sec : 34
> ==================================
>
> Monitor>
> ============================
> Status at greedo V 3.16. 1 (state 1, gstate 1) after 156169 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37030388 tok_hurry : 143241 memb change: 2
> sent pack: 120557 recv pack : 187466 retrans : 104
> u retrans: 0 s retrans : 104 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
> Delta Mes: -20 Delta Pack: 0 Delta sec : -11
> ==================================
>
> Monitor>
> ============================
> Status at jabba V 3.16. 1 (state 1, gstate 1) after 156181 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37031567 tok_hurry : 143252 memb change: 3
> sent pack: 3 recv pack : 595373 retrans : 67407
> u retrans: 67407 s retrans : 0 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 0 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 20 Delta Pack: 0 Delta sec : 12
> ==================================
>
> Monitor> Monitor: send status query
>
> ============================
> Status at obi V 3.16. 1 (state 1, gstate 1) after 156157 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37037647 tok_hurry : 143898 memb change: 1
> sent pack: 2 recv pack : 240617 retrans : 113917
> u retrans: 113876 s retrans : 41 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -91 Delta Pack: 0 Delta sec : -24
> ==================================
>
> Monitor>
> ============================
> Status at lando V 3.16. 1 (state 1, gstate 1) after 156190 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37043456 tok_hurry : 143252 memb change: 3
> sent pack: 119740 recv pack : 120850 retrans : 178
> u retrans: 32 s retrans : 146 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 91 Delta Pack: 0 Delta sec : 33
> ==================================
>
> Monitor>
> ============================
> Status at boba V 3.16. 1 (state 1, gstate 1) after 156156 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37037648 tok_hurry : 143200 memb change: 1
> sent pack: 2 recv pack : 240518 retrans : 141
> u retrans: 0 s retrans : 141 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -91 Delta Pack: 0 Delta sec : -34
> ==================================
>
> Monitor>
> ============================
> Status at greedo V 3.16. 1 (state 1, gstate 1) after 156179 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37042274 tok_hurry : 143241 memb change: 2
> sent pack: 120557 recv pack : 187466 retrans : 104
> u retrans: 0 s retrans : 104 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
> Delta Mes: 71 Delta Pack: 0 Delta sec : 23
> ==================================
>
> Monitor>
> ============================
> Status at jabba V 3.16. 1 (state 1, gstate 1) after 156191 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37043454 tok_hurry : 143252 memb change: 3
> sent pack: 3 recv pack : 595373 retrans : 67407
> u retrans: 67407 s retrans : 0 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 0 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 20 Delta Pack: 0 Delta sec : 12
> ==================================
>
> Monitor> Monitor: send status query
>
> ============================
> Status at obi V 3.16. 1 (state 1, gstate 1) after 156167 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37048360 tok_hurry : 143898 memb change: 1
> sent pack: 2 recv pack : 240617 retrans : 113917
> u retrans: 113876 s retrans : 41 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -91 Delta Pack: 0 Delta sec : -24
> ==================================
>
> Monitor>
> ============================
> Status at boba V 3.16. 1 (state 1, gstate 1) after 156166 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37048361 tok_hurry : 143200 memb change: 1
> sent pack: 2 recv pack : 240518 retrans : 141
> u retrans: 0 s retrans : 141 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: 0 Delta Pack: 0 Delta sec : -1
> ==================================
>
> Monitor>
> ============================
> Status at lando V 3.16. 1 (state 1, gstate 1) after 156201 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37054170 tok_hurry : 143252 memb change: 3
> sent pack: 119740 recv pack : 120850 retrans : 178
> u retrans: 32 s retrans : 146 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 91 Delta Pack: 0 Delta sec : 35
> ==================================
>
> Monitor>
> ============================
> Status at greedo V 3.16. 1 (state 1, gstate 1) after 156189 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37052988 tok_hurry : 143241 memb change: 2
> sent pack: 120557 recv pack : 187466 retrans : 104
> u retrans: 0 s retrans : 104 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
> Delta Mes: -20 Delta Pack: 0 Delta sec : -12
> ==================================
>
> Monitor>
> ============================
> Status at jabba V 3.16. 1 (state 1, gstate 1) after 156201 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37054167 tok_hurry : 143252 memb change: 3
> sent pack: 3 recv pack : 595373 retrans : 67407
> u retrans: 67407 s retrans : 0 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 0 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 20 Delta Pack: 0 Delta sec : 12
> ==================================
>
> Monitor> Monitor: send status query
>
> ============================
> Status at obi V 3.16. 1 (state 1, gstate 1) after 156177 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37058421 tok_hurry : 143898 memb change: 1
> sent pack: 2 recv pack : 240617 retrans : 113917
> u retrans: 113876 s retrans : 41 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -91 Delta Pack: 0 Delta sec : -24
> ==================================
>
> Monitor>
> ============================
> Status at boba V 3.16. 1 (state 1, gstate 1) after 156176 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37058422 tok_hurry : 143200 memb change: 1
> sent pack: 2 recv pack : 240518 retrans : 141
> u retrans: 0 s retrans : 141 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: 0 Delta Pack: 0 Delta sec : -1
> ==================================
>
> Monitor>
> ============================
> Status at lando V 3.16. 1 (state 1, gstate 1) after 156211 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37064231 tok_hurry : 143252 memb change: 3
> sent pack: 119740 recv pack : 120850 retrans : 178
> u retrans: 32 s retrans : 146 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 91 Delta Pack: 0 Delta sec : 35
> ==================================
>
> Monitor>
> ============================
> Status at jabba V 3.16. 1 (state 1, gstate 1) after 156211 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37064228 tok_hurry : 143252 memb change: 3
> sent pack: 3 recv pack : 595373 retrans : 67407
> u retrans: 67407 s retrans : 0 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 0 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 0 Delta Pack: 0 Delta sec : 0
> ==================================
>
> Monitor>
> ============================
> Status at greedo V 3.16. 1 (state 1, gstate 1) after 156199 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37063048 tok_hurry : 143241 memb change: 2
> sent pack: 120557 recv pack : 187466 retrans : 104
> u retrans: 0 s retrans : 104 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
> Delta Mes: -20 Delta Pack: 0 Delta sec : -12
> ==================================
>
> Monitor> Monitor: send status query
>
> ============================
> Status at obi V 3.16. 1 (state 1, gstate 1) after 156187 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37070425 tok_hurry : 143898 memb change: 1
> sent pack: 2 recv pack : 240617 retrans : 113917
> u retrans: 113876 s retrans : 41 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -71 Delta Pack: 0 Delta sec : -12
> ==================================
>
> Monitor>
> ============================
> Status at lando V 3.16. 1 (state 1, gstate 1) after 156221 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37076234 tok_hurry : 143252 memb change: 3
> sent pack: 119740 recv pack : 120850 retrans : 178
> u retrans: 32 s retrans : 146 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 91 Delta Pack: 0 Delta sec : 34
> ==================================
>
> Monitor>
> ============================
> Status at boba V 3.16. 1 (state 1, gstate 1) after 156186 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37070426 tok_hurry : 143200 memb change: 1
> sent pack: 2 recv pack : 240518 retrans : 141
> u retrans: 0 s retrans : 141 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -91 Delta Pack: 0 Delta sec : -35
> ==================================
>
> Monitor>
> ============================
> Status at greedo V 3.16. 1 (state 1, gstate 1) after 156209 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37075052 tok_hurry : 143241 memb change: 2
> sent pack: 120557 recv pack : 187466 retrans : 104
> u retrans: 0 s retrans : 104 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
> Delta Mes: 71 Delta Pack: 0 Delta sec : 23
> ==================================
>
> Monitor>
> ============================
> Status at jabba V 3.16. 1 (state 1, gstate 1) after 156221 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37076232 tok_hurry : 143252 memb change: 3
> sent pack: 3 recv pack : 595373 retrans : 67407
> u retrans: 67407 s retrans : 0 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 0 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 20 Delta Pack: 0 Delta sec : 12
> ==================================
>
> Monitor> Monitor: send status query
>
> ============================
> Status at obi V 3.16. 1 (state 1, gstate 1) after 156197 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37081506 tok_hurry : 143898 memb change: 1
> sent pack: 2 recv pack : 240617 retrans : 113917
> u retrans: 113876 s retrans : 41 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: -91 Delta Pack: 0 Delta sec : -24
> ==================================
>
> Monitor>
> ============================
> Status at boba V 3.16. 1 (state 1, gstate 1) after 156196 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37081507 tok_hurry : 143200 memb change: 1
> sent pack: 2 recv pack : 240518 retrans : 141
> u retrans: 0 s retrans : 141 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 3 Groups : 3 Window : 60
> Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
> Delta Mes: 0 Delta Pack: 0 Delta sec : -1
> ==================================
>
> Monitor>
> ============================
> Status at lando V 3.16. 1 (state 1, gstate 1) after 156231 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37087315 tok_hurry : 143252 memb change: 3
> sent pack: 119740 recv pack : 120850 retrans : 178
> u retrans: 32 s retrans : 146 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 91 Delta Pack: 0 Delta sec : 35
> ==================================
>
> Monitor>
> ============================
> Status at greedo V 3.16. 1 (state 1, gstate 1) after 156219 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37086133 tok_hurry : 143241 memb change: 2
> sent pack: 120557 recv pack : 187466 retrans : 104
> u retrans: 0 s retrans : 104 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 88 Groups : 3 Window : 60
> Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
> Delta Mes: -20 Delta Pack: 0 Delta sec : -12
> ==================================
>
> Monitor>
> ============================
> Status at jabba V 3.16. 1 (state 1, gstate 1) after 156231 seconds :
> Membership : 5 procs in 1 segments, leader is obi
> rounds : 37087313 tok_hurry : 143252 memb change: 3
> sent pack: 3 recv pack : 595373 retrans : 67407
> u retrans: 67407 s retrans : 0 b retrans : 0
> My_aru : 287125 Aru : 285524 Highest seq: 287125
> Sessions : 0 Groups : 3 Window : 60
> Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
> Delta Mes: 20 Delta Pack: 0 Delta sec : 12
> ==================================
>
> Monitor>
>
> --
> Jacob Davies - Operations Manager
> Quios - Ewingz Systems, Inc
> E: jacob.davies at quios.net M: +1 XXX XXX XXXX
> T: +1 XXX XXX XXXX F: +1 XXX XXX XXXX
>
>
>
> _______________________________________________
> Spread-users mailing list
> Spread-users at lists.spread.org
> http://lists.spread.org/mailman/listinfo/spread-users
--
-------------------------------------------------------
Jonathan R. Stanton jonathan at cs.jhu.edu
Dept. of Computer Science
Johns Hopkins University
-------------------------------------------------------
More information about the Spread-users
mailing list