[Spread-users] System hangs using spread for logging.
Jacob Davies
jdavies at quios.net
Tue Mar 19 16:50:44 EST 2002
Hello there,
This is sort of a follow-up to an earlier post by Tom Mornini (a
coworker of mine) about the problems we've been having using
spread in our production environment.
Brief recap: we use spread to distribute logging between various
machines -- 2 machines act as log receivers and both of them log
all copies of messages on three channels, all the other machines
log onto those three channels (one general, one Apache access, one
Apache error). What happens is that after a while running like
this -- has been up to two weeks, has been as short as a couple
of hours -- the system hangs completely, presumably all processes
waiting on being able to write more logging output to spread.
When we restart one spread daemon the system generally starts
working again, with a flood of messages logged as soon as we
restart (presumably the contents of various buffers that are
waiting). Sometimes though we need to restart everything (spread
and the applications) to get them to work again.
Previously we were asked for several cycles of spmonitor output
from while the system was hung, so what follows is that -- the system
was hung completely while I was running spmonitor, nobody was able
to put anything into spread or get anything out.
The machine roles are as follows:
obi: logfile receiver on 3 channels
boba: logfile receiver on 3 channels
lando: logfile sender on 3 channels (high traffic on 2, low on 1)
greedo: logfile sender on 3 channels (high traffic on 2, low on 1)
jabba: logfile sender on 1 channels (very low or zero traffic)
Hope this makes some sense -- I'm curious about what further
debugging I can do to help here -- system call traces? netstat
output?
Monitor> 0
=============
Activate Status
-------------
Enter Proc Name: all
Enter Proc Name:
Monitor: send status query
Monitor>
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 156137 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37014803 tok_hurry : 143898 memb change: 1
sent pack: 2 recv pack : 240617 retrans : 113917
u retrans: 113876 s retrans : 41 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: 286934 Delta Pack: 285524 Delta sec : 156137
==================================
Monitor>
============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 156136 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37014804 tok_hurry : 143200 memb change: 1
sent pack: 2 recv pack : 240518 retrans : 141
u retrans: 0 s retrans : 141 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : -1
==================================
Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 156170 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37020613 tok_hurry : 143252 memb change: 3
sent pack: 119740 recv pack : 120850 retrans : 178
u retrans: 32 s retrans : 146 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 91 Delta Pack: 0 Delta sec : 34
==================================
Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 156159 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37019431 tok_hurry : 143241 memb change: 2
sent pack: 120557 recv pack : 187466 retrans : 104
u retrans: 0 s retrans : 104 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
Delta Mes: -20 Delta Pack: 0 Delta sec : -11
==================================
Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 156171 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37020610 tok_hurry : 143252 memb change: 3
sent pack: 3 recv pack : 595373 retrans : 67407
u retrans: 67407 s retrans : 0 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 0 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 20 Delta Pack: 0 Delta sec : 12
==================================
Monitor> Monitor: send status query
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 156147 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37025760 tok_hurry : 143898 memb change: 1
sent pack: 2 recv pack : 240617 retrans : 113917
u retrans: 113876 s retrans : 41 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -91 Delta Pack: 0 Delta sec : -24
==================================
Monitor>
============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 156146 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37025761 tok_hurry : 143200 memb change: 1
sent pack: 2 recv pack : 240518 retrans : 141
u retrans: 0 s retrans : 141 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : -1
==================================
Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 156180 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37031570 tok_hurry : 143252 memb change: 3
sent pack: 119740 recv pack : 120850 retrans : 178
u retrans: 32 s retrans : 146 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 91 Delta Pack: 0 Delta sec : 34
==================================
Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 156169 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37030388 tok_hurry : 143241 memb change: 2
sent pack: 120557 recv pack : 187466 retrans : 104
u retrans: 0 s retrans : 104 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
Delta Mes: -20 Delta Pack: 0 Delta sec : -11
==================================
Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 156181 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37031567 tok_hurry : 143252 memb change: 3
sent pack: 3 recv pack : 595373 retrans : 67407
u retrans: 67407 s retrans : 0 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 0 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 20 Delta Pack: 0 Delta sec : 12
==================================
Monitor> Monitor: send status query
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 156157 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37037647 tok_hurry : 143898 memb change: 1
sent pack: 2 recv pack : 240617 retrans : 113917
u retrans: 113876 s retrans : 41 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -91 Delta Pack: 0 Delta sec : -24
==================================
Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 156190 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37043456 tok_hurry : 143252 memb change: 3
sent pack: 119740 recv pack : 120850 retrans : 178
u retrans: 32 s retrans : 146 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 91 Delta Pack: 0 Delta sec : 33
==================================
Monitor>
============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 156156 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37037648 tok_hurry : 143200 memb change: 1
sent pack: 2 recv pack : 240518 retrans : 141
u retrans: 0 s retrans : 141 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -91 Delta Pack: 0 Delta sec : -34
==================================
Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 156179 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37042274 tok_hurry : 143241 memb change: 2
sent pack: 120557 recv pack : 187466 retrans : 104
u retrans: 0 s retrans : 104 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
Delta Mes: 71 Delta Pack: 0 Delta sec : 23
==================================
Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 156191 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37043454 tok_hurry : 143252 memb change: 3
sent pack: 3 recv pack : 595373 retrans : 67407
u retrans: 67407 s retrans : 0 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 0 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 20 Delta Pack: 0 Delta sec : 12
==================================
Monitor> Monitor: send status query
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 156167 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37048360 tok_hurry : 143898 memb change: 1
sent pack: 2 recv pack : 240617 retrans : 113917
u retrans: 113876 s retrans : 41 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -91 Delta Pack: 0 Delta sec : -24
==================================
Monitor>
============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 156166 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37048361 tok_hurry : 143200 memb change: 1
sent pack: 2 recv pack : 240518 retrans : 141
u retrans: 0 s retrans : 141 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : -1
==================================
Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 156201 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37054170 tok_hurry : 143252 memb change: 3
sent pack: 119740 recv pack : 120850 retrans : 178
u retrans: 32 s retrans : 146 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 91 Delta Pack: 0 Delta sec : 35
==================================
Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 156189 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37052988 tok_hurry : 143241 memb change: 2
sent pack: 120557 recv pack : 187466 retrans : 104
u retrans: 0 s retrans : 104 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
Delta Mes: -20 Delta Pack: 0 Delta sec : -12
==================================
Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 156201 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37054167 tok_hurry : 143252 memb change: 3
sent pack: 3 recv pack : 595373 retrans : 67407
u retrans: 67407 s retrans : 0 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 0 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 20 Delta Pack: 0 Delta sec : 12
==================================
Monitor> Monitor: send status query
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 156177 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37058421 tok_hurry : 143898 memb change: 1
sent pack: 2 recv pack : 240617 retrans : 113917
u retrans: 113876 s retrans : 41 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -91 Delta Pack: 0 Delta sec : -24
==================================
Monitor>
============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 156176 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37058422 tok_hurry : 143200 memb change: 1
sent pack: 2 recv pack : 240518 retrans : 141
u retrans: 0 s retrans : 141 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : -1
==================================
Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 156211 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37064231 tok_hurry : 143252 memb change: 3
sent pack: 119740 recv pack : 120850 retrans : 178
u retrans: 32 s retrans : 146 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 91 Delta Pack: 0 Delta sec : 35
==================================
Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 156211 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37064228 tok_hurry : 143252 memb change: 3
sent pack: 3 recv pack : 595373 retrans : 67407
u retrans: 67407 s retrans : 0 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 0 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : 0
==================================
Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 156199 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37063048 tok_hurry : 143241 memb change: 2
sent pack: 120557 recv pack : 187466 retrans : 104
u retrans: 0 s retrans : 104 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
Delta Mes: -20 Delta Pack: 0 Delta sec : -12
==================================
Monitor> Monitor: send status query
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 156187 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37070425 tok_hurry : 143898 memb change: 1
sent pack: 2 recv pack : 240617 retrans : 113917
u retrans: 113876 s retrans : 41 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -71 Delta Pack: 0 Delta sec : -12
==================================
Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 156221 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37076234 tok_hurry : 143252 memb change: 3
sent pack: 119740 recv pack : 120850 retrans : 178
u retrans: 32 s retrans : 146 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 91 Delta Pack: 0 Delta sec : 34
==================================
Monitor>
============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 156186 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37070426 tok_hurry : 143200 memb change: 1
sent pack: 2 recv pack : 240518 retrans : 141
u retrans: 0 s retrans : 141 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -91 Delta Pack: 0 Delta sec : -35
==================================
Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 156209 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37075052 tok_hurry : 143241 memb change: 2
sent pack: 120557 recv pack : 187466 retrans : 104
u retrans: 0 s retrans : 104 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
Delta Mes: 71 Delta Pack: 0 Delta sec : 23
==================================
Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 156221 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37076232 tok_hurry : 143252 memb change: 3
sent pack: 3 recv pack : 595373 retrans : 67407
u retrans: 67407 s retrans : 0 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 0 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 20 Delta Pack: 0 Delta sec : 12
==================================
Monitor> Monitor: send status query
============================
Status at obi V 3.16. 1 (state 1, gstate 1) after 156197 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37081506 tok_hurry : 143898 memb change: 1
sent pack: 2 recv pack : 240617 retrans : 113917
u retrans: 113876 s retrans : 41 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: -91 Delta Pack: 0 Delta sec : -24
==================================
Monitor>
============================
Status at boba V 3.16. 1 (state 1, gstate 1) after 156196 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37081507 tok_hurry : 143200 memb change: 1
sent pack: 2 recv pack : 240518 retrans : 141
u retrans: 0 s retrans : 141 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 3 Groups : 3 Window : 60
Deliver M: 286934 Deliver Pk: 287125 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : -1
==================================
Monitor>
============================
Status at lando V 3.16. 1 (state 1, gstate 1) after 156231 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37087315 tok_hurry : 143252 memb change: 3
sent pack: 119740 recv pack : 120850 retrans : 178
u retrans: 32 s retrans : 146 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 91 Delta Pack: 0 Delta sec : 35
==================================
Monitor>
============================
Status at greedo V 3.16. 1 (state 1, gstate 1) after 156219 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37086133 tok_hurry : 143241 memb change: 2
sent pack: 120557 recv pack : 187466 retrans : 104
u retrans: 0 s retrans : 104 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 88 Groups : 3 Window : 60
Deliver M: 287005 Deliver Pk: 287199 Pers Window: 15
Delta Mes: -20 Delta Pack: 0 Delta sec : -12
==================================
Monitor>
============================
Status at jabba V 3.16. 1 (state 1, gstate 1) after 156231 seconds :
Membership : 5 procs in 1 segments, leader is obi
rounds : 37087313 tok_hurry : 143252 memb change: 3
sent pack: 3 recv pack : 595373 retrans : 67407
u retrans: 67407 s retrans : 0 b retrans : 0
My_aru : 287125 Aru : 285524 Highest seq: 287125
Sessions : 0 Groups : 3 Window : 60
Deliver M: 287025 Deliver Pk: 287221 Pers Window: 15
Delta Mes: 20 Delta Pack: 0 Delta sec : 12
==================================
Monitor>
--
Jacob Davies - Operations Manager
Quios - Ewingz Systems, Inc
E: jacob.davies at quios.net M: +1 XXX XXX XXXX
T: +1 XXX XXX XXXX F: +1 XXX XXX XXXX
More information about the Spread-users
mailing list