[Spread-users] spread delay problem
Adrian Revill
adrian.revill at shazamteam.com
Tue Nov 4 12:12:41 EST 2008
We are seeing what looks like a pause in the spread communication of 2s.
The test set up is 2 master, and 3 worker nodes, the spread leader is
one of the masters.
In the test, we apply load to one of the masters, each job is sent out
to all the workers, the workers in this case just immediately send a
reply back. This normally gives a round trip time for each job of 2-4ms
If the load is being applied to the master that is the spread leader, it
works perfectly.
If the load is applied to master that is not the leader, we occasionally
see 2s round trip times.
With spread running at normal priority it is approx 60 in 10000, and
when we use a real time scheduler this drops to 6 in 10000
We assume this is due to the token being lost, but we can not find any
sign of packet loss, as all the boxes are hardly loaded, running about
4% CPU and that is mostly spreadd.
When we actually run in a mode that does some real work and on a larger
cluster we see many more pauses
Can any one suggest anyway we can investigate further what is failing?
regards
Adrian
Just after start up
Monitor>
============================
Status at master2 V 4.00. 0 (state 1, gstate 1) after 301 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 939 tok_hurry : 402 memb change: 2
sent pack: 192 recv pack : 869 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 539 Aru : 539 Highest seq: 539
Sessions : 4 Groups : 9 Window : 60
Deliver M: 1239 Deliver Pk: 1258 Pers Window: 15
Delta Mes: 1239 Delta Pack: 539 Delta sec : 301
==================================
Monitor>
============================
Status at master1 V 4.00. 0 (state 1, gstate 1) after 305 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 940 tok_hurry : 453 memb change: 2
sent pack: 266 recv pack : 795 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 539 Aru : 539 Highest seq: 539
Sessions : 4 Groups : 9 Window : 60
Deliver M: 1239 Deliver Pk: 1258 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : 4
==================================
Monitor>
============================
Status at opto-5 V 4.00. 0 (state 1, gstate 1) after 428 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 939 tok_hurry : 471 memb change: 4
sent pack: 185 recv pack : 915 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 539 Aru : 539 Highest seq: 539
Sessions : 1 Groups : 9 Window : 60
Deliver M: 1277 Deliver Pk: 1308 Pers Window: 15
Delta Mes: 38 Delta Pack: 0 Delta sec : 123
==================================
Monitor>
============================
Status at opto-7 V 4.00. 0 (state 1, gstate 1) after 423 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 940 tok_hurry : 471 memb change: 4
sent pack: 184 recv pack : 916 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 539 Aru : 539 Highest seq: 539
Sessions : 1 Groups : 9 Window : 60
Deliver M: 1277 Deliver Pk: 1308 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : -5
==================================
Monitor>
============================
Status at opto-6 V 4.00. 0 (state 1, gstate 1) after 426 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 939 tok_hurry : 471 memb change: 4
sent pack: 184 recv pack : 916 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 539 Aru : 539 Highest seq: 539
Sessions : 1 Groups : 9 Window : 60
Deliver M: 1277 Deliver Pk: 1308 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : 3
==================================
After one test run
Monitor>
============================
Status at master2 V 4.00. 0 (state 1, gstate 1) after 655 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 53272 tok_hurry : 10882 memb change: 2
sent pack: 50423 recv pack : 31746 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 81790 Aru : 81790 Highest seq: 81790
Sessions : 4 Groups : 9 Window : 60
Deliver M: 42490 Deliver Pk: 82509 Pers Window: 15
Delta Mes: 42490 Delta Pack: 81790 Delta sec : 655
==================================
Monitor>
============================
Status at master1 V 4.00. 0 (state 1, gstate 1) after 660 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 53273 tok_hurry : 11055 memb change: 2
sent pack: 614 recv pack : 81555 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 81790 Aru : 81790 Highest seq: 81790
Sessions : 4 Groups : 9 Window : 60
Deliver M: 42490 Deliver Pk: 82509 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : 5
==================================
Monitor>
============================
Status at opto-5 V 4.00. 0 (state 1, gstate 1) after 782 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 53272 tok_hurry : 10951 memb change: 4
sent pack: 10360 recv pack : 71853 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 81790 Aru : 81790 Highest seq: 81790
Sessions : 1 Groups : 9 Window : 60
Deliver M: 42528 Deliver Pk: 82559 Pers Window: 15
Delta Mes: 38 Delta Pack: 0 Delta sec : 122
==================================
Monitor>
============================
Status at opto-7 V 4.00. 0 (state 1, gstate 1) after 777 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 53273 tok_hurry : 10951 memb change: 4
sent pack: 10363 recv pack : 71845 retrans : 0
u retrans: 0 s retrans : 0 b retrans : 0
My_aru : 81790 Aru : 81790 Highest seq: 81790
Sessions : 1 Groups : 9 Window : 60
Deliver M: 42528 Deliver Pk: 82559 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : -5
==================================
Monitor>
============================
Status at opto-6 V 4.00. 0 (state 1, gstate 1) after 780 seconds :
Membership : 5 procs in 1 segments, leader is master1
rounds : 53272 tok_hurry : 10951 memb change: 4
sent pack: 10359 recv pack : 71849 retrans : 5
u retrans: 5 s retrans : 0 b retrans : 0
My_aru : 81790 Aru : 81790 Highest seq: 81790
Sessions : 1 Groups : 9 Window : 60
Deliver M: 42528 Deliver Pk: 82559 Pers Window: 15
Delta Mes: 0 Delta Pack: 0 Delta sec : 3
==================================
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
More information about the Spread-users
mailing list