[Spread-users] regular message rate pausing?

[John Schultz]

Hi Matt,

It sounds to me like your network multicast or broadcast is not working. 

Performance should usually DEGRADE if you put each daemon in its own 
segment versus all in the same segment.  In a single segment a data 
message is usually only sent once through LAN multicast or broadcast.  
In the setup you have described every data message will be sent N times: 
once for each daemon!  So it is very likely that you have a switch or 
router between your machines that isn't doing multi/broadcast 
correctly.  This causes systematic message losses which have to be 
recovered and the ring slows to a crawl.

If you'd like help getting your system to run better you should send 
your spread.conf file and a detailed output of spmonitor when it is 
running in monitor mode under the troublesome setup.  The fact that you 
are still getting such high speeds speaks to the strength of your 
infrastructure.

Also, the suggestion Mike made shouldn't really make much of a 
difference at all if you are going to be pushing anywhere near that much 
traffic.  His suggestion can help a little bit when messages are sent 
pretty rarely (e.g. - 1 msg/sec).  This heuristic is also easily removed 
completely if you expect lots of continuous traffic.

Cheers!
John

Matt Hurd wrote:

>Just a follow up.
>
>Performance was improved in two ways.  By keeping one machine per
>segment and ending up with five segments each with one machine, packet
>rates were consistent enough.  I'm presuming the implied broadcast by
>having all the machines in one segment had problems with sufficiently
>large messages as the message rate would fall off a cliff ( single
>digit m per sec rates) and the delay in delivering messages would
>grow, seemingly without bound ( tens of seconds and growing ).
>
>Anyway, by keeping each machine in its own segment and tweaking the
>spread parameters a little benchmark of ours is seeing 10,000 of our
>messages a second (reliable mode ~ 20K+ network packets per sec) with
>sustained traffic of > 200 Mbs.  More than sufficient.
>
>I'll have to revisit the broadcast with the new spread params sometime
>and see if the rate is any better, but I suspect not given the
>behaviour Mike reported about the sender needing to be the head of the
>group.  At least with each machine in its own segment it will always
>be the lead of the group...
>
>Thanks for the help.
>
>Regards,
>
>Matt.
>
>
>On 19/10/05, Matt Hurd <matt.hurd at gmail.com> wrote:
>  
>
>>Hi Mike,
>>
>>    
>>
>>>On 18/10/05, Mike Perik <mikep at foxriver.com> wrote:
>>>Is your producer the leader of the group.  Put the machine that is producing
>>>the messages as the first machine in the segment and I *think* you'll see
>>>this go away.  I had the same problem.  It's a side-effect of the way the
>>>token is passed around.  If you search the email list's archive for my name
>>>you'll see several discussions about this behavior of Spread.
>>>      
>>>
>>Thanks, I'll take a look.
>>
>>We did have one segment with five machines in our original test group.
>> We now have kind of a star layout with one machine per segment and
>>five segments as this seemed to improve things but it still has the
>>problem I've mentioned.  I'll try benchmarking with spflood I think
>>and try and take our wrapper out of the equation.
>>
>>Regards,
>>
>>Matt.
>>
>>    
>>
>
>_______________________________________________
>Spread-users mailing list
>Spread-users at lists.spread.org
>http://lists.spread.org/mailman/listinfo/spread-users
>
>  
>
---

John Schultz
Spread Concepts LLC
Phn:  301 498 3233
Cell: 443 838 2200