[Spread-users] main concern on High CPU utilization

Fri Mar 6 12:21:49 EST 2009

Hi Sandeep,

If you are going to use Spread v3, then I recommend you upgrade to at least
v3.17.4 as there were a bunch of bug fixes applied since v3.16.0 and I'd
rather not re-diagnose known bugs here that have already been fixed.

As I said previously, Spread is something of a CPU hog.  If you are sending
significant amounts of traffic through it, then you should expect it to hit
the CPU significantly hard.  No one, to my knowledge, has done in depth
profiling of Spread's code to find its most inefficient and expensive parts,
which surely exist, and optimize them.

So, how much throughput are you pushing through your daemons when you see
this 30% loading?  I am not intimately familiar with the architecture on
which you are running.  So, how strong are the CPUs on which you are running
relative, to say, today's top of the line Intel processors?  For example,
are these CPUs meant for embedding in consumer electronics or are they
general purpose CPUs meant to power desktops or servers?

Along similar lines, I am not familiar with the Power PC OS.  I don't know
how well Power PC supports Spread's way of doing network I/O (i.e. - select
+ non-blocking sockets).  For example, Windows does not support that model
very well, which can cause lower than expected performance and/or higher
than expected CPU utilization.  Do you or anyone else on the mailing list
know how well Power PC supports select + non-blocking socket I/O?

One part of Spread that I know needs optimization is how it deals with lots
of client connections.  How many clients do you have connecting to each
daemon when you see this high load?

If you are seeing 30% load when no clients are connected, your CPUs have
some decent amount of strength and you have a sane OS, then something is
almost surely wrong with your configuration and/or your network that is
causing Spread to do a lot of extra work to overcome that issue.  I
discussed an example of this in a previous email when the segment address
was wrong, which can cause Spread to always lose traffic and have to
overcome it by doing a point to point resend for all traffic.

So, to rule that out, even if you think multicast and broadcast work in your
network, please run the unicast configuration that I suggested and tell me
if you see no difference in the CPU load:

Spread_Segment 0.0.0.0:8800 {
        Node1   192.168.4.70
}

Spread_Segment 0.0.0.0:8800 {
        Node2   192.168.4.72
}

If that doesn't work, another possibility is that your network itself is
seeing high loss due to a bad NIC or switch port, etc.  In a previous email,
I think you said you attached logs of sp_monitor's output, but I did not get
those attachments, nor do I see them in the email archive on spread.org.
So, could you run sp_monitor while the above configuration is running at
high load, monitoring both daemons and report the results by just pasting
them directly into an email?

One final question, did you make that change that we discussed earlier about
reducing the test on Token_counter in To_hold_token() down from 100 to 1?
I'd recommend you do that too as that is how the test is in the newer
versions of Spread v3 too.

Cheers!

---
John Lane Schultz
Spread Concepts LLC
Phn: 443 838 2200 
________________________________________
From: spread-users-bounces at lists.spread.org
[mailto:spread-users-bounces at lists.spread.org] On Behalf Of Sandeep Jeevan
Sent: Friday, March 06, 2009 12:33 AM
To: John Lane Schultz
Cc: spread-users at lists.spread.org
Subject: [Spread-users] main concern on High CPU utilization

Hi John 

I am using version 3.16.00 . Yes Both the daemons are connecting it is being
displayed in the logs .
I had tried all the things that u said in the previous mail but with all the
Configurations its is showing the more than 30% of CPU utilization. I had
also tried the changes in spread_params.h but that seems to be of no use . 
Could you suggest me some another way try to fix high CPU utilization
problem.

Sandeep jeevan
Member Technical(Stack)
Mob:9717892153
VNL  | 246, Phase IV, Udyog Vihar, Gurgaon, Haryana 122 015, INDIA |
+91-124-4311600-609 | F +91-124-4104766 | www.vnl.in

-----Original Message-----
From: John Lane Schultz [mailto:jschultz at spreadconcepts.com] 
Sent: Thursday, March 05, 2009 10:14 PM
To: Sandeep Jeevan
Cc: spread-users at lists.spread.org
Subject: Re: [Spread-users] main concern on High CPU utilization

First off, if those are supposed to be Spread configuration files, then I
think you've got them pretty wrong.  In fact, if you are running Spread
version 4, then I'd be surprised if your daemons are communicating with one
another at all.

In general, you want all the daemons in a configuration to run from the same
configuration file*.  More specifically, the segment portions of the
configurations should be exactly the same: same # of segments, listed in the
same order, each segment address should match up and each segment should
contain the same nodes listed in the same order.

So, the first problem that jumps out at me is that your two different
daemons are running from slightly different configuration files.  In
particular, the address of the segment is different:

Server 1: Gcs_Segment 192.168.4.72:8800 ...
Server 2: Gcs_Segment 192.168.4.70:8800 ...

Normally, with Spread v4, I would expect the two daemons to compute
different configuration hashes and, therefore, refuse to talk to one
another.  I wouldn't expect lots of extra CPU overhead in such a case.  So,
some initial questions are:  What version of Spread are you running?  Are
the daemons reporting that they are connecting with one another on the
command line?

Now, if you are running Spread v3 or if you are running Spread v4 and you
gave us a slightly incorrect listing of your configuration files in your
email and they both had the same segment address, such as:

Server 1: Gcs_Segment 192.168.4.72:8800 ...
Server 2: Gcs_Segment 192.168.4.72:8800 ...

Then that might explain why you are seeing high CPU usage.  When you place
multiple nodes in the same segment, you are telling Spread that if the nodes
in the segment send a message to the Segment address, then all of the nodes
in the segment will likely receive the message (i.e. - the Segment address
is a multicast or broadcast address).

In your file(s) you put one of the machines IPs as the Segment address. 
Bizarrely, because you put the opposite machine's IP address there and you
only have two nodes in the segment, that might actually work, but that is
certainly not the designed intention of the Segment address and I can't
rightly say what should or would happen with the configuration you gave it.

Normally, when someone puts an incorrect Segment address, such as one of the
machine's IP address, then what happens is that the message is sent there
and the other daemons never receive it.  So, they all request re-sends of
the packet, which often simply just sends it to the segment address again. 
There are is a fallback mechanism when the segment address doesn't seem to
be working or only one other daemon requests a resend (as would be in your
case).  In that case, the daemon will eventually resend the message using
unicast, which will work in your configuration.

So, right now, I would say fix your configuration first and see if the CPU
issue remains.  I would recommend running the following configuration on
both of your daemon, listed in order of preference:

If you have multicast working between the servers then:

Spread_Segment 227.227.227.227:8800 {
        Node1   192.168.4.70
        Node2   192.168.4.72
}

If you have broadcast working between the servers and it is a /24 network:

Spread_Segment 192.168.4.255:8800 {
        Node1   192.168.4.70
        Node2   192.168.4.72
}

Otherwise:

Spread_Segment 0.0.0.0:8800 {
        Node1   192.168.4.70
}

Spread_Segment 0.0.0.0:8800 {
        Node2   192.168.4.72
}

Cheers!
John

---
John Lane Schultz
Spread Concepts LLC
Phn: 443 838 2200
Fax: 301 560 8875

Thursday, March 5, 2009, 8:12:32 AM, you wrote:

Hi

Server 1 (IP 192.168.4.70)
Gcs_Segment 192.168.4.72:8800{
               Node1 192.168.4.70
               Node2  192.168.4.72
         }
Server 2 (IP 192.168.4.72)
Gcs_Segment 192.168.4.70:8800{
               Node1 192.168.4.70
               Node2  192.168.4.72
         }
Above are the two given config files for individuals spread deamons

I am using the Power PC operating system (with Motorola processor) .
And I had tried all the things given in the doc of the spread doc available
at the site  but not able to reduce the CPU utilization it is taking around
30%.
I am attaching the monitor snapshots too for analysis .
Could you please send me some doc (Low level Design doc/ doc for
understanding of code flow ) for better understanding of the spread process.
Or Could you please suggest me some way to reduce the CPU utilization.
Waiting for your early response .
Sandeep jeevan
Member Technical(Stack)
Mob:9717892153
VNL  | 246, Phase IV, Udyog Vihar, Gurgaon, Haryana 122 015, INDIA |
+91-124-4311600-609 | F +91-124-4104766 | www.vnl.in

From: John Lane Schultz [mailto:jschultz at spreadconcepts.com]
Sent: Wednesday, March 04, 2009 9:31 PM
To: Sandeep Jeevan
Cc: spread-users at lists.spread.org
Subject: RE: [Spread-users] Problem with token sending module (main concern
on High CPU utilization)

As you can see, the only difference is in the test on the Token_counter. 
This function determines whether or not a ring leader should stop
circulating the token.  When true, this test basically puts the system into
a “dormant” mode, compared to an “active” mode that keeps the token
circulating as fast as possible.

In the 4.x.x version, it will stop circulating the token after all daemons
have acknowledged receiving all traffic and the token makes 1 additional
circulation.  In the 3.x.x. version, it does the same but only after the
token makes 100 additional circulations.

This change was made because we were getting complaints from people that the
token continued to circulate for some time even when no user traffic was
flowing.  So, this way there is likely less token traffic.  The drawback to
doing this is that if a daemon wanted to send after the token stopped
circulating but before the token_hurry timeout, then it would need to send a
request for the token to the ring leader who would then begin circulating it
rather than the token just coming to it automatically if it had continued
circulating.  In other words, if your system has low amounts of activity,
then the optimization that reduces the token traffic will likely increase
the latency of sending and delivering messages.

You can change the number any way you like without harming the overall
functioning of the protocol.  It will just raise or lower how aggressive
Spread is in trying to optimize for low latency.  The higher the # the
longer the token will continue circulating after no new traffic is injected.
Cheers!

---
John Lane Schultz
Spread Concepts LLC
Phn: 443 838 2200

From: spread-users-bounces at lists.spread.org
[mailto:spread-users-bounces at lists.spread.org] On Behalf Of Sandeep Jeevan
Sent: Wednesday, March 04, 2009 12:26 AM
To: John Lane Schultz
Cc: spread-users at lists.spread.org
Subject: Re: [Spread-users] Problem with token sending module (main concern
on High CPU utilization)

Dear John

Could you please guide me for this in version 4.x.x in protocol.c

static  int To_hold_token()
{
  if( ( Memb_state() == OP ||
        ( Memb_state() == GATHER && Memb_token_alive() ) )&&
      Get_retrans(Last_token->type) <= 1      &&
      Aru == Highest_seq && Token_counter > 1 ) return ( 1 );
  else return( 0 );
}

While in version 3.x.x in protocol.c

static  int To_hold_token()
{
  if( ( Memb_state() == OP ||
        ( Memb_state() == GATHER && Memb_token_alive() ) )&&
      Get_retrans(Last_token->type) <= 1      &&
      Aru == Highest_seq && Token_counter > 100 ) return ( 1 );
  else return( 0 );
}

If I change 100 to 1 will it impact my system anyway in 3.x.x

Sandeep jeevan
Member Technical(Stack)
Mob:9717892153
VNL  | 246, Phase IV, Udyog Vihar, Gurgaon, Haryana 122 015, INDIA |
+91-124-4311600-609 | F +91-124-4104766 | www.vnl.in

From: John Lane Schultz [mailto:jschultz at spreadconcepts.com]
Sent: Tuesday, March 03, 2009 8:36 PM
To: Sandeep Jeevan; spread-users at lists.spread.org
Subject: RE: [Spread-users] Problem with token sending module (main concern
on High CPU utilization)

It looks like for the first send he is retransmitting it due to a suspected
token loss or another daemon’s request at line 2.  On lines 7 and 27,
however, this process had just received the token before he sent it on
again.

The current token ring algorithm will forward the token as fast as possible
so long as any of the daemons has user traffic to send to try and minimize
the latency of messages.  Only if the all the daemons have no more user data
to send will the ring leader then hold the token and stop passing it
around.  In that case, he will hold it for the token_hurry timeout before
passing it anyway for failure detection.
Cheers!

---
John Lane Schultz
Spread Concepts LLC
Phn: 443 838 2200

From: spread-users-bounces at lists.spread.org
[mailto:spread-users-bounces at lists.spread.org] On Behalf Of Sandeep Jeevan
Sent: Tuesday, March 03, 2009 1:42 AM
To: spread-users at lists.spread.org
Subject: [Spread-users] Problem with token sending module (main concern on
High CPU utilization)

Following the  logs that I got after enabling the logs.
I am facing a unique problem that is after  line numbered 11 I am
continuously sending token after every 1 milli second and while the token
timing in the memembership.c (200000 ) could any body let me know why it
happens so .

1 [Tue 03 Mar 2009 10:42:03] DL_send: sent a message of 24 bytes to
(192.168.4.70,8801) on channel 5
2 [Tue 03 Mar 2009 10:42:03] Prot_token_hurry: retransmiting token 13 1
3 [Tue 03 Mar 2009 10:42:03] E_handle_events: next event
4 [Tue 03 Mar 2009 10:42:03] E_handle_events: poll select
5 [Tue 03 Mar 2009 10:42:03] E_handle_events: exec handler for fd 5, fd_type
0, priority 1
6 [Tue 03 Mar 2009 10:42:03] DL_recv: received 24 bytes on channel 5
7 [Tue 03 Mar 2009 10:42:03] Received Token
8 [Tue 03 Mar 2009 10:42:03] dispose: disposing pointer 0x1013a0d8 to object
type 20 named scatter
9 [Tue 03 Mar 2009 10:42:03] dispose: disposing pointer 0x101389a0 to object
type 27 named down_link
10[Tue 03 Mar 2009 10:42:03] Send_new_packets: packet 292 sent and inserted
11[Tue 03 Mar 2009 10:42:03] Net_flush_bcast: Flushing with Queued_bytes =
896; num_elements in scat = 2; size of scat0,1 = 32 864
12[Tue 03 Mar 2009 10:42:03] Net_flush_bcast Num_send_needed =0
13[Tue 03 Mar 2009 10:42:03] Net_send_token:before milli 400:Tue Mar  3
10:42:03 2009

[Tue 03 Mar 2009 10:42:03] ifndef ARCH_SCATTER_NONE ::::$$$ DL_send:sendmsg
called ret =24 num_try 0
[Tue 03 Mar 2009 10:42:03] DL_send: sent a message of 24 bytes to
(192.168.4.70,8801) on channel 5
[Tue 03 Mar 2009 10:42:03] dispose: disposing pointer 0x1010f3e0 to object
type 35 named time_event
[Tue 03 Mar 2009 10:42:03] E_queue: dequeued a (first) simillar event
[Tue 03 Mar 2009 10:42:03] E_queue: (first) event queued func 0x1001e574
code 0 data 0x0 in future (0:200000)
[Tue 03 Mar 2009 10:42:03] dispose: disposing pointer 0x10138098 to object
type 35 named time_event
[Tue 03 Mar 2009 10:42:03] E_queue: dequeued a simillar event
[Tue 03 Mar 2009 10:42:03] E_queue: event queued for func 0x10010958 code 0
data 0x0 in future (0:500000)
[Tue 03 Mar 2009 10:42:03] dispose: disposing pointer 0x1010f360 to object
type 8 named token_head_obj
[Tue 03 Mar 2009 10:42:03] E_handle_events: next event
[Tue 03 Mar 2009 10:42:03] E_handle_events: poll select
[Tue 03 Mar 2009 10:42:03] E_handle_events: exec handler for fd 5, fd_type
0, priority 1
[Tue 03 Mar 2009 10:42:03] DL_recv: received 24 bytes on channel 5
[Tue 03 Mar 2009 10:42:03] Received Token
[Tue 03 Mar 2009 10:42:03] Net_send_token:before milli 400:Tue Mar  3
10:42:03 2009

[Tue 03 Mar 2009 10:42:03] ifndef ARCH_SCATTER_NONE ::::$$$ DL_send:sendmsg
called ret =24 num_try 0
[Tue 03 Mar 2009 10:42:03] DL_send: sent a message of 24 bytes to
(192.168.4.70,8801) on channel 5
[Tue 03 Mar 2009 10:42:03] dispose: disposing pointer 0x1010f460 to object
type 35 named time_event
[Tue 03 Mar 2009 10:42:03] E_queue: dequeued a (first) simillar event
[Tue 03 Mar 2009 10:42:03] E_queue: (first) event queued func 0x1001e574
code 0 data 0x0 in future (0:200000)
[Tue 03 Mar 2009 10:42:03] dispose: disposing pointer 0x1010f3e0 to object
type 35 named time_event
[Tue 03 Mar 2009 10:42:03] E_queue: dequeued a simillar event

Sandeep jeevan
Member Technical(Stack)
Mob:9717892153
VNL  | 246, Phase IV, Udyog Vihar, Gurgaon, Haryana 122 015, INDIA |
+91-124-4311600-609 | F +91-124-4104766 | www.vnl.in

The information contained in this e-mail is private & confidential and may
also be legally privileged. If you are not the intended recipient, please
notify us, preferably by e-mail, and do not read, copy or disclose the
contents of this message to anyone.