[Spread-users] Spread performance

[Pete Emerson]

I've written a bare-bones file transfer using Spread 3.17.4 and the  
latest Perl module.

Everything works, but I'm wondering if my performance results are  
typical or if I might be missing something and can squeeze out more.

The sender is simple, and in my tests, it completes the sending in @4  
seconds for a 100MB file. The receiver is threaded (currently limited  
to 10 threads), and all it does is spit the messages into files and  
then send an md5sum (a separate program assembles the chunks and is  
outside my area of interest for this question). On the receiver end,  
all packets are sent to disk in @40 seconds (2.5MB/s). This is on a  
single box that's virtualized to 4 machines, 1 sender, 3 receivers, so  
the network latency should be optimal. I've played with packet size  
(up to 100KB / packet), the usleep time between sending packets, and  
the number of threads, with varying results. I've also changed the  
type of message (I even made it UNRELIABLE_MESS), with little change  
in results. When I use scp for comparison's sake, I'm getting over 47  
MB/s.

Now, I don't expect to deliver my files at 47 MB/s, despite the  
packets not being encrypted as in scp, but I'm not yet convinced that  
I shouldn't be able to do better than 2.5 MB/s. My question is this:  
is my bottleneck something inherent to Spread, or is my code simply  
deficient and I've missed something obvious? If I can't improve the  
speed, then what I'm looking at is needing @19 nodes (with no slowdown  
via the spread/multicast method!) before an iterative scp is slower.   
Any tips and comments would be appreciated, including moving to  
POE::Component::SpreadClient to get rid of the threads, although I am  
not POE fluent.

Here's the core of the sender:

open INFILE, $file or die "Can't read $file: $!\n";
while (($n = read INFILE, $data, $size) != 0) {
       $i++;
       Spread::multicast($mbox, UNRELIABLE_MESS, 'chat1', 1, "$i| 
$data");
       usleep(250);
}

and the core of the receiver (the handler sub simply takes the data  
and shoves it to disk and marks the thread as joinable):

while (1) {
       my @arr = Spread::receive($mbox);
       while (keys %done >= $max_threads) {
               foreach my $id (keys %threads) {
                       if ($done{$id}) {
                               $threads{$id}->join();
                               delete $done{$id};
                               delete $threads{$id};
                       }
               }
       }
       $thread_id++;
       $done{$thread_id} = 0;
       $threads{$thread_id} = threads->create(\&handler, $mbox,  
$thread_id, @arr);
}

Thanks,
Pete