[Spread-users] Question on Spread for high performance database application

John Schultz jschultz at commedia.cnds.jhu.edu
Mon Feb 6 22:17:08 EST 2006


On Mon, 6 Feb 2006, Shilpa Lawande wrote:

> We are evaluating Spread as a replacement for MPI for a high data
> throughput database application running on a cluster of Linux boxes
> with GigE as interconnect.  Our application may need to scale anywhere
> from several hundred to 1000 nodes.

Spread works on a client-daemon peer-to-peer model.  A deployment usually 
consists of a few tens of daemons (e.g. - 5 to 30).  Each daemon can 
handle several hundred client connections.  So, depending on your exact 
system needs and capabilities, Spread may be able to support up to a 
thousand "client nodes."  However, Spread in its current form usually 
cannot scale beyond a few tens of "daemons nodes."

> b) For point-to-point communication between nodes.  The alternative
> for us is to roll out our own using vanilla TCP sockets.  Significant
> portion of our data traffic is point-to-point and we need high data
> throughput here. We are concerned by the extra hop of going through
> the Spread daemon.

Currently, Spread only emulates point-to-point communication using normal 
multicast communication.  In other words, every message is reliably 
multicast to every daemon in the system and then every daemon that is not 
hosting a client destination for the message drops it.  This is obviously 
not optimal for a system that envisions lots of point-to-point 
communication.

> a) Are there any studies on network performance of Spread vs sockets
> for large amounts of data transfer ?

Not in the particular context of your question.  There have been lots of 
studies in general of throughput and latency performance.  In general 
though, I don't think any studies have looked at CPU overhead that much.

http://www.spreadconcepts.com/publications.html
http://www.cnds.jhu.edu/pub/papers/cnds-2004-1.pdf

> b) How much overhead might Spread introduce in terms of CPU usage ?

It depends on how hard you stress your system in terms of # of msgs/sec 
and Mb/sec.  In general though, Spread is a bit more CPU hungry than your 
typical TCP/IP communications, although it is doing considerably more than 
point-to-point communication too.  In the past, we have seen that on 
gigabit networks, when system throughput goes into the *several hundred* 
Mb/s range, Spread has been CPU bound rather than I/O bound.  This of 
course heavily depends on the strength of your processor.

> c) There is a limit of 128 Spread daemons. This means we may need to have
> one daemon for a set of nodes in our cluster. What is the impact of
> the daemon running on same node as client vs. another ?

The main impact is the cost of the client-daemon communication.  When the 
connection is local the communication should stay within the bounds of the 
machine through IPC (unix pipes) or it might still traverse the IP stack 
(unix/windows).  The traffic should never get out onto the network and the 
in memory bandwidth of a computer is usually much greater than that of a 
network.

If you are connecting remotely to a daemon then the client-daemon 
communication goes through a TCP/IP channel across the network thus eating 
bandwidth.  Therefore, the bandwidth optimal deployment of Spread is to 
have a daemon on every machine with client processes.  This way a message 
will usually only traverse the network once rather than potentially many 
times if lots of clients are remote.

> d) Are there any studies comparing Spread to MPI ?

Not to my knowledge.

Feel free to ask any more specific questions either on this mailing list 
or by directly contacting us at Spread Concepts through 
info at spreadconcepts.com

Cheers!

---
John Schultz
Spread Concepts
Phn: 443 838 2200




More information about the Spread-users mailing list