[Spread-users] Will Spread work for me - large numbers of remote locations connected via satellite internet

[Trever Miller]

Good day,

I am interested in using spread but am unsure if it will fit my environment.

Summary of the problem space, followed by questions

Embedded oil field data capture systems (SCADA) in remote locations
connected to the internet via satellite modem service.  The satellite
modem and embedded data collection/transmission system are on the
internet, pingable, and have 1200-3000 ms latency as measured by ping.
The size of each data payload is quite small, under 20K, and is usually
generated hourly.

We use SSH to connect to the systems for diagnostics and to push updated
configurations and programs to the embedded units.   The data
transmission uses scp driven by shell scripts that play games with
secondary files that contain CRC and other info, in an attempt to ensure
that only full complete data report files make the trip.  Validation
happens on the receiving end in the data center.

The scp based data transport is already having issues, with several
retries being the norm as the ssh connection fails quite often in the
handshake/setup phase due to lost or extremely delayed tcp packets.
The data does eventally get thru, and the remote sites do manage to also
pick up their configuration changes or program updates from the
mothership.   I am not comfortable with the amount of failed and
restarted connections in the original 30 trial sites though.

The number of sites is going to grow to 100 soon, and many hundreds
later this year.  Ultimately in the thousands.

I want to use spread as the new data transport mechanism, and also to
push configuration data, program updates out.

I think I will be using the private group thing for talking to
individual sites, and the sites would talk to a small number of groups
hosted at the server side, where data would be grabbed for processing
and insertion into the application that needs the data.

Here's my questions.  Thanks for reading thru all of the above.

The hop protocol, being based on udp, looks like a good fit here.
Unfortunately my understanding of how spread works, is that the hop
protocol is only used between spread daemons.  Meaning I would have to
put a daemon on each of the embedded units.  OK, I can do that, no
problem.   But... other stuff in the docs and on this list indicate that
there is a hard upper limit of 128 daemons.

Is there something I am missing?

Can the client libs use hop to talk over the high latency wan to the
server processes at the data center?

-- 
trever at cyberdex.ca
Cyberdex Systems Consulting Corp.