[Spread-users] patches to libspread/sp.c

[John Robinson]

Daniel F. Savarese wrote:
> Maybe John Robinson,
> who posted the original poll patch for sp.c, can comment on his usage
> at Vertica?

We do not pass large volumes of data through spread; instead it is the 
"control" layer for our application.  Also, we do not expect to share 
either the hosts or the network with other applications (we are greedy). 
  So I did not find a need to change the daemon or the priority 
implementation.

What we do have is potentially thousands of open fd's at a time in the 
spread client.  This means that the fd bit that Mbox winds up on can be 
thousands of positions into the fd_set, with attendant inefficiencies. 
Moreover, it is my understanding (anyone please correct this if wrong) 
that modern Linux (our platform of choice) implements select() using 
poll(), so this seems particularly wasteful.  So having the choice as a 
config option for the product certainly makes sense.

> At the very least, that can be done without changing priority handling
> while still eliminating the need to search through a potentially large
> fd_set.  But I think the select implementation should always remain as the
> cross-platform default implementation, with anything else being a
> configure-time alternative.  In the end, if only a few users are generating
> hundreds of file descriptors via their usage, then it may not be worth
> changing anything at all.  Those of us with edge cases can happily hack
> the code.

That sounds good.

/jr