[Spread-users] producer/consumer queue via spread.

[John Lane Schultz]

Allan Bailey wrote:
> Ok.  So I think I'm going to have to write a "Queue" protocol that the producer
> puts a limited set of items in the queue, and waits for consumers to ack them.
> 
> The next problem is this:  how can I guarantee that only 1 consumer consumes 1 item?
> 
You could have the consumers work in a round robin fashion, where their place in 
the membership list order determines who handles which messages.  For example, 
say the membership list is:

consumerX at daemon1
consumerY at daemon1
consumerA at daemon2
consumerB at daemon2

All the consumers will get the same membership message and perceive the same 
ordering of the membership list (Spread guarantees this).  If you use 
AGREED_MESS service type, then *all consumers will receive all the same messages 
and in the same order*.  So, then you can have consumerX handle the first 
producer msg after the membership message; consumerY will handle the second; 
consumerA handles the third; consumerB handles the fourth; consumerX handles the 
fifth and so on modulo 4 (the size of the membership list).

> I guessing that I will need to work out the ACK aspect of the
> protocol.  I was hoping there would be someway to have multiple
> consumers take an item and ACK it w/o having to check for the race
> condition.  Any pointers would be appreciated.
> 
If you have multiple consumers and a single producer, then each consumer can ACK 
back to the producer every say, 200 msgs, that they read from Spread stating how 
many msgs they have read in the current membership.  Then the producer can block 
if he wants to send but sees a consumer 600 (or more) msgs behind in ACK'ing. 
If you have multiple consumers and producers, then you will probably have to 
have producers also join the group and generalize the above scheme so that no 
one falls too far behind.

I'm not sure what you mean by "race condition."  In Spread, each member joined 
to a group gets *ALL* of the messages sent to that group.  So consumers are not 
"fighting" over getting messages from Spread -- they each get all of the 
messages.  If you only want 1 consumer to handle each message, then you will 
need to do something like the scheme I laid out above.

Cheers!

-- 
John Schultz
Spread Concepts LLC
Phn: 443 838 2200
Fax: 301 560 8875