[Spread-users] producer/consumer queue via spread.
John Lane Schultz
jschultz at spreadconcepts.com
Tue Mar 20 19:43:24 EDT 2007
Allan Bailey wrote:
> Ok. So I think I'm going to have to write a "Queue" protocol that the producer
> puts a limited set of items in the queue, and waits for consumers to ack them.
> The next problem is this: how can I guarantee that only 1 consumer consumes 1 item?
You could have the consumers work in a round robin fashion, where their place in
the membership list order determines who handles which messages. For example,
say the membership list is:
consumerX at daemon1
consumerY at daemon1
consumerA at daemon2
consumerB at daemon2
All the consumers will get the same membership message and perceive the same
ordering of the membership list (Spread guarantees this). If you use
AGREED_MESS service type, then *all consumers will receive all the same messages
and in the same order*. So, then you can have consumerX handle the first
producer msg after the membership message; consumerY will handle the second;
consumerA handles the third; consumerB handles the fourth; consumerX handles the
fifth and so on modulo 4 (the size of the membership list).
> I guessing that I will need to work out the ACK aspect of the
> protocol. I was hoping there would be someway to have multiple
> consumers take an item and ACK it w/o having to check for the race
> condition. Any pointers would be appreciated.
If you have multiple consumers and a single producer, then each consumer can ACK
back to the producer every say, 200 msgs, that they read from Spread stating how
many msgs they have read in the current membership. Then the producer can block
if he wants to send but sees a consumer 600 (or more) msgs behind in ACK'ing.
If you have multiple consumers and producers, then you will probably have to
have producers also join the group and generalize the above scheme so that no
one falls too far behind.
I'm not sure what you mean by "race condition." In Spread, each member joined
to a group gets *ALL* of the messages sent to that group. So consumers are not
"fighting" over getting messages from Spread -- they each get all of the
messages. If you only want 1 consumer to handle each message, then you will
need to do something like the scheme I laid out above.
Spread Concepts LLC
Phn: 443 838 2200
Fax: 301 560 8875
More information about the Spread-users