[Spread-users] Sending messages to a fixed set of servers

[Ciprian Tutu]

Hi Christian,

CS> Now, a general database replication service (as you and Yair propose so
CS> nicely) clearly has stronger global consistency requirements than a
CS> persistent message service. Most importantly: unlike a DBMS, a PMS does
CS> not by itself demand the separation in primary and secondary components,
CS> in a persistent message service all components are primary (well at
CS> least that's what I have in mind, or more general: it should be up to
CS> the application to decide whether it requires that level of 
CS> consistency). Do you agree?

Yes, but there is a trade-off. Using a generic PMS gives the
application the flexibility to use it depending on its own consistency
requirements. For a lot of applications, implementing DBMS semantics
in the messaging layer would be overkill and even inefficient.
However, if you really _do_ need DBMS semantics at application level,
using a PMS as a building block is not efficient as you will need to
do another round of acknowledgements at _application level_ (this is
what we call end-to-end acknowledgements in our work). This will
result in significant performance overhead.

In a nutshell, yes - a PMS can be sufficient and efficient enough for
a lot of applications, but as the semantics requirements get stricter,
you are better off enforcing them in the messaging layer.

CS> The persistent messages of any node N can be delivered anywhere as long
CS> as all previous persistent messages from all past components that 
CS> included N have been delivered there, that's my basic consistency 
CS> requirement, plus correct order (happened-before relations) of 
CS> persistent messages and components. Any objections?

I think you still need to be careful about this specification. When
you say "all previous persistent messages from all past components
have been delivered there" this require a level of knowledge that is
almost equivalent to the strictest (ie DBMS) requirements which we use in the
paper that you read. You cannot know if you got _all_ the messages
that originated in a given component unless you subsequently gain
information from every member of that component. This information can
be acquired either by talking directly with each server of the
original component (as you become reconnected at a later point) or
through relaying from other servers which were connected with a server
from the component at a time later than the time when N disconnected.


CS> At my current level of knowledge, this makes sense to me and should be
CS> possible to develop without the need for component-wide consensus. The
CS> algorithm that I have in mind would
CS> 1) guarantee that a node becomes member of a current persistent session
CS> as long as it can receive its own messages
CS> 2) block a node from sending out new messages until it is member of a
CS> persistent session and
CS> 3) queue incoming persistent messages from another node N until all
CS> preceeding persistent sessions that N participated in have been 
CS> delivered to the application

Okay... now I see how your system is weaker than the global persistent
ordering. That was not obvious when I wrote the paragraph above, but
the comments made there are still valid.

However, I would have to say that this system offers you relatively
limited notion of "persistence". Basically it gives you a
per-component persistent delivery (or per-session, but as you describe
them they are equivalent other than a few additional info). I actually
believe that in order to get more use out of such system you do need
to have sessions which are not mapped onto components. If I understand
correctly what you described, a node N that is part of a session S and
temporarily crashes, it will receive only the messages that were sent
within S _before_ N's crash was detected. When N's crash is detected,
a new regular component is delivered by the GC and, according to your
specification, a new session is installed. Since this session does not
contain N, N will not receive the messages that he lost while he was
disconnected/crashed which was what I thought the notion of
"persistent session" would be (similar to a subscription).

If that is not what you need, but merely the guarantee that you don't
lose messages that were sent while you were still connected to a
certain component, then your solution will do that.

Ciprian