[Spread-users] patches to libspread/sp.c

[John Schultz]

On Sun, 2 Sep 2007, Daniel F. Savarese wrote:
> I don't know if it's intended for the original code to process only a 
> single high/medium priority event because the comment says
> /* Handle all high and medium priority fd events */
> but
>  treated = 1
> makes the loop exit after the first such event is handled.

The code first handles all expired timed events.  Next, it handles all 
ready high priority events.  Only if no high priority events are ready, 
then it handles all ready medium priority events.  Next, only if the 
minimum processing threshold is set to low priority, then it handles a 
single low priority event in a round robin fashion.

> Warning: the epoll patch processes high/medium and low priority
> events differently than the original code, which may have adverse
> consequences.  It processes all ready high priority events first,
> then all ready medium priority events, then all ready low priority
> events as opposed to a single ready high or medium ready event plus
> a single low priority event.

If you process all ready events regardless of priority level, then the 
higher priority events really only get processing preference when the 
minimum processing threshold is raised (i.e. - lower priority levels are 
blocked), which happens relatively rarely in Spread (e.g. - during daemon 
membership changes).  So, your change largely removes any real meaning for 
the different levels of event priority.

The reason Spread's code only processes a single low priority event before 
calling select again is that higher priority events might have become 
ready while processing that single low priority event.  This way higher 
priority events actually get priority-based preference for processing.

I don't know why the code doesn't similarly only process a single medium 
priority event before calling select again (to defer to high priority 
events that might have become ready).  As the code is currently, medium 
priority events can actually starve while low priority always make some 
progress.

It is *very* true, however, that select and even poll are expensive system 
calls.  Your approach minimizes the number of calls to select/poll, but 
only offers priority-based filtering of events.  The creators of Spread 
would need to discuss if the priority-based processing preference 
currently in Spread is worth the overhead cost of calling select/poll 
(much) more often.

> It's intended only for the case where your spread daemons are serving
> a lot of connections and you may benefit from avoiding select plus
> linear search.

The general idea of using poll (or even more advanced OS event systems 
such as kqueue) and better data structures does sound good to me.

Cheers!

---
John Schultz
Spread Concepts
Phn: 443 838 2200