[Spread-users] Sess_validate_read_header: Message has illegal type field 0x80000080
Scott Barvick
sbarvick at revasystems.com
Thu Apr 21 17:32:41 EDT 2005
After more experimentation and upgrading to 3.17.3 (which still showed
the problem), I believe this is the same as the Java problems - we are
trying to send more data into the TCP socket than it can hold. With big
sends and high rates, a temporary increase in load on the daemon will
cause the clients to overrun the TCP buffer. This resulted in the
Sess_read reading in all 0s for the header and then appropriately
calling it invalid.
Is there any way for the SP_multicast() to monitor the TCP queue and
block on a send until there is enough room?
Thanks,
Scott
On Tue, 2005-04-19 at 20:18, Ryan Caudy wrote:
> The interaction there seems fairly normal -- if I had to guess, I
> would say that this points to a memory-corrupting bug. I assume this
> is version 3.17.2 or 3.17.3?
>
> Cheers,
> Ryan
>
> On 4/19/05, Scott Barvick <sbarvick at revasystems.com> wrote:
> > Greetings,
> >
> > I'm getting the following error when running with a few test systems,
> > and I'm curious if others have seen anything similar. I believe we are
> > hitting it medium hard with sends between 2 systems. When I disconnect
> > one system with a hard stop of the app, the other system sees this,
> > processes the membership changes, but then a short time later kills the
> > client session when it receives a type field that has no type bits set
> > (only the endian bit - 0x80000080).
> >
> > I turned on SESSION and GROUP debug logging and included the output
> > below. I was looking through the code to see how a message can get
> > through without the (FIFO_MESS | SELF_DISCARD) bits set as we send them
> > with the SP_multicast() call. It probably is significant that the group
> > just dropped from 2 members to 1 member (the sender), but this works
> > fine in the steady state operation, even with only one member.
> >
> > Any similar experience or thoughts?
> >
> > Thanks,
> > Scott
> >
> > -------------------
> >
> > [...] lots more where this came from
> >
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > ***** Other system goes down ******
> > Send_join: State is 4
> > Send_join: State is 4
> > Memb_handle_token: handling form2 token
> > Handle_form2 in FORM
> > Memb_transitional
> > G_handle_trans_memb:
> > G_handle_trans_memb in GOP
> > G_handle_trans_memb: Received trans memb id of: {proc_id: -1408236782
> > time: 1113940766}
> > Memb_regular
> > Membership id is ( -1408236782, 1113940767)
> > --------------------
> > Configuration at testsys8 is:
> > Num Segments 1
> > 1 239.16.3.18 4803
> > testsys8 172.16.3.18
> > ====================
> > G_handle_reg_memb: with (172.16.3.18, 1113940767) id
> > G_handle_reg_memb in GTRANS
> > G_handle_reg_memb: skipping state transfer for group RTestGroup.
> > G_handle_reg_memb: skipping state transfer for group TTestGroup.
> > G_handle_reg_memb: skipping state transfer for group GTestGroup.
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > ******** start to receive membership messages ******
> > received TRANSITIONAL membership for group RTestGroup
> > Received REGULAR membership for group RTestGroup with 1 members, where I
> > am member 0:
> > #RTEST0#testsys8
> > grp id is -1408236782 1113940767 1
> > Due to NETWORK change. VS set has 1 members:
> > #RTEST0#testsys8
> > received TRANSITIONAL membership for group TTestGroup
> > received TRANSITIONAL membership for group GTestGroup
> > Received REGULAR membership for group TTestGroup with 1 members, where I
> > am member 0:
> > #TTEST0#testsys8
> > grp id is -1408236782 1113940767 1
> > Due to NETWORK change. VS set has 1 members:
> > #TTEST0#testsys8
> > Received REGULAR membership for group GTestGroup with 1 members, where I
> > am member 0:
> > #TTEST0#testsys8
> > grp id is -1408236782 1113940767 1
> > Due to NETWORK change. VS set has 1 members:
> > #TTEST0#testsys8
> > ***** we thought things were ok *******
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> >
> > ****** Something isn't right ************
> > Sess_read: Message has type field 0x80000080
> > Sess_validate_read_header: Message has illegal type field 0x80000080
> > SP_error: (-8) Connection closed by spread
> > Sess_kill: killing session RTEST0 ( mailbox 14 )
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> > Sess_read: Message has type field 0x800000c4
> > Sess_read: queueing message of type 4 with len 0 to the protocol
> >
> > _______________________________________________
> > Spread-users mailing list
> > Spread-users at lists.spread.org
> > http://lists.spread.org/mailman/listinfo/spread-users
> >
More information about the Spread-users
mailing list