<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>The 100 ms SpreadConnection socket timeout (in the Java interface) causes problems when latency is introduced into a network and/or messages are very large.</div><div><br></div><div>The error occurs when the delay between chunks of a fragmented Spread message is greater than 100 ms. When this happens, one of the four socket reads in the SpreadConnection's internal_receive() function throws an InterruptedIOException. The next time internal_receive is called, it incorrectly assumes that the next few bytes are the start of a new header, not the remainder of the previous message. </div><div><br></div><div>Parsing this garbage header results in NegativeArraySize exceptions (if the multiplication overflows), OutOfMemory errors (if numGroups is sufficiently large) at line 1197 in SpreadConnection.java (shown below), and/or "Illegal Message: Message Dropped" Spread exceptions (if numGroups or datalen is < 0, line 1119).</div><div><br></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><span style="color: #941c64">byte</span> buffer[] = <span style="color: #941c64">new</span> <span style="color: #941c64">byte</span>[numGroups * <span style="color: #0431c3">MAX_GROUP_NAME</span>];</div></div><div><br></div><div><div><div>This behavior is consistent with quite a few other threads, but I *think* they all came to the wrong conclusion/solution:</div><div><br></div><div><a href="http://thread.gmane.org/gmane.network.spread.user/1630/focus=1636">http://thread.gmane.org/gmane.network.spread.user/1630/focus=1636</a></div><div><a href="http://thread.gmane.org/gmane.network.spread.user/1848/focus=1858">http://thread.gmane.org/gmane.network.spread.user/1848/focus=1858</a></div></div></div><div><br></div><div><div><div>I was able to trigger the bug by introducing lag with tc/netem on the spread daemon's host machine (I'm running a java client that implements the BasicMessageListener interface on a different machine):</div><div><br></div><div>tc qdisc add dev eth0 root netem delay 200ms 10ms distribution normal</div></div><div><br></div><div>(To remove the lag: tc qdisc del dev eth0 root) </div></div><div><br></div><div>My temporary solution is to increase the timeout to 5 seconds (SpreadConnection.java, line 1637):</div><div><br></div><div><span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; "><span style="color: #0431c3">connection</span></span><span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; ">.</span><span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; "><span style="color: #0431c3">socket</span></span><span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; ">.setSoTimeout(5000);</span></div><div><br></div><div><div>I'm no network Guru, but my quick fix doesn't smell right (I think we'll just run into the problem again on a really laggy network). Is the timeout even necessary? Is there any reason why the timeout was set to 100 ms? Currently, I'm leaning towards a configurable timeout option.</div></div><div><br></div><div>Thanks,</div><div> Marek</div></body></html>