Hi Ken,<div> Thanks for offering to debug this issue remotely for me, but I am glad to let you know that I just fixed this issue. It turns out that it had something to do with my graphics driver. I did a step by step debugging of my rosnode and found out that the interrupted system call error occurred only when openrave was rendering objects loaded in its environment. I have no concrete explanation as to why this happens, but switching to an older driver for my graphics card seems to solve this problem (and a few unrelated ones as well). </div>
<div> I also came across this very interesting read on the connection between Unix sockets and the interrupted system call error: <a href="http://www.madore.org/~david/computers/connect-intr.html">http://www.madore.org/~david/computers/connect-intr.html</a></div>
<div>It turns out that socket implementations in Unix are not all that robust, so "maybe" my faulty graphics driver was taking longer than usual to make reads/writes causing the socket to return with an error, whereas the connection was still being attempted. So, when the code in xmlrpc.py tried to make a connection again, it saw that this connection already existed and returned an error.. <a href="http://www.madore.org/~david/computers/connect-intr.html"></a></div>
<div>The following debug messages in the log files saying that some connections already existed seems to give some backing to my theory:</div><div>rosout-1.log:[roscpp_internal] [2011-01-31 16:31:26,526] [thread 0x7f2d043cc760]: [DEBUG] Publisher update for [/rosout]: already have these connections: </div>
<div><div>rosout-1.log:[roscpp_internal] [2011-01-31 16:31:36,397] [thread 0x7f2cfe2d7700]: [DEBUG] Publisher update for [/rosout]: <a href="http://luk:51689/">http://luk:51689/</a>, already have these connections: </div>
<div>rosout-1.log:[roscpp_internal] [2011-01-31 16:31:39,300] [thread 0x7f2cfe2d7700]: [DEBUG] Publisher update for [/rosout]: already have these connections: <a href="http://luk:51689/">http://luk:51689/</a>, </div><div>
rosout.log:Publisher update for [/rosout]: already have these connections: </div><div>rosout.log:Publisher update for [/rosout]: <a href="http://luk:51689/">http://luk:51689/</a>, already have these connections: </div><div>
rosout.log:Publisher update for [/rosout]: already have these connections: <a href="http://luk:51689/">http://luk:51689/</a>, </div></div><div><br></div><div>Thanks again for your help Ken and congrats to WG and you on releasing Diamondback beta!</div>
<div>-Arjun.</div><div><br></div><div><br></div><div><div class="gmail_quote">On Tue, Feb 1, 2011 at 12:10 AM, Ken Conley <span dir="ltr"><<a href="mailto:kwc@willowgarage.com">kwc@willowgarage.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">The patched xmlrpc.py is intended to catch interrupted system calls.<br>
For whatever reason, the exception is not matching the except blocks.<br>
An aggressive workaround for you would be to just ignore all<br>
exceptions unless on shutdown, e.g. to change the end to:<br>
<div class="im"><br>
except Exception as e:<br>
if self.is_shutdown:<br>
pass<br>
<br>
</div>but this isn't an acceptable general solution. This is the sort of<br>
problem that could be fix in <5 mins if I can get to a terminal where<br>
this happening, but is otherwise difficult to offer debug-by-e-mail<br>
advise.<br>
<font color="#888888"><br>
- Ken<br>
</font><div><div></div><div class="h5"><br>
<br>
On Mon, Jan 31, 2011 at 2:17 PM, Arjun <<a href="mailto:akarjun@gmail.com">akarjun@gmail.com</a>> wrote:<br>
> Hi Ken,<br>
> I made the changes you asked me to in xmlrpc.py and here's the debug message<br>
> I get on the console now:<br>
> Unhandled exception in thread started by <bound method TCPServer.run of<br>
> <rospy.impl.tcpros_base.TCPServer object at 0x29cf4d0>><br>
> Traceback (most recent call last):<br>
> File "/home/aarumbak/ros/ros/core/rospy/src/rospy/impl/tcpros_base.py",<br>
> line 141, in run<br>
> Creating action server for manipulation/right_arm/put<br>
> [ERROR] 1296509499.251038: ERROR: error running XML-RPC server:<br>
> Traceback (most recent call last):<br>
> File "/home/aarumbak/ros/ros/core/rospy/src/rospy/impl/msnode.py", line<br>
> 86, in run<br>
> super(ROSNode, self).run()<br>
> File "/home/aarumbak/ros/ros/core/roslib/src/roslib/xmlrpc.py", line 246,<br>
> in run<br>
> raise Exception("unhandled exception [%s]"%(str(e)))<br>
> Exception: unhandled exception [(4, 'Interrupted system call')]<br>
> (client_sock, client_addr) = self.server_sock.accept()<br>
> File "/usr/lib/python2.6/socket.py", line 197, in accept<br>
> sock, addr = self._sock.accept()<br>
> socket.error: [INFO] 1296509499.252298: Manipulation applet is dying. RIP.<br>
> [Errno 4] Interrupted system call<br>
> I've also attached the log file for my rosnode to this email. The exception<br>
> says that it is an Interrupted system call and it appears to be the socket<br>
> accept call. I was wondering if it would make sense to write a signal<br>
> handler or wrap this with some kind of No_Interrupts macro so that the<br>
> accept call doesnt get interrupted again. But this would ofcourse be<br>
> cosmetic and not treat the real cause. I am going to try debugging this<br>
> more, since getting the software to work is absolutely essential for my<br>
> project. But if I cant go much further, I will try running this on Ubuntu<br>
> 10.04, which for some reason does not seem to give this problem. Thanks a<br>
> lot and please let me know if there are other things I should try to fix<br>
> this.<br>
> -Arjun.<br>
><br>
><br>
><br>
> On Mon, Jan 31, 2011 at 2:59 PM, Ken Conley <<a href="mailto:kwc@willowgarage.com">kwc@willowgarage.com</a>> wrote:<br>
>><br>
>> Hi Arjun,<br>
>><br>
>> The behavior is a bit weird. The socket.error should have been caught<br>
>> in xmlrpc instead of passed on. In Python 2.6, socket.error should be<br>
>> a subclass of IOError.<br>
>><br>
>> Can you try adding a block like this to xmrpc.py and seeing if it<br>
>> changes the behavior? It's basically a copy of the IOError branch,<br>
>> but with the specific socket.error type.<br>
>><br>
>> - Ken<br>
>><br>
>> Before the final except block, after the IOError block:<br>
>><br>
>> except socket.error as (errno, errstr):<br>
>> # check for interrupted call, which can occur if we're<br>
>> # embedded in a program using signals. All other<br>
>> # exceptions break _run.<br>
>> if self.is_shutdown:<br>
>> pass<br>
>> elif errno != 4:<br>
>> self.is_shutdown = True<br>
>> logger.error("serve forever IOError: %s,<br>
>> %s"%(errno, errstr))<br>
>> raise<br>
>><br>
>><br>
>> Also, for more debugging, modify the final except block:<br>
>><br>
>> except Exception as e:<br>
>> if self.is_shutdown:<br>
>> pass<br>
>> else:<br>
>> raise Exception("unhandled exception [%s]"(str(e)) )<br>
>><br>
>> On Mon, Jan 31, 2011 at 9:08 AM, Arjun <<a href="mailto:akarjun@gmail.com">akarjun@gmail.com</a>> wrote:<br>
>> > Hi Ken,<br>
>> > I got the ros core code from the ros1.2 branch again today and rebuilt<br>
>> > it.<br>
>> > It has the code from the patch you sent me and I can also confirm that<br>
>> > the<br>
>> > xmlrpc.py file does end with the code snippet from your email. The debug<br>
>> > message on the console output now has more information and so does the<br>
>> > log<br>
>> > file for my rosnode. The console debug message for my node now says,<br>
>> > Traceback (most recent call last):<br>
>> > File<br>
>> > "/home/aarumbak/ros/ros/core/rospy/src/rospy/impl/tcpros_base.py",<br>
>> > line 141, in run<br>
>> > (client_sock, client_addr) = self.server_sock.accept()<br>
>> > File "/usr/lib/python2.6/socket.py", line 197, in accept<br>
>> > sock, addr = self._sock.accept()<br>
>> > socket.error: [Errno 4] Interrupted system call<br>
>> > [ERROR] 1296493143.981902: ERROR: error running XML-RPC server:<br>
>> > Traceback (most recent call last):<br>
>> > File "/home/aarumbak/ros/ros/core/rospy/src/rospy/impl/msnode.py",<br>
>> > line<br>
>> > 86, in run<br>
>> > super(ROSNode, self).run()<br>
>> > File "/home/aarumbak/ros/ros/core/roslib/src/roslib/xmlrpc.py", line<br>
>> > 221,<br>
>> > in run<br>
>> > self.server.serve_forever()<br>
>> > File "/usr/lib/python2.6/SocketServer.py", line 224, in serve_forever<br>
>> > r, w, e = select.select([self], [], [], poll_interval)<br>
>> > error: (4, 'Interrupted system call')<br>
>> > I have also attached the log files with this email. Thanks a lot!<br>
>> > -Arjun.<br>
>> ><br>
>> ><br>
>> > On Mon, Jan 31, 2011 at 2:37 AM, Ken Conley <<a href="mailto:kwc@willowgarage.com">kwc@willowgarage.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hi Arjun,<br>
>> >><br>
>> >> The necessary info is missing due to a bug in the call to the logger.<br>
>> >> Can you try the attached patch?<br>
>> >><br>
>> >> Also, can you confirm that your roslib/src/roslib/xmlrpc.py ends with<br>
>> >> the code block below?<br>
>> >><br>
>> >> thanks,<br>
>> >> Ken<br>
>> >><br>
>> >> while not self.is_shutdown:<br>
>> >> try:<br>
>> >> self.server.serve_forever()<br>
>> >> except IOError as (errno, errstr):<br>
>> >> # check for interrupted call, which can occur if we're<br>
>> >> # embedded in a program using signals. All other<br>
>> >> # exceptions break _run.<br>
>> >> if self.is_shutdown:<br>
>> >> pass<br>
>> >> elif errno != 4:<br>
>> >> self.is_shutdown = True<br>
>> >> logger.error("serve forever IOError: %s,<br>
>> >> %s"%(errno, errstr))<br>
>> >> raise<br>
>> >> except:<br>
>> >> if self.is_shutdown:<br>
>> >> pass<br>
>> >> else:<br>
>> >> raise<br>
>> >><br>
>> >><br>
>> >> On Sun, Jan 30, 2011 at 10:10 PM, Arjun <<a href="mailto:akarjun@gmail.com">akarjun@gmail.com</a>> wrote:<br>
>> >> > Hi Ken,<br>
>> >> > Thanks a lot for offering to take a look. I've attached the log file<br>
>> >> > for<br>
>> >> > my<br>
>> >> > node and also the master.log file with this email. I mentioned<br>
>> >> > openrave<br>
>> >> > only<br>
>> >> > because the previous person with the socket error used it as well and<br>
>> >> > I<br>
>> >> > was<br>
>> >> > wondering if there was some connection there.<br>
>> >> > -Arjun.<br>
>> >> ><br>
>> >> > On Mon, Jan 31, 2011 at 12:29 AM, Ken Conley <<a href="mailto:kwc@willowgarage.com">kwc@willowgarage.com</a>><br>
>> >> > wrote:<br>
>> >> >><br>
>> >> >> Hi Arjun,<br>
>> >> >><br>
>> >> >> I would need the log file from your actual node instead. The rosout<br>
>> >> >> log file is just for the rosout node. There should be a log file if<br>
>> >> >> you "roscd log" and look for your node's name.<br>
>> >> >><br>
>> >> >> Regardless, the patch for xmlrpc.py is not relevant here, as this is<br>
>> >> >> a<br>
>> >> >> different section of code. The log file would hopefully provide<br>
>> >> >> more<br>
>> >> >> detail as to whether the above error is the cause or just a symptom.<br>
>> >> >> I haven't used rospy inside of openrave, so I'm not sure I can be of<br>
>> >> >> much help, though I could think of ways to make the code more robust<br>
>> >> >> to whatever the problem is.<br>
>> >> >><br>
>> >> >> - Ken<br>
>> >> >><br>
>> >> >> On Sun, Jan 30, 2011 at 1:46 AM, Arjun <<a href="mailto:akarjun@gmail.com">akarjun@gmail.com</a>> wrote:<br>
>> >> >> > Hi all,<br>
>> >> >> > I am getting a socket error when I launch my program (which uses<br>
>> >> >> > openrave).<br>
>> >> >> > The launch file launches a single node running on the same machine<br>
>> >> >> > as<br>
>> >> >> > the<br>
>> >> >> > roscore. I am using Ubuntu 10.10 and my Python install is version<br>
>> >> >> > 2.6. I<br>
>> >> >> > did<br>
>> >> >> > look up the archives and found that someone else had mentioned<br>
>> >> >> > this<br>
>> >> >> > same<br>
>> >> >> > problem about a couple of weeks ago and Ken Conley had addressed<br>
>> >> >> > it.<br>
>> >> >> > I<br>
>> >> >> > followed the advice from Ken Conley in that thread and changed my<br>
>> >> >> > .rosinstall file to install from the ros1.2 branch instead just<br>
>> >> >> > for<br>
>> >> >> > the<br>
>> >> >> > ros<br>
>> >> >> > stack. This gave me the latest xmlrpc.py file, which I verified<br>
>> >> >> > with<br>
>> >> >> > the<br>
>> >> >> > previous thread, but I still get the same error.<br>
>> >> >> > Here's the error:<br>
>> >> >> > Unhandled exception in thread started by <bound method<br>
>> >> >> > TCPServer.run<br>
>> >> >> > of<br>
>> >> >> > <rospy.impl.tcpros_base.TCPServer object at 0x37a0990>><br>
>> >> >> > Traceback (most recent call last):<br>
>> >> >> > File<br>
>> >> >> > "/home/aarumbak/ros/ros/core/rospy/src/rospy/impl/tcpros_base.py",<br>
>> >> >> > line 141, in run<br>
>> >> >> > (client_sock, client_addr) = self.server_sock.accept()<br>
>> >> >> > File "/usr/lib/python2.6/socket.py", line 197, in accept<br>
>> >> >> > sock, addr = self._sock.accept()<br>
>> >> >> > socket[INFO] 1296375790.973933: Manipulation applet is dying. RIP.<br>
>> >> >> > .error: [Errno 4] Interrupted system call<br>
>> >> >> > FYI, we run this software on Ubuntu 10.04 and I've never seen this<br>
>> >> >> > error<br>
>> >> >> > before. I got this error on Ubuntu 9.10 before (strangely the<br>
>> >> >> > problem<br>
>> >> >> > went<br>
>> >> >> > away then) and now in 10.10. Any help would be much appreciated.<br>
>> >> >> > -Arjun.<br>
>> >> >> > attachment: relevant log file.<br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> > _______________________________________________<br>
>> >> >> > ros-users mailing list<br>
>> >> >> > <a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
>> >> >> > <a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> _______________________________________________<br>
>> >> >> ros-users mailing list<br>
>> >> >> <a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
>> >> >> <a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
>> >> ><br>
>> >> ><br>
>> >> > _______________________________________________<br>
>> >> > ros-users mailing list<br>
>> >> > <a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
>> >> > <a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
>> >> ><br>
>> >> ><br>
>> >><br>
>> >> _______________________________________________<br>
>> >> ros-users mailing list<br>
>> >> <a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
>> >> <a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
>> >><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > ros-users mailing list<br>
>> > <a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
>> > <a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
>> ><br>
>> ><br>
>> _______________________________________________<br>
>> ros-users mailing list<br>
>> <a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
>> <a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
><br>
><br>
> _______________________________________________<br>
> ros-users mailing list<br>
> <a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
> <a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
><br>
><br>
_______________________________________________<br>
ros-users mailing list<br>
<a href="mailto:ros-users@code.ros.org">ros-users@code.ros.org</a><br>
<a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
</div></div></blockquote></div><br></div>