<div><br></div><div>Ok, applied cedric's patches and the rpc outgoing request is down from an average of about 98ms to 56ms, also the returning response is down to an average of 3ms. Which is starting to seem reasonable.<br>
</div><div><br></div><div>About 40-45ms of that outgoing time is spent in the xmlrpcclient->execute call in master::execute (when contacting the master to initialise an rpc link). I didn't bury down any further into the xmlrpc library.</div>
<div><br></div><div>With cedric's patches too, the msg latency dropped to about 6ms, which is getting close to the raw tcp/ip socket times I tested. So its starting to seem reasonable.</div><div><br></div><div>Nice analysis on the timers cedric! And thank you everyone for the input and pointers. Now we've just one alignment trap issue to worry about.<br>
</div><div><br></div><div>Regards,</div><div>Daniel Stonier.</div><div><br></div><div class="gmail_quote">On 6 August 2010 08:13, Daniel Stonier <span dir="ltr"><<a href="mailto:d.stonier@gmail.com">d.stonier@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br><br><div class="gmail_quote"><div class="im">On 5 August 2010 15:35, Cedric Pradalier <span dir="ltr"><<a href="mailto:cedric.pradalier@mavt.ethz.ch" target="_blank">cedric.pradalier@mavt.ethz.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">
Hello Daniel,<br>
<br>
Could this be linked with the high-frequency loops I mentioned in my
posts last week? <br>
Or, could the latency be affected (positively or negatively) by the
changes I was proposing?<br>
<br>
Cheers<div><div><br></div></div></div></blockquote><div><br></div></div><div>That one's on my todo list for this morning already!</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"><div><div>
<br>
On 08/05/10 08:27, Daniel Stonier wrote:
</div></div><blockquote type="cite"><div><div><br>
<br>
<div class="gmail_quote">On 14 July 2010 15:51, Daniel Stonier <span dir="ltr"><<a href="mailto:d.stonier@gmail.com" target="_blank">d.stonier@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">
<div>
<div>On 14 July 2010 03:52, Josh Faust <<a href="mailto:jfaust@willowgarage.com" target="_blank">jfaust@willowgarage.com</a>>
wrote:<br>
>> == NUM_PTS 1000 ==<br>
>><br>
>> board avg_ser avg_deser<br>
>> intel i5 0.000011 0.000025<br>
>> arm1176jzf-s 0.001328 0.003354<br>
><br>
> Ok, so serialization is definitely much slower, but with the
latency test<br>
> you're using it shouldn't affect anything. Good to know though.<br>
><br>
>><br>
>> == Latency Results ==<br>
>><br>
>> IPC NoTCP Bypass<br>
>> intel i5 : 0.4ms 0.10ms 0.05ms<br>
>> arm1176jzf-s : 25.0ms 4.50ms 3.50ms<br>
>><br>
>> ====== Raw Socket Tests ======<br>
>><br>
>> Tests the latencies of sending a single char from server to
client,<br>
>> written using simple posix socket code.<br>
>><br>
>> intel i5 : 0.1ms<br>
>> arm1176jzf-s : 1.4ms<br>
><br>
> Do you have any way of profiling exactly what's taking time,
either with<br>
> gprof, google perf tools, or something else?<br>
> Josh<br>
<br>
</div>
</div>
Since the slowdowns were across the board I started doubting the<br>
platform underneath the ros - ran some test programs I have back on<br>
our very simple non-ros platform and noticed a significant difference.<br>
So I'll rebuild the platform, check some of the flags used on the old<br>
build and make sure I match them on the new one and see how that goes.<br>
Will try gprof in parallel too I think, but its going to be a couple<br>
of days recompiling. Will let you know how it goes and thanks for the<br>
advice.<br>
<div>
<div><br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Got waylaid by another project for a while - sorry to drag out
old cobwebs!<br>
<br>
Optimised everything else, made sure I'm using the correct flags for
the platform and the tcp/ip msg latencies have dropped from 25ms ->
9-10ms, which is starting to perhaps seem reasonable. The rpc latencies
are still horrendous though - ~100ms for one process to get in contact
with another (and about 5ms for the return trip). I managed to get
gprof working on it and compared it to the same processes running on
the intel i5, but its showing up nothing of considerable note. Just
some boost spinlocks contributing an extra 10% worth of cpu time (which
probably dont show up on the i5 simply because they trigger on and off
so fast). But certainly nothing obvious.<br>
<br>
So, I'm guessing that the process (or roscore) is idling/hanging
somewhere. I'm going to have to manually hack some of the ros code and
pull some timings to figure out just where I think (don't know of a
better way).<br>
<br>
The results are at <a href="http://snorriheim.dnsdojo.com/tmp/rpc_latency" target="_blank">http://snorriheim.dnsdojo.com/tmp/rpc_latency</a>
if interested.<br>
</div>
<div> </div>
</div>
</div></div><pre>_______________________________________________
ros-users mailing list
<div><a href="mailto:ros-users@code.ros.org" target="_blank">ros-users@code.ros.org</a>
<a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a>
</div></pre></blockquote>
<br>
<br>
<pre cols="80">--
Dr. Cedric Pradalier
<a href="http://www.asl.ethz.ch/people/cedricp" target="_blank">http://www.asl.ethz.ch/people/cedricp</a>
</pre></div>
<br>_______________________________________________<br>
ros-users mailing list<br>
<a href="mailto:ros-users@code.ros.org" target="_blank">ros-users@code.ros.org</a><br>
<a href="https://code.ros.org/mailman/listinfo/ros-users" target="_blank">https://code.ros.org/mailman/listinfo/ros-users</a><br>
<br></blockquote></div></div></div><br><br clear="all"><br>-- <br><div><div class="h5">Phone : +82-10-5400-3296 (010-5400-3296)<br>Home: <a href="http://snorriheim.dnsdojo.com/" target="_blank">http://snorriheim.dnsdojo.com/</a><br>
Yujin Robot: <a href="http://www.yujinrobot.com/" target="_blank">http://www.yujinrobot.com/</a><br>
Embedded Control Libraries: <a href="http://snorriheim.dnsdojo.com/redmine/wiki/ecl" target="_blank">http://snorriheim.dnsdojo.com/redmine/wiki/ecl</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Phone : +82-10-5400-3296 (010-5400-3296)<br>Home: <a href="http://snorriheim.dnsdojo.com/">http://snorriheim.dnsdojo.com/</a><br>Yujin Robot: <a href="http://www.yujinrobot.com/">http://www.yujinrobot.com/</a><br>
Embedded Control Libraries: <a href="http://snorriheim.dnsdojo.com/redmine/wiki/ecl">http://snorriheim.dnsdojo.com/redmine/wiki/ecl</a><br>