<div dir="ltr"><div>Please keep in mind that any proposed solution needs to be not only implemented but also be maintained in the future.</div><div>E.g. currently we do not run any mail server at all so introducing that to the build farm would improve complexity and maintenance effort.</div>
<div>This would also increase the need for infrastructure when running custom build farm in the future.</div><div><br></div>If someone has specific proposals how the buildfarm should suppress error message not relevant to the maintainer and would like to contribute a pull request which integrates that improved notification that would be highly appreciated. With Groovy scripts we are already able to retrigger jobs if known error cases appear in the console output. With a little bit of extra work it would be possible to suppress those emails. But then again Jenkins will also send a notification when the job gets back to stable (and detecting that reliable as well will again require more effort).<div>
<br></div><div>Imo we should aim to address the actual problems: e.g. the farm should not loose network connectivity that frequently. If this would only happen once / twice per year we would not need to change the notification system at all (since the maintainers could easily ignore the emails in such rare error conditions). Another approach would be to identify these fatal conditions earlier (e.g. using Groovy scripts) and pause the farm automatically before hundreds of jobs are failing for the same reason.</div>
<div><br></div><div>- Dirk</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 2, 2014 at 4:35 AM, Mike Purvis <span dir="ltr"><<a href="mailto:mpurvis@clearpathrobotics.com" target="_blank">mpurvis@clearpathrobotics.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Not sure how much work this would be compared with modifying the plugin, but putting all the email through an SMTP relay might make it possible to filter by text strings, or simply introduce a 1hr send delay across the board, plus some rules like "if there's more than 1000 emails queued up, assume a system problem and hold/blackhole all of them.<span class="HOEnZb"><font color="#888888"><div>
<br></div><div>M.</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On 2 April 2014 03:30, Tully Foote <span dir="ltr"><<a href="mailto:tfoote@osrfoundation.org" target="_blank">tfoote@osrfoundation.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Preventing email overload is definitely something we work hard to avoid. We know that if there are too many false positives people will simply ignore them. <div>
<br></div><div>And we shutdown the farm as soon as we diagnosed the issue: <a href="http://status.ros.org/" target="_blank">http://status.ros.org/</a> Unfortunately when you run a very parallelized system if there's a systematic failure, such as the code hosting going down, a lot of jobs fail quickly. </div>
<div><br></div><div>One thing from travis testing Travis distinguishes between build/test errors vs configuration errors. There's a ticket open to add this enhancement <a href="https://github.com/ros-infrastructure/buildfarm/issues/116" target="_blank">https://github.com/ros-infrastructure/buildfarm/issues/116</a> but unfortunately this is not something that Jenkins differentiates so it will take a lot of doing to make this happen on top. An approach I could see for this is to customize the emailing plugin and be able to pass it flags earlier in the process confirm that the configuration and setup has completed successfully. And likewise the actual results should be shown the same way too with the job aborting instead of failing when the configuration/setup phase fails. </div>
<span><font color="#888888">
<div><br></div><div>Tully<br>
<div>
<br></div><div><br></div></div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Apr 1, 2014 at 10:22 PM, Dave Coleman <span dir="ltr"><<a href="mailto:davetcoleman@gmail.com" target="_blank">davetcoleman@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>+1!!</div><div> </div><div><br></div><div><br></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 1 Apr 2014 17:45:05 -0500<br>
From: "David Lu!!" <<a href="mailto:davidlu@wustl.edu" target="_blank">davidlu@wustl.edu</a>><br>
To: <a href="mailto:ros-release@code.ros.org" target="_blank">ros-release@code.ros.org</a><br>
Subject: [ros-release] Fwd: Torrents of Emails<br>
Message-ID:<br>
<CABd+9SqtEdzDXEEhaHT=2V8Xb7B1ofi2Tq8Lo4SvU3b7z=<a href="mailto:ZGBw@mail.gmail.com" target="_blank">ZGBw@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<div><br>
<br>
So I'd like to start a hopefully constructive discussion of the build<br>
farm's email practices. I've attached a picture of the onslaught my<br>
inbox just received. I'm fairly certain none of the builds failing is<br>
my fault. But it does lead me to some questions.<br>
<br>
1) Are there settings for email that I haven't set to reduce the<br>
number of emails I get? Or am I automatically subscribed because I'm a<br>
maintainer?<br>
2) Is there a way to condense the emails? If I got a single email<br>
telling me which packages I maintain failed to build, but this seems a<br>
bit much.<br>
3) I've whined about this before, but is there some way we can make<br>
the error messages more legible? I'm sure for people familiar with the<br>
build farm, the errors may make sense, but as a maintainer, I have no<br>
idea what I'm supposed to do. If the answer is nothing, why am I<br>
getting email?<br>
<br>
I realize accomplishing some of these tasks will likely involve<br>
substantial amounts of work, but I feel it merits discussion<br>
nonetheless.<br>
<br>
-David<br></div>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: Screenshot from 2014-04-01 17:30:36.png<br>
Type: image/png<br>
Size: 635975 bytes<br>
Desc: not available<br>
URL: <<a href="http://lists.ros.org/pipermail/ros-release/attachments/20140401/9ae7884d/attachment.png" target="_blank">http://lists.ros.org/pipermail/ros-release/attachments/20140401/9ae7884d/attachment.png</a>><br>
<br>
------------------------------<div><br>
<br>
_______________________________________________<br>
ros-release mailing list<br>
<a href="mailto:ros-release@code.ros.org" target="_blank">ros-release@code.ros.org</a><br>
<a href="http://lists.ros.org/mailman/listinfo/ros-release" target="_blank">http://lists.ros.org/mailman/listinfo/ros-release</a><br>
<br>
<br></div>
End of ros-release Digest, Vol 30, Issue 2<br>
******************************************<br>
</blockquote></div><br></div></div>
<br>_______________________________________________<br>
ros-release mailing list<br>
<a href="mailto:ros-release@code.ros.org" target="_blank">ros-release@code.ros.org</a><br>
<a href="http://lists.ros.org/mailman/listinfo/ros-release" target="_blank">http://lists.ros.org/mailman/listinfo/ros-release</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
ros-release mailing list<br>
<a href="mailto:ros-release@code.ros.org" target="_blank">ros-release@code.ros.org</a><br>
<a href="http://lists.ros.org/mailman/listinfo/ros-release" target="_blank">http://lists.ros.org/mailman/listinfo/ros-release</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
ros-release mailing list<br>
<a href="mailto:ros-release@code.ros.org">ros-release@code.ros.org</a><br>
<a href="http://lists.ros.org/mailman/listinfo/ros-release" target="_blank">http://lists.ros.org/mailman/listinfo/ros-release</a><br>
<br></blockquote></div><br></div>