[ros-release] [ros-users] Jenkins spams me with same error - 404

[Tully Foote]

Hi David,

We work very hard to keep the amount of emails that jenkins sends to a
minimum. I group the fauilres into three categories. First there are the
good ones, which are the failures where something is failing to build or
test and the maintainer can fix them. The second is that there are specific
known issues on a specific arch or paltform which the maintainers cannot or
chooses not to fix. And sometimes there are systematic failures such as a
service goes offline that will fail jobs early in unrecoverable ways. (For
example a hosting provider like github or ubuntu going offline, or a
filesystem getting corrupted)

For the first case, we need to let the maintainers know that they need to
fix it. At the moment we do not support arch/target specific blacklisting,
but as we deploy the next generation buildfarm we will gain support for
that. And finally for the systematic failures we do our best to turn the
farm off as quickly as possible. Though due to the fact that we're running
a large system, this often results in a deluge of emails. (We can sometimes
run >40 jobs in parallel, which means a momentary outage can
simulteaneously break 40 jobs, and there's a queue of sometimes thousands
of jobs so if an executor fails quickly it can clear the queue.)

But these systematic events are rare, so we have to play the tradeoff
between the severity and the effort to find a way to catch all failures. We
already have spent a bunch of time making things like the hash sum mismatch
errors much more robust, with auto retry 10 times before failing, which is
a cause of some of the intermittent failures on the current farm.

We completely understand that too many emails will make people numb, and
start ignoring/filtering the emails. But we also want to provide everyone
with prompt notifications when there is a failure.

So I guess I'd ask that you bring up any specific instances where you feel
like you're getting too many emails and we can try to address that.

Tully

PS please use the mailing lists on lists.ros.org not code.ros.org it's
forwarding right now but the canonical addresses are lists.ros.org

On Thu, Nov 19, 2015 at 3:04 PM, David Lu!! <davidvlu at gmail.com> wrote:

> [changed to ros-release]
>
> As for all the emails, it might be time to revive this conversation.
> http://lists.ros.org/pipermail/ros-release/2014-April/004376.html
>
> On Sat, Nov 14, 2015 at 2:56 AM, Vladimir Ermakov via ros-users
> <ros-users at lists.ros.org> wrote:
> > Never mind, jenkins rebuild that package when i wrote last email.
> >
> > 2015-11-14 10:53 GMT+03:00 Vladimir Ermakov <vooon341 at gmail.com>:
> >>
> >> Thank you, Tully.
> >>
> >> All passed except
> >> http://jenkins.ros.org/job/ros-jade-mavros_binarydeb_vivid_i386/45/
> >> But now is apt-get related.
> >>
> >> W: Failed to fetch
> >>
> http://us.archive.ubuntu.com/ubuntu/dists/vivid-updates/main/binary-i386/Packages
> >> Hash Sum mismatch
> >>
> >> Looks like squid related, but that errors at least do not spam :)
> >>
> >> // Sad that jenkins have no "errored" state like travis.
> >>
> >>
> >> 2015-11-14 9:20 GMT+03:00 Tully Foote <tfoote at osrfoundation.org>:
> >>>
> >>> Hi Vladimir,
> >>>
> >>> I just finished fixing that temporarily on your release jobs. You
> should
> >>> have just received one more email saying it's passing now.
> >>>
> >>> I've ticketed it at:
> >>> https://github.com/ros-infrastructure/buildfarm/issues/219
> >>>
> >>> Tully
> >>>
> >>> On Fri, Nov 13, 2015 at 9:59 PM, Vladimir Ermakov via ros-users
> >>> <ros-users at lists.ros.org> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> I receive lots of messages from jenkins with same error. Can you
> please
> >>>> fix it?
> >>>> http://jenkins.ros.org/job/ros-jade-mavros-msgs_sourcedeb/98/console
> >>>>
> >>>> urllib2.HTTPError: HTTP Error 404: Not Found
> >>>> (https://raw.github.com/ros/rosdistro/master/index.yaml)
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ros-users mailing list
> >>>> ros-users at lists.ros.org
> >>>> http://lists.ros.org/mailman/listinfo/ros-users
> >>>>
> >>>
> >>
> >
> >
> > _______________________________________________
> > ros-users mailing list
> > ros-users at lists.ros.org
> > http://lists.ros.org/mailman/listinfo/ros-users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ros.org/pipermail/ros-release/attachments/20151123/216ed9ab/attachment.html>