[ros-release] Build Failure due to rosdep

Wed Apr 1 22:58:37 UTC 2015

Hi Piyush,

Unfortunately I believe this is an inherent issue with debian repositories
managed by reprepro. (the tool we use for managing the repository)
Basically when new packages get uploaded it regenerates the metadata and in
the process there's a race condition for fetching the files being
regenerated.

Note this is a much shorter period than the Hash sum mismatch that we
already detect and retry. On the new buildfarm we can catch this one as
well and retry.

Tully

On Wed, Apr 1, 2015 at 3:54 PM, Piyush via ros-release <
ros-release at lists.ros.org> wrote:

> Hey William,
>
> Since we're on the subject, I've been seeing a number of failures
> because of the following message, just in case you can do something
> about it. I believe this error is independent of recent github
> outages:
>
> running command ['reprepro', '-v', '-b', '/var/www/repos/building',
> '-S', 'misc', 'include', u'saucy',
>
> '/tmp/upload/ros-indigo-bwi-gazebo-entities_saucy_i386/ros-indigo-bwi-gazebo-entities_0.3.2-0saucy-20150401-2152-+0000_i386.changes']
> Exporting indices...
> Removing /tmp/upload/ros-indigo-bwi-gazebo-entities_saucy_i386
> + sudo apt-get update -c
> <
> http://jenkins.ros.org/job/ros-indigo-bwi-gazebo-entities_binarydeb_saucy_i386/ws/apt.conf
> >
> -o Apt::Architecture=i386
> Get:1 http://repos.ros.org saucy Release.gpg [198 B]
> Get:2 http://repos.ros.org saucy Release [3,779 B]
> Err http://repos.ros.org saucy Release
>
> Get:3 http://us.archive.ubuntu.com saucy Release.gpg [933 B]
> Get:4 http://us.archive.ubuntu.com saucy-updates Release.gpg [933 B]
> Hit http://us.archive.ubuntu.com saucy Release
> Ign http://us.archive.ubuntu.com saucy Release
> Hit http://us.archive.ubuntu.com saucy-updates Release
> Ign http://us.archive.ubuntu.com saucy-updates Release
> Ign http://us.archive.ubuntu.com saucy/main Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy/restricted Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy/universe Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy/multiverse Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy/main i386 Packages/DiffIndex
> Ign http://us.archive.ubuntu.com saucy/restricted i386 Packages/DiffIndex
> Ign http://us.archive.ubuntu.com saucy/universe i386 Packages/DiffIndex
> Ign http://us.archive.ubuntu.com saucy/multiverse i386 Packages/DiffIndex
> Hit http://us.archive.ubuntu.com saucy/main TranslationIndex
> Hit http://us.archive.ubuntu.com saucy/multiverse TranslationIndex
> Hit http://us.archive.ubuntu.com saucy/restricted TranslationIndex
> Hit http://us.archive.ubuntu.com saucy/universe TranslationIndex
> Ign http://us.archive.ubuntu.com saucy-updates/main Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy-updates/restricted
> Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy-updates/universe Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy-updates/multiverse
> Sources/DiffIndex
> Ign http://us.archive.ubuntu.com saucy-updates/main i386
> Packages/DiffIndex
> Ign http://us.archive.ubuntu.com saucy-updates/restricted i386
> Packages/DiffIndex
> Ign http://us.archive.ubuntu.com saucy-updates/universe i386
> Packages/DiffIndex
> Ign http://us.archive.ubuntu.com saucy-updates/multiverse i386
> Packages/DiffIndex
> Hit http://us.archive.ubuntu.com saucy-updates/main TranslationIndex
> Hit http://us.archive.ubuntu.com saucy-updates/multiverse TranslationIndex
> Hit http://us.archive.ubuntu.com saucy-updates/restricted TranslationIndex
> Hit http://us.archive.ubuntu.com saucy-updates/universe TranslationIndex
> Hit http://us.archive.ubuntu.com saucy/main Sources
> Hit http://us.archive.ubuntu.com saucy/restricted Sources
> Hit http://us.archive.ubuntu.com saucy/universe Sources
> Hit http://us.archive.ubuntu.com saucy/multiverse Sources
> Hit http://us.archive.ubuntu.com saucy/main i386 Packages
> Hit http://us.archive.ubuntu.com saucy/restricted i386 Packages
> Hit http://us.archive.ubuntu.com saucy/universe i386 Packages
> Hit http://us.archive.ubuntu.com saucy/multiverse i386 Packages
> Hit http://us.archive.ubuntu.com saucy/main Translation-en
> Hit http://us.archive.ubuntu.com saucy/multiverse Translation-en
> Hit http://us.archive.ubuntu.com saucy/restricted Translation-en
> Hit http://us.archive.ubuntu.com saucy/universe Translation-en
> Hit http://us.archive.ubuntu.com saucy-updates/main Sources
> Hit http://us.archive.ubuntu.com saucy-updates/restricted Sources
> Hit http://us.archive.ubuntu.com saucy-updates/universe Sources
> Hit http://us.archive.ubuntu.com saucy-updates/multiverse Sources
> Hit http://us.archive.ubuntu.com saucy-updates/main i386 Packages
> Hit http://us.archive.ubuntu.com saucy-updates/restricted i386 Packages
> Hit http://us.archive.ubuntu.com saucy-updates/universe i386 Packages
> Hit http://us.archive.ubuntu.com saucy-updates/multiverse i386 Packages
> Hit http://us.archive.ubuntu.com saucy-updates/main Translation-en
> Hit http://us.archive.ubuntu.com saucy-updates/multiverse Translation-en
> Hit http://us.archive.ubuntu.com saucy-updates/restricted Translation-en
> Hit http://us.archive.ubuntu.com saucy-updates/universe Translation-en
> Fetched 5,843 B in 3s (1,766 B/s)
> Reading package lists...
> W: A error occurred during the signature verification. The repository
> is not updated and the previous index files will be used. GPG error:
> http://repos.ros.org saucy Release: The following signatures couldn't
> be verified because the public key is not available: NO_PUBKEY
> 5523BAEEB01FA116
>
> W: GPG error: http://us.archive.ubuntu.com saucy Release: The
> following signatures couldn't be verified because the public key is
> not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
> W: GPG error: http://us.archive.ubuntu.com saucy-updates Release: The
> following signatures couldn't be verified because the public key is
> not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
> W: Failed to fetch http://repos.ros.org/repos/building/dists/saucy/Release
>
> Piyush
>
> On Wed, Apr 1, 2015 at 3:41 PM, William Woodall via ros-release
> <ros-release at lists.ros.org> wrote:
> > I think it would be better to just tell the user what's going on. If you
> can
> > reliably tell when you should retry, then you can also just tell the
> user to
> > try it again. For the build farm, retrying immediately usually never
> works
> > because its takes time for a failure to get resolved upstream, so we
> retry
> > several times with a reasonable cool off period and that seems to help a
> > bit. For users though I think this kind of thing is not particularly
> worth
> > it.
> >
> > All of that being said if you come up with a way to communicate to the
> user
> > what's going on and also is more robust to intermittent failures, that's
> ok
> > to propose as a pull request. However, I'll point out that most tools
> don't
> > retry and I think there is a good rational behind that to reduce the
> amount
> > of magic between the user and the problems when they occur. For example,
> > programs that don't automatically retry git, apt-get, Homebrew, curl
> > (without an option), wget (without an option), chrome, safari, firefox,
> > etc... Obviously "just because others don't do it" isn't sufficient
> reason
> > but it is a good indicator.
> >
> > On Wed, Apr 1, 2015 at 1:17 PM, G.A. vd. Hoorn - 3ME
> > <g.a.vanderhoorn at tudelft.nl> wrote:
> >>
> >> On 1-4-2015 22:15, William Woodall wrote:
> >> > The new build farm will retry on connection failures but it doesn't
> >> > really
> >> > make sense to retry on 404 in my opinion. It's a really _bad_ failure
> >> > mode
> >> > to temporarily replace valid files with 404 responses. But I imagine
> >> > GitHub
> >> > was being stressed in ways that aren't considered normal operation.
> >>
> >> I agree about the 404 not being the best status code to return, but I
> >> was more asking about whether rosdep itself should try a bit harder.
> >>
> >> Normal desktop users also see this, and quite often assume there is
> >> actually something really wrong, while a simple retry (a la wget) would
> >> let things continue.
> >>
> >>
> >> > On Wed, Apr 1, 2015 at 1:13 PM, G.A. vd. Hoorn - 3ME via ros-release <
> >> > ros-release at lists.ros.org> wrote:
> >> >
> >> >> On 1-4-2015 22:08, William Woodall via ros-release wrote:
> >> >>> That's a problem with GitHub, I think an artifact of the recent DDoS
> >> >> attack
> >> >>> against them. The file that was reportedly 404 above is now
> available.
> >> >>> Unfortunately there's not much we can do about this sort of failure.
> >> >> Would it be an idea to increase the nr of attempts?
> >> >>
> >> >> It seems rosdep gives up after a single 404. Given that github
> >> >> experiences a lot of transient failures (we also see quite some
> >> >> questions like this on ROS Answers), a retry or 2 might go a long
> way?
> >> >>
> >> >>
> >> >>> On Wed, Apr 1, 2015 at 12:36 PM, Shaun Edwards via ros-release <
> >> >>> ros-release at lists.ros.org> wrote:
> >> >>>
> >> >>>> All,
> >> >>>>
> >> >>>> I have had two builds/releases failure for the same issue
> associated
> >> >> with
> >> >>>> rosdep (see snippet below).  rosdep has some errors, but in the
> past
> >> >> this
> >> >>>> has not caused a failure.  I see this same error on the pre-release
> >> >> server
> >> >>>> and when trying to do a release (locally) under bloom.  Both
> systems
> >> >>>> are
> >> >>>> Indigo.
> >> >>>>
> >> >>>> Has anybody else seen this problem?
> >> >>>>
> >> >>>>
> >> >>>> See <http://jenkins.ros.org/job/prerelease-indigo-descartes/
> >> >>>> ARCH_PARAM=amd64,UBUNTU_PARAM=trusty,label=prerelease/1/>
> >> >>>>
> >> >>>> ------------------------------------------
> >> >>>>
> >> >>>> Executing command 'rosdep init'
> >> >>>> Wrote /etc/ros/rosdep/sources.list.d/20-default.list
> >> >>>> Recommended: please run
> >> >>>>
> >> >>>>         rosdep update
> >> >>>>
> >> >>>> Executing command 'rosdep update'
> >> >>>> Warning: running 'rosdep update' as root is not recommended.
> >> >>>>   You should run 'sudo rosdep fix-permissions' and invoke 'rosdep
> >> >> update'
> >> >>>> again without sudo.
> >> >>>> ERROR: error loading sources list:
> >> >>>>         HTTP Error 404: Not Found (
> >> >> https://raw.githubusercontent.com/ros/
> >> >>>> rosdistro/master/groovy/distribution.yaml)
> >> >>>> reading in sources list data from /etc/ros/rosdep/sources.list.d
> >> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
> >> >>>> rosdep/osx-homebrew.yaml
> >> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
> >> >>>> rosdep/base.yaml
> >> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
> >> >>>> rosdep/python.yaml
> >> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
> >> >>>> rosdep/ruby.yaml
> >> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
> >> >>>> releases/fuerte.yaml
> >> >>>> Query rosdistro index https://raw.githubusercontent.
> >> >>>> com/ros/rosdistro/master/index.yaml
> >> >>>> Add distro "groovy"
> >> >>>> /!\  Failed to execute command '['rosdep', 'update']' with return
> >> >>>> code 1
> >> >>>> Failed to execute command '['rosdep', 'update']' with return code 1
> >> >>>> I: Copying back the cached apt archive contents
> >> >>>> I: unmounting /var/cache/pbuilder/ccache filesystem
> >> >>>> I: unmounting
> >> >>>> <http://jenkins.ros.org/job/prerelease-indigo-descartes/
> >> >>>> ARCH_PARAM=amd64,UBUNTU_PARAM=trusty,label=prerelease/ws/>
> filesystem
> >> >>>> I: unmounting /home/rosbuild filesystem
> >> >>>> I: unmounting dev/pts filesystem
> >> >>>> I: unmounting proc filesystem
> >> >>>> I: cleaning the build env
> >> >>>> I: removing directory /var/cache/pbuilder/build//31574 and its
> >> >>>> subdirectories
> >> >>>> Build step 'Execute shell' marked build as failure
> >> >>>> Recording test results
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> ros-release mailing list
> >> >>>> ros-release at lists.ros.org
> >> >>>> http://lists.ros.org/mailman/listinfo/ros-release
> >> >>>>
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> ------------------------------------------------------------------------
> >> >>>
> >> >>> _______________________________________________
> >> >>> ros-release mailing list
> >> >>> ros-release at lists.ros.org
> >> >>> http://lists.ros.org/mailman/listinfo/ros-release
> >> >> _______________________________________________
> >> >> ros-release mailing list
> >> >> ros-release at lists.ros.org
> >> >> http://lists.ros.org/mailman/listinfo/ros-release
> >> >>
> >> >
> >> >
> >> >
> >>
> >
> >
> >
> > --
> > William Woodall
> > ROS Development Team
> > william at osrfoundation.org
> > http://wjwwood.io/
> >
> > _______________________________________________
> > ros-release mailing list
> > ros-release at lists.ros.org
> > http://lists.ros.org/mailman/listinfo/ros-release
> >
> _______________________________________________
> ros-release mailing list
> ros-release at lists.ros.org
> http://lists.ros.org/mailman/listinfo/ros-release
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ros.org/pipermail/ros-release/attachments/20150401/7046d97d/attachment-0001.html>