Hi Piyush,

Unfortunately I believe this is an inherent issue with debian repositories managed by reprepro. (the tool we use for managing the repository) Basically when new packages get uploaded it regenerates the metadata and in the process there's a race condition for fetching the files being regenerated.

Note this is a much shorter period than the Hash sum mismatch that we already detect and retry. On the new buildfarm we can catch this one as well and retry.

Tully

On Wed, Apr 1, 2015 at 3:54 PM, Piyush via ros-release <ros-release@lists.ros.org> wrote:
Hey William,

Since we're on the subject, I've been seeing a number of failures
because of the following message, just in case you can do something
about it. I believe this error is independent of recent github
outages:

running command ['reprepro', '-v', '-b', '/var/www/repos/building',
'-S', 'misc', 'include', u'saucy',
'/tmp/upload/ros-indigo-bwi-gazebo-entities_saucy_i386/ros-indigo-bwi-gazebo-entities_0.3.2-0saucy-20150401-2152-+0000_i386.changes']
Exporting indices...
Removing /tmp/upload/ros-indigo-bwi-gazebo-entities_saucy_i386
+ sudo apt-get update -c
<http://jenkins.ros.org/job/ros-indigo-bwi-gazebo-entities_binarydeb_saucy_i386/ws/apt.conf>
-o Apt::Architecture=i386
Get:1 http://repos.ros.org saucy Release.gpg [198 B]
Get:2 http://repos.ros.org saucy Release [3,779 B]
Err http://repos.ros.org saucy Release

Get:3 http://us.archive.ubuntu.com saucy Release.gpg [933 B]
Get:4 http://us.archive.ubuntu.com saucy-updates Release.gpg [933 B]
Hit http://us.archive.ubuntu.com saucy Release
Ign http://us.archive.ubuntu.com saucy Release
Hit http://us.archive.ubuntu.com saucy-updates Release
Ign http://us.archive.ubuntu.com saucy-updates Release
Ign http://us.archive.ubuntu.com saucy/main Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy/restricted Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy/universe Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy/multiverse Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy/main i386 Packages/DiffIndex
Ign http://us.archive.ubuntu.com saucy/restricted i386 Packages/DiffIndex
Ign http://us.archive.ubuntu.com saucy/universe i386 Packages/DiffIndex
Ign http://us.archive.ubuntu.com saucy/multiverse i386 Packages/DiffIndex
Hit http://us.archive.ubuntu.com saucy/main TranslationIndex
Hit http://us.archive.ubuntu.com saucy/multiverse TranslationIndex
Hit http://us.archive.ubuntu.com saucy/restricted TranslationIndex
Hit http://us.archive.ubuntu.com saucy/universe TranslationIndex
Ign http://us.archive.ubuntu.com saucy-updates/main Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy-updates/restricted Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy-updates/universe Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy-updates/multiverse Sources/DiffIndex
Ign http://us.archive.ubuntu.com saucy-updates/main i386 Packages/DiffIndex
Ign http://us.archive.ubuntu.com saucy-updates/restricted i386
Packages/DiffIndex
Ign http://us.archive.ubuntu.com saucy-updates/universe i386 Packages/DiffIndex
Ign http://us.archive.ubuntu.com saucy-updates/multiverse i386
Packages/DiffIndex
Hit http://us.archive.ubuntu.com saucy-updates/main TranslationIndex
Hit http://us.archive.ubuntu.com saucy-updates/multiverse TranslationIndex
Hit http://us.archive.ubuntu.com saucy-updates/restricted TranslationIndex
Hit http://us.archive.ubuntu.com saucy-updates/universe TranslationIndex
Hit http://us.archive.ubuntu.com saucy/main Sources
Hit http://us.archive.ubuntu.com saucy/restricted Sources
Hit http://us.archive.ubuntu.com saucy/universe Sources
Hit http://us.archive.ubuntu.com saucy/multiverse Sources
Hit http://us.archive.ubuntu.com saucy/main i386 Packages
Hit http://us.archive.ubuntu.com saucy/restricted i386 Packages
Hit http://us.archive.ubuntu.com saucy/universe i386 Packages
Hit http://us.archive.ubuntu.com saucy/multiverse i386 Packages
Hit http://us.archive.ubuntu.com saucy/main Translation-en
Hit http://us.archive.ubuntu.com saucy/multiverse Translation-en
Hit http://us.archive.ubuntu.com saucy/restricted Translation-en
Hit http://us.archive.ubuntu.com saucy/universe Translation-en
Hit http://us.archive.ubuntu.com saucy-updates/main Sources
Hit http://us.archive.ubuntu.com saucy-updates/restricted Sources
Hit http://us.archive.ubuntu.com saucy-updates/universe Sources
Hit http://us.archive.ubuntu.com saucy-updates/multiverse Sources
Hit http://us.archive.ubuntu.com saucy-updates/main i386 Packages
Hit http://us.archive.ubuntu.com saucy-updates/restricted i386 Packages
Hit http://us.archive.ubuntu.com saucy-updates/universe i386 Packages
Hit http://us.archive.ubuntu.com saucy-updates/multiverse i386 Packages
Hit http://us.archive.ubuntu.com saucy-updates/main Translation-en
Hit http://us.archive.ubuntu.com saucy-updates/multiverse Translation-en
Hit http://us.archive.ubuntu.com saucy-updates/restricted Translation-en
Hit http://us.archive.ubuntu.com saucy-updates/universe Translation-en
Fetched 5,843 B in 3s (1,766 B/s)
Reading package lists...
W: A error occurred during the signature verification. The repository
is not updated and the previous index files will be used. GPG error:
http://repos.ros.org saucy Release: The following signatures couldn't
be verified because the public key is not available: NO_PUBKEY
5523BAEEB01FA116

W: GPG error: http://us.archive.ubuntu.com saucy Release: The
following signatures couldn't be verified because the public key is
not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: GPG error: http://us.archive.ubuntu.com saucy-updates Release: The
following signatures couldn't be verified because the public key is
not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: Failed to fetch http://repos.ros.org/repos/building/dists/saucy/Release

Piyush

On Wed, Apr 1, 2015 at 3:41 PM, William Woodall via ros-release
<ros-release@lists.ros.org> wrote:
> I think it would be better to just tell the user what's going on. If you can
> reliably tell when you should retry, then you can also just tell the user to
> try it again. For the build farm, retrying immediately usually never works
> because its takes time for a failure to get resolved upstream, so we retry
> several times with a reasonable cool off period and that seems to help a
> bit. For users though I think this kind of thing is not particularly worth
> it.
>
> All of that being said if you come up with a way to communicate to the user
> what's going on and also is more robust to intermittent failures, that's ok
> to propose as a pull request. However, I'll point out that most tools don't
> retry and I think there is a good rational behind that to reduce the amount
> of magic between the user and the problems when they occur. For example,
> programs that don't automatically retry git, apt-get, Homebrew, curl
> (without an option), wget (without an option), chrome, safari, firefox,
> etc... Obviously "just because others don't do it" isn't sufficient reason
> but it is a good indicator.
>
> On Wed, Apr 1, 2015 at 1:17 PM, G.A. vd. Hoorn - 3ME
> <g.a.vanderhoorn@tudelft.nl> wrote:
>>
>> On 1-4-2015 22:15, William Woodall wrote:
>> > The new build farm will retry on connection failures but it doesn't
>> > really
>> > make sense to retry on 404 in my opinion. It's a really _bad_ failure
>> > mode
>> > to temporarily replace valid files with 404 responses. But I imagine
>> > GitHub
>> > was being stressed in ways that aren't considered normal operation.
>>
>> I agree about the 404 not being the best status code to return, but I
>> was more asking about whether rosdep itself should try a bit harder.
>>
>> Normal desktop users also see this, and quite often assume there is
>> actually something really wrong, while a simple retry (a la wget) would
>> let things continue.
>>
>>
>> > On Wed, Apr 1, 2015 at 1:13 PM, G.A. vd. Hoorn - 3ME via ros-release <
>> > ros-release@lists.ros.org> wrote:
>> >
>> >> On 1-4-2015 22:08, William Woodall via ros-release wrote:
>> >>> That's a problem with GitHub, I think an artifact of the recent DDoS
>> >> attack
>> >>> against them. The file that was reportedly 404 above is now available.
>> >>> Unfortunately there's not much we can do about this sort of failure.
>> >> Would it be an idea to increase the nr of attempts?
>> >>
>> >> It seems rosdep gives up after a single 404. Given that github
>> >> experiences a lot of transient failures (we also see quite some
>> >> questions like this on ROS Answers), a retry or 2 might go a long way?
>> >>
>> >>
>> >>> On Wed, Apr 1, 2015 at 12:36 PM, Shaun Edwards via ros-release <
>> >>> ros-release@lists.ros.org> wrote:
>> >>>
>> >>>> All,
>> >>>>
>> >>>> I have had two builds/releases failure for the same issue associated
>> >> with
>> >>>> rosdep (see snippet below).  rosdep has some errors, but in the past
>> >> this
>> >>>> has not caused a failure.  I see this same error on the pre-release
>> >> server
>> >>>> and when trying to do a release (locally) under bloom.  Both systems
>> >>>> are
>> >>>> Indigo.
>> >>>>
>> >>>> Has anybody else seen this problem?
>> >>>>
>> >>>>
>> >>>> See <http://jenkins.ros.org/job/prerelease-indigo-descartes/
>> >>>> ARCH_PARAM=amd64,UBUNTU_PARAM=trusty,label=prerelease/1/>
>> >>>>
>> >>>> ------------------------------------------
>> >>>>
>> >>>> Executing command 'rosdep init'
>> >>>> Wrote /etc/ros/rosdep/sources.list.d/20-default.list
>> >>>> Recommended: please run
>> >>>>
>> >>>>         rosdep update
>> >>>>
>> >>>> Executing command 'rosdep update'
>> >>>> Warning: running 'rosdep update' as root is not recommended.
>> >>>>   You should run 'sudo rosdep fix-permissions' and invoke 'rosdep
>> >> update'
>> >>>> again without sudo.
>> >>>> ERROR: error loading sources list:
>> >>>>         HTTP Error 404: Not Found (
>> >> https://raw.githubusercontent.com/ros/
>> >>>> rosdistro/master/groovy/distribution.yaml)
>> >>>> reading in sources list data from /etc/ros/rosdep/sources.list.d
>> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
>> >>>> rosdep/osx-homebrew.yaml
>> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
>> >>>> rosdep/base.yaml
>> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
>> >>>> rosdep/python.yaml
>> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
>> >>>> rosdep/ruby.yaml
>> >>>> Hit https://raw.githubusercontent.com/ros/rosdistro/master/
>> >>>> releases/fuerte.yaml
>> >>>> Query rosdistro index https://raw.githubusercontent.
>> >>>> com/ros/rosdistro/master/index.yaml
>> >>>> Add distro "groovy"
>> >>>> /!\  Failed to execute command '['rosdep', 'update']' with return
>> >>>> code 1
>> >>>> Failed to execute command '['rosdep', 'update']' with return code 1
>> >>>> I: Copying back the cached apt archive contents
>> >>>> I: unmounting /var/cache/pbuilder/ccache filesystem
>> >>>> I: unmounting
>> >>>> <http://jenkins.ros.org/job/prerelease-indigo-descartes/
>> >>>> ARCH_PARAM=amd64,UBUNTU_PARAM=trusty,label=prerelease/ws/> filesystem
>> >>>> I: unmounting /home/rosbuild filesystem
>> >>>> I: unmounting dev/pts filesystem
>> >>>> I: unmounting proc filesystem
>> >>>> I: cleaning the build env
>> >>>> I: removing directory /var/cache/pbuilder/build//31574 and its
>> >>>> subdirectories
>> >>>> Build step 'Execute shell' marked build as failure
>> >>>> Recording test results
>> >>>>
>> >>>> _______________________________________________
>> >>>> ros-release mailing list
>> >>>> ros-release@lists.ros.org
>> >>>> http://lists.ros.org/mailman/listinfo/ros-release
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> ------------------------------------------------------------------------
>> >>>
>> >>> _______________________________________________
>> >>> ros-release mailing list
>> >>> ros-release@lists.ros.org
>> >>> http://lists.ros.org/mailman/listinfo/ros-release
>> >> _______________________________________________
>> >> ros-release mailing list
>> >> ros-release@lists.ros.org
>> >> http://lists.ros.org/mailman/listinfo/ros-release
>> >>
>> >
>> >
>> >
>>
>
>
>
> --
> William Woodall
> ROS Development Team
> william@osrfoundation.org
> http://wjwwood.io/
>
> _______________________________________________
> ros-release mailing list
> ros-release@lists.ros.org
> http://lists.ros.org/mailman/listinfo/ros-release
>
_______________________________________________
ros-release mailing list
ros-release@lists.ros.org
http://lists.ros.org/mailman/listinfo/ros-release