[ros-release] Build Farm Spam and Debugging

Thu Apr 7 19:42:56 UTC 2016

I apologize for using the pejorative term spam. I was using it to
refer to the large quantities of unwanted and arguably unnecessary
emails. My ideal build-farm related emails would have the following
qualities. (This is an ideal, not a list of demands.)

1) One email per problem.
  * If my code doesn't compile on any machine, it should email me
once, even if it means that the build fails for multiple machines.
  * If I haven't changed anything, I don't need another email 15
minutes later. Maybe if I haven't fixed it in N days, send another
email.
2) Clear Messages
  * One email subject was "Build failed in Jenkins:
Ksrc_dJ__map_msgs__debian_jessie__source #96". As someone who has lots
of experience breaking the build, I can kinda parse this out, but
there's lots of garbage in there. Something like "[jenkins] map_msgs
source failed to build on jade (debian) #96" reads nicer.
  * At the very bottom of the email and log is the relevant error
message, but its never easy for me to pick out what the actual problem
is on these, whether it was something I did, something OSRF changed on
the build farm, or some momentary github hiccup.
3) Relevant to me
  * I'm unsure whether this is still an issue in the new build farm,
but I very rarely care if upstream or downstream packages momentarily
break.

I'm sure none of these is easy to implement, and would require lots of
intelligence programmed in that doesn't exist yet. However, I
occasionally consider forwarding all my build farm emails to another
account which sends me summaries of them just to cut down.

Thanks for fixing the problem and for the offer of future help.

Sincerely,
David!!

On Thu, Apr 7, 2016 at 2:57 PM, Dirk Thomas via ros-release
<ros-release at lists.ros.org> wrote:
> Jackie and I came up with a fix for the Debian jobs:
> https://github.com/ros-infrastructure/ros_buildfarm/pull/281 It fixes the
> locale on Debian and afterwards is able to parse and decode the manifest
> correctly. The job in question has passed now.
>
> The emails are sent for a reason and not really "spam". They are notifying
> you about a problem with a package you. Please feel free to ask questions on
> this mailing list if a job is "bugging" you with emails and you are not sure
> how to address it. That is always better then to ignore the notifications.
>
> Thanks,
> - Dirk
>
> On Thu, Apr 7, 2016 at 10:55 AM, Dirk Thomas <dthomas at osrfoundation.org>
> wrote:
>>
>> Before drawing any conclusions please take a close look to the actual
>> error message:
>>
>> ```
>> Traceback (most recent call last):
>>   File "/tmp/ros_buildfarm/scripts/release/get_sources.py", line 34, in
>> <module>
>>     sys.exit(main())
>>   File "/tmp/ros_buildfarm/scripts/release/get_sources.py", line 30, in
>> main
>>     args.os_name, args.os_code_name, args.source_dir)
>>   File "/tmp/ros_buildfarm/ros_buildfarm/sourcedeb_job.py", line 52, in
>> get_sources
>>     pkg = parse_package(sources_dir)
>>   File "/usr/lib/python3/dist-packages/catkin_pkg/package.py", line 370,
>> in parse_package
>>     return parse_package_string(f.read(), filename, warnings=warnings)
>>   File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
>>     return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 183:
>> ordinal not in range(128)
>> ```
>>
>> Decoding a non-ASCII character when then codec is set to `ascii` can't
>> work. Since `!` is a valid ASCII character that can't be the reason. So it
>> is likely the `é`.
>>
>> What needs to change here? Well, the first important fact to note is that
>> the sourcedeb jobs for the same package pass on all Ubuntu platforms. So
>> obviously the buildfarm as well as catkin_pkg are able to handle this.
>>
>> Why is it failing for this Debian job then? Debian has just recently been
>> added and there must be something different. Looking at the console output
>> again the following warnings just a few lines above should be enough:
>>
>> ```
>> perl: warning: Setting locale failed.
>> perl: warning: Please check that your locale settings:
>> LANGUAGE = (unset),
>> LC_ALL = (unset),
>> LANG = "en_US.UTF-8"
>>     are supported and installed on your system.
>> perl: warning: Falling back to the standard locale ("C").
>> perl: warning: Setting locale failed.
>> perl: warning: Please check that your locale settings:
>> LANGUAGE = (unset),
>> LC_ALL = (unset),
>> LANG = "en_US.UTF-8"
>>     are supported and installed on your system.
>> perl: warning: Falling back to the standard locale ("C").
>> ```
>>
>> While the job tries to use UTF-8 it fails to do so on Debian and falls
>> back to the standard locale "C". And with that locale it is simply
>> impossible to decode the special character.
>>
>> As a conclusion the recent modification to enable Debian jobs
>> (https://github.com/ros-infrastructure/ros_buildfarm/commit/5106e704ecdb2d55ac513da66ec8d699731bb859#diff-8859c2a0d6adc3dc403698f904632818)
>> needs fixing since it is not enabling the UTF-8 locale as expected.
>>
>> Cheers,
>> - Dirk
>>
>>
>> On Thu, Apr 7, 2016 at 10:07 AM, Jackie Kay via ros-release
>> <ros-release at lists.ros.org> wrote:
>>>
>>> I also find the build farm spam quite annoying. I get the same
>>> notification failure whenever anything on Kinetic breaks. :)
>>>
>>> Glancing at the package.xml for map_msgs, could it be that the accented
>>> "é" in the author field (Stéphane Magnenat) is breaking unicode parsing, not
>>> the exclamation points?
>>>
>>>
>>> https://github.com/ros-planning/navigation_msgs/blob/jade-devel/map_msgs/package.xml
>>>
>>> I will look into sanitizing the characters so that we can support package
>>> maintainers and authors with unicode characters in their names.
>>>
>>>
>>> On Thu, Apr 7, 2016 at 9:32 AM, Tully Foote via ros-release
>>> <ros-release at lists.ros.org> wrote:
>>>>
>>>> I think the exclamation points in your name are breaking the unicode
>>>> parsing on the debian target. Looking at the documentation the maintainer
>>>> field only supports ASCII characters.
>>>>
>>>> The debian maintainer field format is defined by RFC822
>>>> https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Maintainer
>>>> There's a note about problems with full stop.
>>>>
>>>> And RFC822 can be seen here
>>>> https://www.w3.org/Protocols/rfc822/3_Lexical.html
>>>>
>>>> I think it's likely that debian is more strict in it's interpretation of
>>>> the formatting. And the quickest solution will be to remove the characters
>>>> from the maintainer name. It could either be patched in the jessie specific
>>>> branch in the release repo. The simplest solution is a rerelease without the
>>>> characters.
>>>>
>>>> You could open an issue on
>>>> https://github.com/ros-infrastructure/ros_buildfarm and we could look at
>>>> sanitizing the characters in that field, but developing the correct
>>>> replacement/substitituion policy may be a challenge.
>>>>
>>>> B) Sourcedebs are triggered on a 15 minute cycle since people want them
>>>> to come out quickly. We typically will roll back any releases which fail
>>>> their sourcedeb jobs since it's rarely a platform specific issue. And if the
>>>> sourcedeb is not working we might as well pull it from the farm.
>>>>
>>>> Tully
>>>>
>>>> On Thu, Apr 7, 2016 at 6:27 AM, David Lu!! via ros-release
>>>> <ros-release at lists.ros.org> wrote:
>>>>>
>>>>> I am but a humble navigation code monkey, and am not trained in the
>>>>> ways of the build farm. Can someone help me interpret what's going on
>>>>> that's causing
>>>>> A) A specific build to fail e.g.
>>>>>
>>>>> http://build.ros.org/job/Ksrc_dJ__map_msgs__debian_jessie__source/87/console
>>>>> B) So many emails (see attached)
>>>>>
>>>>> _______________________________________________
>>>>> ros-release mailing list
>>>>> ros-release at lists.ros.org
>>>>> http://lists.ros.org/mailman/listinfo/ros-release
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ros-release mailing list
>>>> ros-release at lists.ros.org
>>>> http://lists.ros.org/mailman/listinfo/ros-release
>>>>
>>>
>>>
>>> _______________________________________________
>>> ros-release mailing list
>>> ros-release at lists.ros.org
>>> http://lists.ros.org/mailman/listinfo/ros-release
>>>
>>
>
>
> _______________________________________________
> ros-release mailing list
> ros-release at lists.ros.org
> http://lists.ros.org/mailman/listinfo/ros-release
>