Another point about testing vs public or shadow-fixed is that if you haven't released yet, then your new version of your package will never deployed on top of the exact debs in public, because you'll release your new version into shadow-fixed (with all other staged releases) and then they are all moved to public all at once. Therefore, a prerelease which tests on top of public is a configuration which never exists in the wild (unless at the time of your prerelease the binaries in shadow-fixed exactly match what is in public, i.e. right after a sync). So I'd argue testing on top of public doesn't really make sense and testing on top of shadow-fixed better reflects what is actually going to happen after you release.

Also as to the purpose of a prerelease is to answer the question is my release going to work when we sync to pubic next time, which I can break down into three questions:

1. Did I break my thing?
2. Did I break someone that uses me?
3. Did someone else break me?

In my opinion, I think 3 is just as valid to be testing for as 1 or 2 is.

Cheers,

On Mon, Nov 16, 2015 at 12:01 PM, Austin Hendrix via ros-release <ros-release@lists.ros.org> wrote:
I see the PR build as separate from the prerelease - prerelease tests compatibility with the rest of the ecosystem, whereas the PR build would be a deb build to validate that the package has been bloom'ed correctly, and that it builds debs.

Paired releases is definitely worth considering; esp since each repo that is released is usually built as many separate debs. If we wanted to test that all debs within a released unit build correctly, we'd have to set up some kind of temporary deb repo for use during the test process. This is probably the same regardless of whether a unit is a single repo, or several linked repos that are released simultaneously.

-Austin

On 11/16/2015 11:37 AM, Tully Foote wrote:


On Mon, Nov 16, 2015 at 11:32 AM, Austin Hendrix via ros-release <ros-release@lists.ros.org <mailto:ros-release@lists.ros.org>> wrote:

    For test builds for PRs, why not do the test (deb)build against
    the stable repository, instead of shadow-fixed or building?

    It won't catch issues caused by changes in upstream packages, but
    by testing against stable you should always have dependencies
    available (and it's stable, so the dependencies are less likely to
    be broken).


The core will be stable. But if we only test against the public release repository. If you want to do a paired release of two packages you would have to wait a full release cycle before the first one would be available to attempt to build against the previously released one.


    -Austin

    On 11/16/2015 10:39 AM, Tully Foote via ros-release wrote:

        Hi,

        Indeed we agree that getting more prereleases run is
        important. That's why we've worked very hard to make sure that
        it as well as all the jobs are reproducible.

        The web hosted solution has elements which are nice for users.
        However, it was an expensive pain to maintain. It also
        provided an inconsistent user experience, especially if
        there's a large delay due to waiting for yours or others jobs
        to run. As long as it's reproducible it's better to users
        leverage their own resources and know reliably when things
        will run.

        For preventing broken releases I do also want to integrate
        test builds into the PR validation. It will more likely be a
        test debbuild rather than a prerelease. And it has trouble if
        the buildfarm is in the middle of a large rebuild and all the
        dependencies are not immediately available.

        Tully

        On Sat, Oct 31, 2015 at 5:16 AM, Daniel Stonier
        <d.stonier@gmail.com <mailto:d.stonier@gmail.com>
        <mailto:d.stonier@gmail.com <mailto:d.stonier@gmail.com>>> wrote:


            Aye, thanks from me too.

            I'd like to +1 Jack's comments about preferring the web
        service
            that was previously available.

            1. Getting things right on one web server somewhere is always
            going to be far easier than getting it right on thousands
        of users
            systems. Even if docker does make this proposition easier,
        what we
            have seen above is it still gets awkward when the dependencies
            shift (e.g. needing a custom version of docker). I also
        ran into
            problems because of python3 interfering with my environment.

            2. Getting users to pre-release is a desirable thing. Less
            rosdistro PR's to approve, less red blips on the build
        chart, less
            latency for Tully to wrap up an official release into
        public. The
            less barriers there are for them to do this, I feel the
        better and
            easier the maintainence will be.

            Daniel.


            On 27 October 2015 at 06:12, Mani Monajjemi via ros-release
            <ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>
        <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>> wrote:

                Thanks Tully. pre-release script now works fine
        without any issue.

                - Mani

                Mani Monajjemi

                On Mon, Oct 26, 2015 at 11:08 AM, Tully Foote via
        ros-release
                <ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>
         <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>>
                wrote:

                    I've pushed ros_buildfarm 0.2.1 it should now be
        usable
                    from the debian packages.

                    Tully

                    On Mon, Oct 26, 2015 at 10:42 AM, William Woodall via
                    ros-release <ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>
                     <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>> wrote:

                        On Mon, Oct 26, 2015 at 10:09 AM, Jack O'Quin via
                        ros-release <ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>
                         <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>> wrote:



                            On Mon, Oct 26, 2015 at 11:24 AM, Daniel
        Stonier
                            via ros-release <ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>
                             <mailto:ros-release@lists.ros.org

        <mailto:ros-release@lists.ros.org>>> wrote:


                                It has been a while since I've built open
                                source debs, but this is above and
        beyond the
                                effort required to prerelease for me
        though.
                                It used to be such a fundamental part
        of the
                                process - what is the current
        thinking? Given
                                that its been out for at least three
        months,
                                are most people just guessing,
        rebuilding on
                                the farm, guessing again? Is there a
        planned
                                remedy on the horizon?


                            Basically, yes.

                            Presently, the pre-release tests take more
        effort
                            than just hoping for the best and then fixing
                            things that break.


                        Why? I've been using them for rviz and it seems to
                        work fairly well. What's holding up making them
                        useful? It is just the need to install it into a
                        virtualenv first?

                                My 2 cents - I'd really love to see this
                                working again ;) ;) ;) ;)


                        Again, what's not working? Is there an issue
        on Github
                        tracking the problem?


                                Saves alot of time for me not having
        to ping
                                pong back and forth trying to get my
                                dependencies right and I'm sure it
        makes the
                                job easier on the other end avoiding
        having so
                                many red blips on the radar so often.


                            +1 I would find it helpful, too. I much prefer
                            running the tests.

                            The pre-docker web interface was very
        convenient.
                            I think this could be, too, although it's
        annoying
                            that the Trusty version of docker is too
        old to use.


                        There's nothing to be done about that
        unfortunately.

                            --                      joq

        _______________________________________________
                            ros-release mailing list
        ros-release@lists.ros.org <mailto:ros-release@lists.ros.org>
                            <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>
        http://lists.ros.org/mailman/listinfo/ros-release




                        --                 William Woodall
                        ROS Development Team
        william@osrfoundation.org <mailto:william@osrfoundation.org>
                         <mailto:william@osrfoundation.org
        <mailto:william@osrfoundation.org>>
        http://wjwwood.io/

        _______________________________________________
                        ros-release mailing list
        ros-release@lists.ros.org <mailto:ros-release@lists.ros.org>
                         <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>
        http://lists.ros.org/mailman/listinfo/ros-release



        _______________________________________________
                    ros-release mailing list
        ros-release@lists.ros.org <mailto:ros-release@lists.ros.org>
         <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>
        http://lists.ros.org/mailman/listinfo/ros-release



        _______________________________________________
                ros-release mailing list
        ros-release@lists.ros.org <mailto:ros-release@lists.ros.org>
         <mailto:ros-release@lists.ros.org
        <mailto:ros-release@lists.ros.org>>
        http://lists.ros.org/mailman/listinfo/ros-release




            --     Phone : +82-10-5400-3296 <tel:%2B82-10-5400-3296>
        <tel:%2B82-10-5400-3296> (010-5400-3296)
            Home: http://snorriheim.dnsdojo.com/
            Yujin Inno: http://inno.yujinrobot.com/
            <http://rnd.yujinrobot.com/>




        _______________________________________________
        ros-release mailing list
         ros-release@lists.ros.org <mailto:ros-release@lists.ros.org>
        http://lists.ros.org/mailman/listinfo/ros-release


    _______________________________________________
    ros-release mailing list
    ros-release@lists.ros.org <mailto:ros-release@lists.ros.org>
    http://lists.ros.org/mailman/listinfo/ros-release



_______________________________________________
ros-release mailing list
ros-release@lists.ros.org
http://lists.ros.org/mailman/listinfo/ros-release



--
William Woodall
ROS Development Team
william@osrfoundation.org
http://wjwwood.io/