[Ros-release] Thoughts on prereleases etc.

After deploying the new rosdistro implementation we have reviewed the current prereleases and devel jobs, how they work and their limitations:

Devel jobs serve the purpose of continuous integration:

* monitors a specific branch/tag in a source repository
* is triggered automatically on change
* works exactly like prereleases
* with a single repository
* without performing the depends_on builds/tests


Prerelease jobs serve the purpose of regression testing:

* triggered manually with a list of repositories
* each with a specific version numbers or “latest” (released into gbp) or from source

* stage 1:
   * all packages from the repositories are checked out in a single catkin workspace
   * if all packages have the build_type catkin
     * performs “cmake/make” and “make run_tests” on the workspace
   * else
     * performs “catkin_make_isolated --install” (without testing)
   * using two different build methods results in an inconsistent behavior

* stage 2:
   * find all package which directly and indirectly depend on the above packages
   * checks out all of them in a single workspace
   * if all packages have the build_type catkin
     * performs “cmake/make” and “make run_tests” on the workspace
   * else
     * errors out

Based on that we identified a couple of issues and considered potential changes to improve the functionality:

* it is not checkable if a pull request will break building on the farm, even if it would not touch the apt repo a later release of a lower level package would not build completely since the released
package could not be rebuild
* solution: perform a deploy test on each pull request, a deploy test does the same as the sourcedeb and binarydeb jobs on the farm without publishing the results to an apt repository
* ticketed: https://github.com/ros-infrastructure/jenkins_scripts/issues/34

* catkin_make_isolated should be able to run tests (for the packages with catkin build_type) since this is a missing feature for these and it is likely that more packages might use that approach in
the future
* ticketed: https://github.com/ros/catkin/issues/412
* after that stage 2 could also use "catkin_make_isolated" when packages with non-catkin build_type are present

* when building a set of packages missing system dependencies might not be caught if another package depends on it
* it would require a per-package apt isolation which is not feasible
* for the debian packages the predeploy test will catch that

* when building a set of packages missing ros package dependencies might not be caught if the dependent package is processed before
* this could be addressed in cmi by providing a specific environment for each package, merging the environments of all dependent packages instead of everything built before
* for the debian packages the predeploy test will catch that

* limit the number of packages considered for depends_on
   * limiting the dependency depth is not reasonable in a lot of cases
   * specifying a dependency set is also not feasible
   * but the prerelease website should indicate number of depends_on packages
     * ticketed: https://github.com/ros-infrastructure/prerelease_website/issues/11

If you have any feedback or thoughts on this please share them with us.

- Dirk