[ros-users] REP 134 catkin_make_isolated

Thu Mar 14 20:14:28 UTC 2013

Hi all,

attached is REP134, which describes and justifies the 
catkin_make_isolated scripts that was introduced with Groovy.
It can also be read on github:
https://github.com/tkruse/rep/blob/repisolated/rep-0134.rst

This is an informational REP, so no voting is required, though comments 
are welcome.

catkin_make_isolated was necessary to allow building non-standard cmake 
projects within a catkin workspace. Such a project has a CMakeLists.txt, 
but does not use catkin macros. It still requires a package.xml to be 
build using catkin_make_isolated. catkin_make_isolated is currently only 
required to build workspaces containing pcl, opencv or flann from 
source, most ROS users should not require it. However, the isolation 
allows to detect certain errors in the build files that catkin by 
default misses, and can help everyone check their build files.

The implementation was done by William Woodall and Dirk Thomas, I just 
created the REP accordingly.

regards,
   Thibault
-------------- next part --------------
REP: 134
Title: catkin_make_isolated for non-standard catkin packages
Author: William Woodall, Dirk Thomas, Thibault Kruse
Status: Active
Type: Informational
Content-Type: text/x-rst
Created: 15-Dec-2012
Post-History:

Abstract
========

This REP describes the new command `catkin_make_isolated` to be
delivered with catkin [1]_, to allow isolated builds of packages and
in that process also build of plain cmake projects.

Specification
=============

The REP introduces a new tag to use in the exports section
of the package.xml::

  <build_type>catkin(default)|cmake</build_type>

Catkin commands will use the value of this tag to determine how to treat a
package.  Currently only two values are defined, "catkin" and "cmake",
but further build types may be added later (possible examples could be
make, autotools, rosbuild). "catkin" is the default value for this
property that will be assumed in the absence of the tag.

Having a package with a different build_type than "catkin" in the
catkin workspace will make catkin_make fail (or the equivalent call to
cmake using the toplevel CMakeLists.txt). Instead, the user will have
to invoke a new command `catkin_build_isolated`, which performs the
build of all workspace projects in a different way catkin usually
does. Each catkin package will be build using isolated cmake and make
invocations for each package. This is necessary as non-standard catkin
packages do not act compliantly within the configuration process (they
do not generate catkin-compliant configuration files during
configuration so that other packages could depend on them after
configuration).

`catkin_make_isolated` by default uses a build and devel space named
differently than the default catkin build and devel spaces:
`build_isolated` instead of `build`, and `devel_isolated` instead of
`devel`.  `catkin_make_isolated` uses the information from all catkin
package manifests in the workspace to build a valid ordering of build
steps.  Then for each package, a build and devel subfolder are created
in the build and devel space. For each package, `catkin_make_isolated`
then invokes cmake and make in an environment that includes all devel
spaces of dependencies of a given package. Later invocations just
invoke make.

`catkin_make_isolated` will not create a CMakeLists.txt in the source
space, in contrast to catkin_make, as this is only needed for the
non-isolated build process. Similarly, there is no option to invoke
configuration and build of multiple packages in a workspace with
single commands other than by invoking `catkin_make_isolated`, in
contrast to catkin_make which is practically equivalent to invoking
cmake and make manually.

Any other build system than catkin to be supported has to satisfy the
following constraints:

 * building packages with the build system must produce in the devel
   space artifacts such that using the devel environment the
   executables can be invoked using rosrun, and other packages can be
   build on top relying on standard cmake dependency resolution
   (find_package) * installing packages installs to the installspace,
   that is passed to `catkin_make_isolated` using the
   `--install-space` option.

Additionally, build systems should strive to support the following:

 * Script files should not be duplicated in the devel space, but relay
   files be used that point to the source files
 * The source file tree should not be changed by build invocations,
   the build space should be used for build artifacts that
   belong into the develspace.

catkin_make_isolated options
----------------------------

`catkin_make_isolated` has the same options as catkin_make where those
can be applied. catkin_make_isolated also requires extra options::

  --devel DEVEL, --devel-space DEVEL
                        Sets the target parent devel space (default "devel_isolated")
  --merge               Build each catkin package into a common devel space.
  --install-space INSTALL_SPACE
                        Sets the target install space (default
                        "install_isolated")
  --install             Causes each catkin package to be installed.

With `catkin_make` those options are not provided as they are passed
as CMAKE variables / targets instead. In the isolated build, that is
not possible because the actual cmake variables have to be set
differently for each package, hence these options only need to exist
in `catkin_make_isolated`.

Options that are currently shared with catkin_make::

  -j [JOBS], --jobs [JOBS]
  --force-cmake
  --no-color
  --pkg PKGNAME [PKGNAME ...]
  -q, --quiet
  --cmake-args [CMAKE_ARGS [CMAKE_ARGS ...]]
  --make-args [MAKE_ARGS [MAKE_ARGS ...]]

As catkin evolves, such helper options may change, this REP is not an
attempt to capture all options for all time.

By default `catkin_make_isolated` will create a build folder for package foo at::

  workspace/build_isolated/foo/

whereas catkin_make only creates one buildspace for all packages at:

  workspace/buildspace

By default `catkin_make_isolated` will
create a build folder for package foo at::

  workspace/devel_isolated/foo/

whereas `catkin_make` only creates one develspace for all packages at::

  workspace/devel

The develspace folder::

  workspace/devel_isolated

will contain `setup.*sh` files to use to create a runtime environment.

For both standard as well as non-standard packages,
catkin_make_isolated generates environment files (`env.sh`,
´setup.sh´) in the packages develspace. The isolated buildspace of a
package can be used to (re-)build the package using the make command,
provided the prerequisite packages are in the CMAKE_PREFIX_PATH. The
`env.sh` file in the isolated develspace of the package can be used to
run cmake with a suitable environment.

A `--merge` option allows to use a single develspace for all
packages, while the buildspaces remain isolated.

Passing `--install` will invoke `make install` for normal catkin
packages in each isolated buildspace, and in non-standard packages, the
behavior will be defined by each alternative build system support.

CMake build system
------------------

The initial support for non-standard catkin packages using the plain cmake
build system is implemented by invoking make in the isolated
buildspace of a package. After that the cmake project is installed to
the develspace to comply with the develspace concept.

Motivation
==========

Catkin defines a catkin package to be a folder containing a
package.xml file satisfying the catkin syntax and a CMakeList.txt.

Catkin provides cmake macros and variables, in particular the
`catkin_package()` macro, which performs necessary steps for a catkin
package to be build successfully in the catkin build process.  This
relates to the configure process of a package also generating suitable
configuration files for dependent packages, as well as placing
suitable build artifacts in the so called devel space.

However it is technically feasible to create a catkin package that
does not use the provided catkin cmake macros or variables. The
CMakeLists.txt then defines a build process according to e.g. plain
cmake rules. Such a project which is not compliant with catkin breaks
the default catkin build process (mainly due to the lack of
catkin-generated cmake configuration file, and due to the absence of
valid devel space files).

Several libraries in the ROS ecosystem have a complex CMakeLists.txt
setup, and some maintainers may be reluctant to maintain a
catkin-compliant CMakeLists.txt.

This introduces the notion of a non-standard catkin package. Such a
package has a package.xml file, but does not follow any convention
about the CMakeLists.txt. This REP defines basic support for an
alternative build process that can also include such non-standard
catkin packages. This measure is a workaround and not currently
intended for the majority of catkin users. Current caveats are very
low speed, and a folder layout that is different from the layout
generated by `catkin_make`. The develspace may also contain duplicate
artifacts from the source space for non-standard catkin packages,
whereas for standard catkin packages duplication is avoided. The
target audience are users who want to build core ros packages from
source, rather than using pre-packaged distributions.

The first packages to be included as non-standard catkin packages like this
are libraries like flann, kdl, opencv, pcl.

Rationale
=========

The build process catkin establishes bulk-processes all catkin
projects within the same workspace folder as if they were part of a
single cmake project. This design was chosen to speed up the
configuration and build processes, and to allow for a single build
space to allow easy cross-compilation. Rules exist for developers to
follow to reduce the risk of undesired cmake namespace collisions
between catkin packages (Some namespace collisions are desired to
detect conflicts early that have to be resolved anyway).

Catkin also defines a devel space which acts like an install space in
many respects. By complying to catkin rules for CMakeLists.txt,
developers ensure that a catkin package can be deployed to a devel
space properly. The devel space artifacts are generated by invoking
the "make" command, with the default target (no 'install' necessary).

A non-standard catkin package does not use catkin macros (in
particular not catkin_package, which is essential to the catkin
workspace), so during it's configuration process it does not generate
files for dependent packages to depend on it. This breaks the
bulk-processing approach of the default catkin build process.

Instead, non-standard catkin projects can be supported by dropping the bulk
processing approach for an isolated build approach. This configures
and builds each package before configuring and building dependent
packages.

Alternatives / Concerns
=======================

This section reasons about alternative design choices and why they
were rejected.

catkin_make command option
--------------------------

An option would have been to extend catkin_make with an option like
--isolated. However the commandline syntax between catkin_make and
catkin_make_isolated varies somewhat, as well as the build result.
However, currently discussed future changes to catkin_make might
change this (consider `catkin_cmake` command).

catkin_make_isolated with parallel builds
-----------------------------------------

The initial prototype of `catkin_make_isolated` invokes cmake and make
for projects in sequence. Technically, it would be possible to
paralleliize this process for better performance when any 2 packages
do not depend on each other. That's a technically valid alternative
that may be implemented in the future.

https://github.com/ros/catkin/issues/330

catkin_make_isolated isolated environments
------------------------------------------

The initial prototype of `catkin_make_isolated` provides an `env.sh`
in the package's subfolder in the develspace to generate an
environment such that dependencies can be found by cmake. With the
prototype, the environment may also contain entries locating packages
that were not listed as dependencies for a package, which may cause
confusion and mask missing build information. It is thinkable to
instead provide a way to set up a build environment that strictly only
has entries for dependendencies of a package from the workspace.

That's a technically valid alternative that may be implemented in the future.

https://github.com/ros/catkin/issues/367

Reference Implementation
========================

A catkin_make_isolated command has been released with ROS Groovy.

References
==========

.. [1] Catkin build system documentation
   (http://ros.org/wiki/catkin)

Copyright
=========

This document has been placed in the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: