REP: 134 Title: catkin_make_isolated for non-standard catkin packages Author: William Woodall, Dirk Thomas, Thibault Kruse Status: Active Type: Informational Content-Type: text/x-rst Created: 15-Dec-2012 Post-History: Abstract ======== This REP describes the new command `catkin_make_isolated` to be delivered with catkin [1]_, to allow isolated builds of packages and in that process also build of plain cmake projects. Specification ============= The REP introduces a new tag to use in the exports section of the package.xml:: catkin(default)|cmake Catkin commands will use the value of this tag to determine how to treat a package. Currently only two values are defined, "catkin" and "cmake", but further build types may be added later (possible examples could be make, autotools, rosbuild). "catkin" is the default value for this property that will be assumed in the absence of the tag. Having a package with a different build_type than "catkin" in the catkin workspace will make catkin_make fail (or the equivalent call to cmake using the toplevel CMakeLists.txt). Instead, the user will have to invoke a new command `catkin_build_isolated`, which performs the build of all workspace projects in a different way catkin usually does. Each catkin package will be build using isolated cmake and make invocations for each package. This is necessary as non-standard catkin packages do not act compliantly within the configuration process (they do not generate catkin-compliant configuration files during configuration so that other packages could depend on them after configuration). `catkin_make_isolated` by default uses a build and devel space named differently than the default catkin build and devel spaces: `build_isolated` instead of `build`, and `devel_isolated` instead of `devel`. `catkin_make_isolated` uses the information from all catkin package manifests in the workspace to build a valid ordering of build steps. Then for each package, a build and devel subfolder are created in the build and devel space. For each package, `catkin_make_isolated` then invokes cmake and make in an environment that includes all devel spaces of dependencies of a given package. Later invocations just invoke make. `catkin_make_isolated` will not create a CMakeLists.txt in the source space, in contrast to catkin_make, as this is only needed for the non-isolated build process. Similarly, there is no option to invoke configuration and build of multiple packages in a workspace with single commands other than by invoking `catkin_make_isolated`, in contrast to catkin_make which is practically equivalent to invoking cmake and make manually. Any other build system than catkin to be supported has to satisfy the following constraints: * building packages with the build system must produce in the devel space artifacts such that using the devel environment the executables can be invoked using rosrun, and other packages can be build on top relying on standard cmake dependency resolution (find_package) * installing packages installs to the installspace, that is passed to `catkin_make_isolated` using the `--install-space` option. Additionally, build systems should strive to support the following: * Script files should not be duplicated in the devel space, but relay files be used that point to the source files * The source file tree should not be changed by build invocations, the build space should be used for build artifacts that belong into the develspace. catkin_make_isolated options ---------------------------- `catkin_make_isolated` has the same options as catkin_make where those can be applied. catkin_make_isolated also requires extra options:: --devel DEVEL, --devel-space DEVEL Sets the target parent devel space (default "devel_isolated") --merge Build each catkin package into a common devel space. --install-space INSTALL_SPACE Sets the target install space (default "install_isolated") --install Causes each catkin package to be installed. With `catkin_make` those options are not provided as they are passed as CMAKE variables / targets instead. In the isolated build, that is not possible because the actual cmake variables have to be set differently for each package, hence these options only need to exist in `catkin_make_isolated`. Options that are currently shared with catkin_make:: -j [JOBS], --jobs [JOBS] --force-cmake --no-color --pkg PKGNAME [PKGNAME ...] -q, --quiet --cmake-args [CMAKE_ARGS [CMAKE_ARGS ...]] --make-args [MAKE_ARGS [MAKE_ARGS ...]] As catkin evolves, such helper options may change, this REP is not an attempt to capture all options for all time. By default `catkin_make_isolated` will create a build folder for package foo at:: workspace/build_isolated/foo/ whereas catkin_make only creates one buildspace for all packages at: workspace/buildspace By default `catkin_make_isolated` will create a build folder for package foo at:: workspace/devel_isolated/foo/ whereas `catkin_make` only creates one develspace for all packages at:: workspace/devel The develspace folder:: workspace/devel_isolated will contain `setup.*sh` files to use to create a runtime environment. For both standard as well as non-standard packages, catkin_make_isolated generates environment files (`env.sh`, ´setup.sh´) in the packages develspace. The isolated buildspace of a package can be used to (re-)build the package using the make command, provided the prerequisite packages are in the CMAKE_PREFIX_PATH. The `env.sh` file in the isolated develspace of the package can be used to run cmake with a suitable environment. A `--merge` option allows to use a single develspace for all packages, while the buildspaces remain isolated. Passing `--install` will invoke `make install` for normal catkin packages in each isolated buildspace, and in non-standard packages, the behavior will be defined by each alternative build system support. CMake build system ------------------ The initial support for non-standard catkin packages using the plain cmake build system is implemented by invoking make in the isolated buildspace of a package. After that the cmake project is installed to the develspace to comply with the develspace concept. Motivation ========== Catkin defines a catkin package to be a folder containing a package.xml file satisfying the catkin syntax and a CMakeList.txt. Catkin provides cmake macros and variables, in particular the `catkin_package()` macro, which performs necessary steps for a catkin package to be build successfully in the catkin build process. This relates to the configure process of a package also generating suitable configuration files for dependent packages, as well as placing suitable build artifacts in the so called devel space. However it is technically feasible to create a catkin package that does not use the provided catkin cmake macros or variables. The CMakeLists.txt then defines a build process according to e.g. plain cmake rules. Such a project which is not compliant with catkin breaks the default catkin build process (mainly due to the lack of catkin-generated cmake configuration file, and due to the absence of valid devel space files). Several libraries in the ROS ecosystem have a complex CMakeLists.txt setup, and some maintainers may be reluctant to maintain a catkin-compliant CMakeLists.txt. This introduces the notion of a non-standard catkin package. Such a package has a package.xml file, but does not follow any convention about the CMakeLists.txt. This REP defines basic support for an alternative build process that can also include such non-standard catkin packages. This measure is a workaround and not currently intended for the majority of catkin users. Current caveats are very low speed, and a folder layout that is different from the layout generated by `catkin_make`. The develspace may also contain duplicate artifacts from the source space for non-standard catkin packages, whereas for standard catkin packages duplication is avoided. The target audience are users who want to build core ros packages from source, rather than using pre-packaged distributions. The first packages to be included as non-standard catkin packages like this are libraries like flann, kdl, opencv, pcl. Rationale ========= The build process catkin establishes bulk-processes all catkin projects within the same workspace folder as if they were part of a single cmake project. This design was chosen to speed up the configuration and build processes, and to allow for a single build space to allow easy cross-compilation. Rules exist for developers to follow to reduce the risk of undesired cmake namespace collisions between catkin packages (Some namespace collisions are desired to detect conflicts early that have to be resolved anyway). Catkin also defines a devel space which acts like an install space in many respects. By complying to catkin rules for CMakeLists.txt, developers ensure that a catkin package can be deployed to a devel space properly. The devel space artifacts are generated by invoking the "make" command, with the default target (no 'install' necessary). A non-standard catkin package does not use catkin macros (in particular not catkin_package, which is essential to the catkin workspace), so during it's configuration process it does not generate files for dependent packages to depend on it. This breaks the bulk-processing approach of the default catkin build process. Instead, non-standard catkin projects can be supported by dropping the bulk processing approach for an isolated build approach. This configures and builds each package before configuring and building dependent packages. Alternatives / Concerns ======================= This section reasons about alternative design choices and why they were rejected. catkin_make command option -------------------------- An option would have been to extend catkin_make with an option like --isolated. However the commandline syntax between catkin_make and catkin_make_isolated varies somewhat, as well as the build result. However, currently discussed future changes to catkin_make might change this (consider `catkin_cmake` command). catkin_make_isolated with parallel builds ----------------------------------------- The initial prototype of `catkin_make_isolated` invokes cmake and make for projects in sequence. Technically, it would be possible to paralleliize this process for better performance when any 2 packages do not depend on each other. That's a technically valid alternative that may be implemented in the future. https://github.com/ros/catkin/issues/330 catkin_make_isolated isolated environments ------------------------------------------ The initial prototype of `catkin_make_isolated` provides an `env.sh` in the package's subfolder in the develspace to generate an environment such that dependencies can be found by cmake. With the prototype, the environment may also contain entries locating packages that were not listed as dependencies for a package, which may cause confusion and mask missing build information. It is thinkable to instead provide a way to set up a build environment that strictly only has entries for dependendencies of a package from the workspace. That's a technically valid alternative that may be implemented in the future. https://github.com/ros/catkin/issues/367 Reference Implementation ======================== A catkin_make_isolated command has been released with ROS Groovy. References ========== .. [1] Catkin build system documentation (http://ros.org/wiki/catkin) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: