Hi All,
I'd like to take a step back from the current discussion because I feel that there are several somewhat independent issues being talked without separation.
Of the things being discussed at the moment I have identified the following 4 issues. First is how are dependency flags passed to new code. Second is what is the FHS layout. Third is how do we deal with resources. And 4th is an issue of modularity and code portability/namespacing.
In the design of the new system we've sought to keep the usability of the ROS buildsystem while leveraging as much of the standard infrastructure as possible.
For setting cflags and lflags it is standard to do this in your CMakeLists.txt You can set what flags your cmake component exports in that component. And then you use the CMake Find components which will pass those things to you automatically. This means that people no longer need to learn a new buildsystem to us ROS, they can just use standard CMake.
It also means that we no longer have to maintain the equivalent functionality, which is already built into CMake. (There are other options we looked at too like pkg-config) However, by building catkin on top of CMake we are able to provide much of the niceties of the ROS build system where you can overlay working copy on top of installed systems and you can develop and test in build space without requiring installation to test.
With the FHS layout we are again working toward becoming more standard. I believe this is much less controversial, though there's some debate about the exact layout. The one thing I would like to point out is that makes the other requirements much more complicated, especially the capability to do overlays.
We have always espoused writing libraries which are useable outside of ROS and then writing ROS wrappers. By maintaining the pure CMake find modules infrastructure, we have reached the point that ROS packages when installed can be used by non-ROS packaged systems.
As we go along this line part of what we are doing is narrowing the difference is a ROS package and what is a standard software package. Part of the restructuring of the core ROS dependencies which has gone along with this has been to cleanly separate the ROS message generation and the ROS communication infrastructure. This now allows packages to depend on the message datatypes without incuring the cost of depending on the runtime requirements.
We are also responding to a trend we have been watching where we are releasing more and more unary stacks. Back in the day when we first added the concept of stacks many people asked why we had both stacks and packages. Jack has relatively concisely stated the primary benefits of separating the interface unit and the release unit if the release unit contains multiple interface units. However if we continue the trend toward releasing unary stacks this will not be a problem. The biggest thing that we need to do for this is to make the release process easier. (Which has been addressed separately with the bloom package)
One of the primary motivators for releasing stacks which contained multiple packages was that we were getting too many packages to keep track of easily. By organizing into stacks we were able to decrease the burden of understanding the dependency tree by decreasing the number of elements in the tree. Unfortunately as we keep growing this has become a two fold issue. First we now have enough stacks that we have the same problem as we had before with the packages. And secondly, by bundling packages into the same release we also bundle their dependencies. As such we often end up with significantly more dependencies than might be optimal. (For example take one of my stacks geometry, it has dependencies on bullet, eigen, and kdl to provide convenient converter functions for them, despite only needing bullet as a core dependency. In packaging it we've chosen to maintain that grouping for more user friendly operation, but many power users complain rightlfully that it's significantly bloated. ) Thus as the pressure to not bundle dependencies is up and if we can ease the challenge of releasing the pressure to keep the number of release units down is lower. Also from experience, large stacks are a pain to maintain because the have software inside with very different timelines, and consequently you have to try to shoehorn them into the same timeline to be able to release.
The issue of ease of sharing is also a good point. We have many repositories which have just random unreleased packages which right now is relatively easy to grab and add to your system from source. We should make sure to address this use case. When actually doing releases with proper versioning having one release unit per DVCS makes the maintainer's life easier. However, it's not required, and we can make sure to keep it easy for a researcher to just have a flat repo with many "stacks" in it.
Lastly is the issue of resources (ala launch files, meshes, etc) With some of our python module development we have been successfully experimenting with using python's resource finding libraries and setuptools. And we have been discussing creating a rosfind tool(executable and native library) for finding resources. This would be the equivilant of 'rospack find' but implemented to work in the new environment, and support cross-compiling better. And as I mentioned in the FHS paragraph this is one of the complicating cases. Since we want to follow FHS we need to explicitly declare what we want to install, and what type of resource it is. Standard files, such as libraries, headers, and python source automatically go to the right place. However non standard files such as meshes, launch files etc are what need to be declared to support the install target.
I hope this helps get a bit of perspective on what we've been working toward. We are coming up on the freeze, however this discussion and features are important enought that we can also consider changing the deadline. In the last release we had a dual push first to release Fuerte and then to release Precise support. Pushing it back to be syncronized with Ubuntu's release could allow us to avoid the double push and do it at the same time. (This is much better as you only need to iterate through the whole ecosystem once. )
Tully
On Wed, Jul 25, 2012 at 4:18 AM, Lorenz Mösenlechner
<moesenle@in.tum.de> wrote:
On Wed, Jul 25, 2012 at 1:00 PM, Jack O'Quin <
jack.oquin@gmail.com> wrote:
> On Wed, Jul 25, 2012 at 4:03 AM, Lorenz Mösenlechner <
moesenle@in.tum.de> wrote:
>> In addition, I don't think that it is possible to convert all ros
>> packages to catkin. Catkin is all about building but what about the
>> packages that don't contain code? There are a lot of packages out
>> there that contain only launch files and dependencies to build
>> everything that is necessary to start these launch files. The most
>> prominent examples would be pr2_bringup or pr2_gazebo. I don't really
>> see how that is possible in a catkinized world.
>
> Those packages will have to be converted, too, because the launch
> scripts will need to be installed somewhere. In that sense, XML files
> *are* code.
managing dependencies and building everything that is necessary.
At the moment, I guess most people have a lot of packages in a
repository, not necessarily organized in stacks and possibly with some
packages being broken. In particular, some of the packages are there
only for specifying dependencies that are required to run something,
e.g. for bringing up the robot. With rosmake, getting everything built
that is required for running the robot is easy. With catkin, I would
either need to build everything in that repo which might not work due
to broken packages or I would need to create something like a build
workspace that symlinks only the packages I want to build. Both
solutions would require some manual work.
I think one reason for the great success of ros was that it made it
really easy to use other people's code and get the dependencies
resolved with just one call to rosmake. With catkin, the user needs to
be more organized and building something just requires more knowledge
and more work.
Lorenz
--
Tully Foote
tfoote@willowgarage.com(650) 475-2827