[ros-users] [Discourse.ros.org] [Next Generation ROS] Downloading Dependencies

William Woodall ros.discourse at gmail.com
Wed Mar 15 04:12:22 UTC 2017

To answer this properly, I need to layout the workflow we usually use and the different tools we use to accomplish this.

Here's the steps first (without details):

- Generate a list of things to download, i.e. "what do I need to download to build `rviz`?"
- Download everything based on that list
- Install the remaining dependencies, e.g. non-ROS things like `qt` and `boost`
- Build everything, in the correct order
- Redistribute binaries

This basic formula is how I start working on everything (except maybe redistributing binaries).
You can see the pattern in our from source instructions on the wiki:


And now that list again with more notes in between:

- Generate a list of things to download, i.e. "what do I need to download to build `rviz`?"
  - We use `rosinstall_generator` for this and it draws on data in the "rosdistro"
  - it generates a `.rosinstall` file based on your query
  - it only gets the location of things that are "ROS packages", i.e. things released in the rosdistro
  - it does not figure out how to download things like qt or boost
  - it is used in ROS 1 and will likely be used more-or-less unchanged in ROS 2
  - can also answer "what do I need to download to use `rviz`, minus `ros_comm` and its dependencies?"
    - you might want to do this so you can install `ros_comm` and down from binaries
- Download everything based on that list
  - We use `wstool` for this (some power users use `vcstool`, we use it in ROS 2)
  - Given a `.rosinstall` (or a similar list of things to go get), it can fetch all the source code
  - We'll likely use both in ROS 1 and ROS 2
- Install the remaining dependencies, e.g. non-ROS things like `qt` and `boost`
  - It can also install binaries for "ROS things" if you don't intend to build them locally and there are binaries for them on the target OS.
  - In ROS 1 we use `rosdep` for this, it would install remaining dependencies
  - In ROS 2 we just have instructions in the ros2 github wiki on what to install, we don't use `rosdep` yet
- Build everything, in the correct order
  - used to help us build software locally
    - either to do development or
    - to build for a new OS/architecture which lacks binaries
  - software might be anything supported by the build tool, like:
    - catkin or
    - ament or
    - cmake or
    - python with setup.py files
    - etc...
  - we use mostly catkin in ROS 1 and mostly ament in ROS 2, but that doesn't really affect this part of the process
  - all result in an "install-like" folder which can be used locally
- Redistribute binaries
  - One option is to use `bloom` and release on build.ros.org through rosdistro
    - `bloom` converts the contents of `package.xml` to debian (or others like fedora) packaging files
    - this happens on a per package basis (no bundling)
    - build.ros.org uses these files to build binaries and upload them to packages.ros.org
  - Another option is something like what @Mike_Purvis presented at ROSCon, which builds lots of things at once and bundles the result
  - Both options should work for ROS 2, we're currently looking at any changes that would need to be made for `bloom` to work on ROS 2 packages (shouldn't be much)

Ok, so that's the breadth of what we do with some details.
Now I'll try to answer you directly by quoting your questions:

[quote="geoffviola, post:1, topic:1489"]
I didn't see anything about plans for a new rosdep for ament.

Though `rosdep` does have some ROS specific capabilities, at its core is just takes dependency names, like `qt` or `boost`, and decides what needs to be done on a system to "satisfy" those as dependencies.
This is a pretty generic "abstraction" over the OS's package manager, and is not really ROS specific.
And so it should work perfectly fine with ROS 2 as-is.

That being said we have thought about improvements to `rosdep` and dreamed of removing the the ROS specific stuff from it so that it's more generic and potentially useful to many other projects outside of ROS.
Specifically @NikolausDemmel worked on something called `xylem` while interning at OSRF, and it was meant to be the spiritual successor to `rosdep`:


Despite his really awesome work, I've not had the resources to capitalize on it and push for it to replace `rosdep`.
I've always hoped we could make use of it in ROS 2, but it's really an orthogonal issue as ROS 2 could easily use `rosdep` as well, it would just be nice to use `xylem`'s new hotness :wink:.
@NikolausDemmel put together some extensive design documentation and the tool is really close to being ready for use.
If you're interested in tools like `rosdep` I highly encourage anyone to have a look at his work:


So, to answer you, we don't need a new `rosdep` for `ament`, but we'd like to have one :smile:.

[quote="geoffviola, post:1, topic:1489"]
Based on the ROS 2 binary installations, I figure that the remote dependencies are managed by brew for Mac, Chocolatey for Windows, and apt for Ubuntu. Are there plans to have a generic tool to download these dependencies based on the package.xml like rosdep?

Yes, we'll likely use `rosdep`.
We are currently working on packaging some dependencies like `tinyxml` and `eigen` for chocolatey on Windows, and still sort of feeling out what the best thing to do there is, see:

- https://github.com/nuclearsandwich/tinyxml-choco/
- https://github.com/nuclearsandwich/eigen-choco

These aren't official, and will likely move, but for those interested in seeing what's been tried...

[quote="geoffviola, post:1, topic:1489"]
Unfortunately, the previous version of rosdep didn't handle downloading repositories directly.

Actually `rosdep` does have this ability, it's called "rosdep source installer" (I think).
They're not used much anymore, because they're terribly difficult to maintain.
They looked something like this:

    9.04: liblog4cxx10-dev
    8.10: |
      if [ ! -f /opt/ros/lib/liblog4cxx.so.10 ] ; then
        mkdir -p ~/ros/ros-deps
        cd ~/ros/ros-deps
        wget --tries=10 http://pr.willowgarage.com/downloads/apache-log4cxx-0.10.0-wg_patched.tar.gz
        tar xzf apache-log4cxx-0.10.0-wg_patched.tar.gz
        cd apache-log4cxx-0.10.0
        ./configure --prefix=/opt/ros
        sudo make install
    8.04: |
      if [ ! -f /opt/ros/lib/liblog4cxx.so.10 ] ; then
        mkdir -p ~/ros/ros-deps
        cd ~/ros/ros-deps
        wget --tries=10 http://pr.willowgarage.com/downloads/apache-log4cxx-0.10.0-wg_patched.tar.gz
        tar xzf apache-log4cxx-0.10.0-wg_patched.tar.gz
        cd apache-log4cxx-0.10.0
        ./configure --prefix=/opt/ros
        sudo make install


Since then we've deliberately chosen to avoid stuff like this and try to only use OS package managers to resolve dependencies.
The other point is that you can emulate this by creating a cmake package (or catkin/ament package) which does this "from source build" for external dependencies.
We do this some for ROS 2, for example with `poco` (cmake based) because we needed it on Windows and didn't know how to package it yet, see:


Basically, if it cannot find the right version of poco, then it downloads and builds it in place.

`xylem` also touched on this topic:

http://xylem.readthedocs.io/en/latest/design.html#goals (search for `source installer`)

As for cross-compiling, there are entire toolchains for achieving this, like OpenEmbedded, Yocto, and others.
I am not confident that is a role `rosdep` needs to play, but we can certainly make sure to help support those patterns if we can (i.e. not get in the way).

[quote="geoffviola, post:1, topic:1489"]
Ultimately, it would be nice to have one command to convert a repository/package(s) into artifact(s).

`bloom` can provide you with Debian packaging files on a per package basis, and the Debian packaging toolchain can provide you with binary artifacts.
The same is true for any packaging scheme (Fedora, Homebrew, Chocolatey), in principle.
ROS Answers on how to do this locally, see:


[quote="geoffviola, post:1, topic:1489"]
Another tool to help get there would be to convert a package or packages into workspace.

Not sure what you mean by that.
We have tools that will build lots of local packages into a single destination folder.
We have people using this ability to wrap up the result and distribute that, see @Mike_Purvis's presentation.
I think these tools should continue to work for ROS 2 with little to no changes.

Sorry for the long post, but hopefully it answered your questions.

[Visit Topic](https://discourse.ros.org/t/downloading-dependencies/1489/2) or reply to this email to respond.

More information about the ros-users mailing list