For testing, we generally upload bag files to a web-accessible
location, then use build macros to download those test files on
demand. This enables fast checkout of the test packages, with the
overhead of downloading the test data enforced on running the tests.
Storing bag files outside of version control does have its issues --
bag files, especially when messages are changed, do have versions, so
a best practice is often to encode the version in the filename if it
is not stored in a VCS.

For collecting the files, we configure our roslaunch scripts to log
bag files to separate partitions, one for diagnostics data, one for
researchers. As you allude to, writing of large data files has system
configuration issues. For example, logging of sensor data can quickly
overwhelm a partition or saturate a drive. roslaunch biases towards
explicitly configuring output locations for these reasons.

 - Ken

On Tue, Mar 2, 2010 at 8:38 PM, Jack O'Quin <jack.oquin@gmail.com> wrote:
> On Sat, Feb 27, 2010 at 12:47 PM, Rosen Diankov <rosen.diankov@gmail.com> wrote:
>> with regards to reading files on the system (which will always be
>> necessary unless morgan/brian/ken/josh plan on developing a FS on top
>> of ROS), wasn't there some helper functions makes locating files in
>> ros packages easier? For example specifying a filename of
>> "ros-pkg://pkgname/myfile" will automatically look for myfile in
>> pkgname. In regards to roslaunch, you can specify files using $(find
>> pkgname)/myfile, although the node itself will lose the knowledge that
>> myfile came from a ROS package.
>>
>> The decision to set the working directory to ros-root for all nodes
>> makes file usage more explicit, and hence easier for others to work
>> with and understand what's going on. If octave had a better way of
>> managing paths, perhaps there wouldn't be a need for 'cwd' at all ;0)
>
> This seems to suggest that only package files should be read or
> written from roslaunch.
>
> I am concerned with establishing a clean test methodology with the
> ability to read and write bag files obtained on the robot. Experience
> suggests that using SVN for that purpose works OK only as long as the
> bag files are moderately small, but for Velodyne LIDAR or camera data
> the files get too large for SVN to handle efficiently (many megabytes
> per second).
>
> How do the ROS developers deal with this problem? Log files are
> automatically saved in ~/.ros/log subdirectories.  Is there some
> equivalent method for saving bags? Should they be saved in the same
> subdirectory? How does roslaunch support this methodology?
>
> Do I need to create test directories with manifest.xml files to fit
> the procrustean bed of roslaunch? What is the recommended method of
> archiving these data?
> --
>  joq
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> ros-users mailing list
> ros-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ros-users
> _______________________________________________
> ros-users mailing list
> ros-users@code.ros.org
> https://code.ros.org/mailman/listinfo/ros-users
>