[ros-users] Recovering data from corrupted bag files?

[Jeremy Leibs]

Hi Andrew,

On Fri, Nov 12, 2010 at 4:03 PM, Andrew Chambers <achamber at gmail.com> wrote:
> I captured a large bag file (8GB) onto a flash drive and I'm having
> some problems with the bag file. I'm hoping someone has some thoughts
> about how to recover the data.
>
> After doing a rosbag check on the file, I'm prompted to reindex. After
> reindexing, I can play and get info on the bag file but the duration
> of the file is much shorter than it should be.
>
> You can see the size is 8GB but the duration is only 252 seconds.
> Normally when recording bag files on this system, we get about 5MB/sec
> but here it is 341 MB/sec. I was using bviplus
> (http://bviplus.sourceforge.net/) to look at the raw data in the bag
> file.

Thanks for the report.  You do appear to have a corrupt bag.  There is
a completely empty region in the bag file from
51a55200 to 51d95200.  Seems like some number of blocks didn't get
written.  This is unfortunately partially overlapping with the
connection index for chunk 1263, which is where rosbag chokes reading
in the index.

The problem you are seeing in the size is a bug in the reindexer.
Once it determines the new index and writes it in place, it never
truncates the bag.  So, the remaining 7 gigs worth of file hang around
and take up space, but aren't accessible or visible to the rosbag
tools.  I've ticketed this (https://code.ros.org/trac/ros/ticket/3142)

However, just truncating the bag doesn't help you recover the data.
The good news is that most of the data seems to be there, and
accessible via the master index.  The bad news is the way the tools
are written don't necessarily make it trivial to fix this problem at
the moment.  I've captured this at
https://code.ros.org/trac/ros/ticket/3143.

--Jeremy