[ros-users] Storing many files in a database ROS style

Jeremy Leibs leibs at willowgarage.com
Fri Jun 11 22:30:14 UTC 2010


So, if you are just looking to go from memory to disk and back to memory
again, bag files are possibly an entirely reasonable choice starting with
cturtle.  To save, you could write all the messages into a bag.  And then
when you start your program you could give it a bag to load all the messages
from again.

However, the caveat is that the only structure of messages you can store at
the moment is an ordered (or unordered) set.  You can abuse the time
parameter to create an arbitrary ordering.  However, if you have some other
data structure of messages, e.g., a dictionary mapping strings to messages,
there is no way to map that into a bag file.

Clearly there would be great value in allowing bags to be created and
indexed using arbitrary key-types such as strings or ints, but it will still
be quite some time before that gets implemented.

So, if your database state can be captured completely by a suitable ordered
set of messages, go ahead and try using bags.  Otherwise you might have to
implement something on your own.  If you do have to implement your own file
format, you should take a look at the ROS serialization API.  All ROS
messages can fairly easily be serialized into an appropriate binary block
for writing to file and that same binary block, when loaded back from the
file can be deserialized into a message.

--Jeremy


On Fri, Jun 11, 2010 at 2:43 PM, Billy Okal <b.okal at jacobs-university.de>wrote:

> Yes I wanted a more ROS approach to the storage, I could of course just
> convert them back to text files and store them, but I I was looking for
> something smarter than that and I though bag files could do some magic here,
> if not at least this should be ticketed or something like that( a way to
> store msg data directly into mem and be able to load it and do stuff on it)
>
> thanks
>
>
> On Fri, Jun 11, 2010 at 5:23 PM, Jeremy Leibs <leibs at willowgarage.com>wrote:
>
>> Billy,
>>
>> I'm sorry I'm not entirely sure what it is that you are looking for.  If
>> you want things to be fast, it sounds like you can load all of these files
>> as messages into memory and work with them there without too much trouble.
>> 1000 files of 4000 floats is only about 30MB, even if you store them as
>> doubles you should be fine.
>>
>> Are you just looking for a way to write the messages out to disk and load
>> them in again at a later point?  It sounds like you already have a way of
>> reading your text files and converting them to messages.  Is it that you
>> haven't implemented a way of writing them back out to your text format, or
>> are you wanting to use a different, ROS-native, format to store them?
>>
>> If you could elaborate a little bit on what it is that you are looking for
>> a solution to I can maybe point you in the right direction.
>>
>> --Jeremy
>>
>>
>> On Fri, Jun 11, 2010 at 1:38 PM, Billy Okal <b.okal at jacobs-university.de>wrote:
>>
>>> Dear Ros Users,
>>>
>>> I am writting a ros package to emulate a database in which I want to
>>> store many(1000+) text files(each containing about 4000 float values).
>>> Originally I though of using bag files to store this info but now it looks
>>> like thats not what I want. Basically I want to store these files and be
>>> able to perform Addition, Deletion, and Querying(compare one text file
>>> against all the rest to infer some properties of grasps). Right now I have
>>> made Ros msg representing the text files and can move them around ros, but
>>> the problem is storing these messages in some lasting memory so that I just
>>> load the database(which I can also make a ros message) in the beginning and
>>> to stuff then write it back.
>>>
>>> Does anybody have experience in this or something related or just some
>>> ideas I could use to go about this? Also I need to querying to be really
>>> fast(the goal is real time control)
>>>
>>> --
>>> Best Regards,
>>>
>>> Billy Okal
>>>
>>> sure vi is user friendly, its just particular about who to be friends
>>> with
>>>
>>> _______________________________________________
>>> ros-users mailing list
>>> ros-users at code.ros.org
>>> https://code.ros.org/mailman/listinfo/ros-users
>>>
>>>
>>
>> _______________________________________________
>> ros-users mailing list
>> ros-users at code.ros.org
>> https://code.ros.org/mailman/listinfo/ros-users
>>
>>
>
>
> --
> Best Regards,
>
> Billy Okal
>
> sure vi is user friendly, its just particular about who to be friends with
>
> _______________________________________________
> ros-users mailing list
> ros-users at code.ros.org
> https://code.ros.org/mailman/listinfo/ros-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ros.org/pipermail/ros-users/attachments/20100611/f2c192aa/attachment-0003.html>


More information about the ros-users mailing list