[ros-users] Storing many files in a database ROS style

Fri Jun 11 22:52:25 UTC 2010

Well, right now I have the following

each text file has 6 columns, I represent each line using a ros msg of six
entities
then represent the whole file as array of the line msg which looks ok to me.

then I was thinking of creating a master msg(just a thought) which I call
database which will contain arrays of the file msgs, this way I can iterate
using the Header entity which I have in every file msg when I want to locate
a particular file and then load the contents from there.

The catch here is that I have a bunch of services which I want to stuff like
add file(so request is a file msg), then I should be able to sort of append
the new msg to the database msg and this should be storable to a file in the
state I leave it in.

of course if I could store msgs(directly) it would be much easier to just
store the file msgs. I am really not sure what to do, I think I am going to
look into the folowing for now:

- try to implement a simple conversion from text file->msg then msg-> text
file
- try to implement some sort of file system to directly store msg files
- since bags require me to have topics, and I have mainly services, i dont
think there is much I can do about them right now
- alternatively I can just use sqlite to have a small database and make
interfaces for accessing it, but then will mess with my nice communication
of msgs containing the info.

On Fri, Jun 11, 2010 at 6:30 PM, Jeremy Leibs <leibs at willowgarage.com>wrote:

> So, if you are just looking to go from memory to disk and back to memory
> again, bag files are possibly an entirely reasonable choice starting with
> cturtle.  To save, you could write all the messages into a bag.  And then
> when you start your program you could give it a bag to load all the messages
> from again.
>
> However, the caveat is that the only structure of messages you can store at
> the moment is an ordered (or unordered) set.  You can abuse the time
> parameter to create an arbitrary ordering.  However, if you have some other
> data structure of messages, e.g., a dictionary mapping strings to messages,
> there is no way to map that into a bag file.
>
> Clearly there would be great value in allowing bags to be created and
> indexed using arbitrary key-types such as strings or ints, but it will still
> be quite some time before that gets implemented.
>
> So, if your database state can be captured completely by a suitable ordered
> set of messages, go ahead and try using bags.  Otherwise you might have to
> implement something on your own.  If you do have to implement your own file
> format, you should take a look at the ROS serialization API.  All ROS
> messages can fairly easily be serialized into an appropriate binary block
> for writing to file and that same binary block, when loaded back from the
> file can be deserialized into a message.
>
> --Jeremy
>
>
>
> On Fri, Jun 11, 2010 at 2:43 PM, Billy Okal <b.okal at jacobs-university.de>wrote:
>
>> Yes I wanted a more ROS approach to the storage, I could of course just
>> convert them back to text files and store them, but I I was looking for
>> something smarter than that and I though bag files could do some magic here,
>> if not at least this should be ticketed or something like that( a way to
>> store msg data directly into mem and be able to load it and do stuff on it)
>>
>> thanks
>>
>>
>> On Fri, Jun 11, 2010 at 5:23 PM, Jeremy Leibs <leibs at willowgarage.com>wrote:
>>
>>> Billy,
>>>
>>> I'm sorry I'm not entirely sure what it is that you are looking for.  If
>>> you want things to be fast, it sounds like you can load all of these files
>>> as messages into memory and work with them there without too much trouble.
>>> 1000 files of 4000 floats is only about 30MB, even if you store them as
>>> doubles you should be fine.
>>>
>>> Are you just looking for a way to write the messages out to disk and load
>>> them in again at a later point?  It sounds like you already have a way of
>>> reading your text files and converting them to messages.  Is it that you
>>> haven't implemented a way of writing them back out to your text format, or
>>> are you wanting to use a different, ROS-native, format to store them?
>>>
>>> If you could elaborate a little bit on what it is that you are looking
>>> for a solution to I can maybe point you in the right direction.
>>>
>>> --Jeremy
>>>
>>>
>>> On Fri, Jun 11, 2010 at 1:38 PM, Billy Okal <b.okal at jacobs-university.de
>>> > wrote:
>>>
>>>> Dear Ros Users,
>>>>
>>>> I am writting a ros package to emulate a database in which I want to
>>>> store many(1000+) text files(each containing about 4000 float values).
>>>> Originally I though of using bag files to store this info but now it looks
>>>> like thats not what I want. Basically I want to store these files and be
>>>> able to perform Addition, Deletion, and Querying(compare one text file
>>>> against all the rest to infer some properties of grasps). Right now I have
>>>> made Ros msg representing the text files and can move them around ros, but
>>>> the problem is storing these messages in some lasting memory so that I just
>>>> load the database(which I can also make a ros message) in the beginning and
>>>> to stuff then write it back.
>>>>
>>>> Does anybody have experience in this or something related or just some
>>>> ideas I could use to go about this? Also I need to querying to be really
>>>> fast(the goal is real time control)
>>>>
>>>> --
>>>> Best Regards,
>>>>
>>>> Billy Okal
>>>>
>>>> sure vi is user friendly, its just particular about who to be friends
>>>> with
>>>>
>>>> _______________________________________________
>>>> ros-users mailing list
>>>> ros-users at code.ros.org
>>>> https://code.ros.org/mailman/listinfo/ros-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> ros-users mailing list
>>> ros-users at code.ros.org
>>> https://code.ros.org/mailman/listinfo/ros-users
>>>
>>>
>>
>>
>> --
>> Best Regards,
>>
>> Billy Okal
>>
>> sure vi is user friendly, its just particular about who to be friends with
>>
>> _______________________________________________
>> ros-users mailing list
>> ros-users at code.ros.org
>> https://code.ros.org/mailman/listinfo/ros-users
>>
>>
>
> _______________________________________________
> ros-users mailing list
> ros-users at code.ros.org
> https://code.ros.org/mailman/listinfo/ros-users
>
>

-- 
Best Regards,

Billy Okal

sure vi is user friendly, its just particular about who to be friends with
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ros.org/pipermail/ros-users/attachments/20100611/3d9b0fac/attachment-0003.html>