[ros-users] [Discourse.ros.org] [Next Generation ROS] Optional Fields in Message

[Tully Foote]

Thanks for the pointer to the changes between Protobuf 2 and Protobuf 3. A few other searches found some other interesting opinions about proto2 vs proto3 such as ["all fields are optional"](http://stackoverflow.com/a/30201500) and that the proto3 version is ["simpler'](https://stackoverflow.com/questions/31801257/why-required-and-optional-is-removed-in-protocol-buffers-3).

Although the answer that "everything is optional" is a bit tongue in cheek. I think that it actually has a valuable insight. In particular for the API, some fields may not need to be filled. And this depends completely on the message documentation. Messages are both a datatype as well as a documented standard on how to fill them out. Some messages provide multiple fields but [only some are considered valid based on a flag](http://docs.ros.org/api/visualization_msgs/html/msg/Marker.html). Similarly some messages use [variable length arrays as optional fields](http://docs.ros.org/kinetic/api/sensor_msgs/html/msg/JointState.html) where there are some configurations where some arrays may be empty. Which is basically using the implicit flag of the length of the array, which is an introspectable property of the message.

Using the empty arrays gains the benefit of not needing to send the empty data fields. (Less the overhead inherent to communicating that the array is empty.) 

Two other advantages of "optional" data fields are the ability to not send the data if it's not used and "backwards" compatibility between two versions of a message where an "optional" field has been added and the receiver can automatically drop the extra field that it does not know about. 

Not actually sending blank data can be covered as above by using variable length arrays. And the linked article above from the Capn Proto FAQ covers how the "compatibility" breaks down quickly. 

The other major change that supporting optional fields as protobuf does is that it requires us to switch to method based access. Such that you must query whether or not a field exists before you ask for it. Our community has shows a strong preference for having member based access to the datatypes for the ability to implement high performance methods using things like iterators without going through accessors or making a local copy that would cause significant overhead.

With the new bounded arrays, we could pick up a new common message pattern in the case that you want to have an optional field. 

```
...
sensor_msgs/Image optional_image_field[<=1]  # Optional image in this message.
...
```

Though when designing messages like this it needs to be kept in mind that adding this complexity will force all subscribers to handle both cases. Or if there's multiple optional fields the number of cases can grow combinatorically.

[quote="geoffviola, post:3, topic:991"]
That would be nice and could be another way of getting the same thing. For example, somehow relating the 1D gyro to the 9D IMU message without much overhead or specification. I was mostly thinking about the inherent compatibility with one message type.
[/quote]

I'm not sure that optional fields are the best approach for this. In this case I'd recommend that using the full 9D message with the appropriate zero entries in the covariance. This means that all the tools can be written against a single message. We've developed the same policy for 2D vs 3D poses. Where if you're working in 2D you still use the 3D points. And then the messages can be rendered following the same code path in all the tools instead of needing to have two different code paths.  This approach has proven very powerful as primatives used by the 2D navigation stack which only supports 2D navigation can be reused when people are starting to do 3D free space navigation and almost all the tools still work.


[quote="Daniel_Stonier, post:6, topic:991, full:true"]
As for a direct use case, we ran into one recently when wanting to write a ROStful (ROS-REST conversion interface) client. As a client to a REST server api, it may need to set outgoing optional fields and parse incoming optional fields. A fixed ros message is not rich enough to transmit/receive this kind of information.
[/quote]

Can you clarify what you mean by this? Are you trying to represent an arbitrary REST message?






---
[Visit Topic](https://discourse.ros.org/t/optional-fields-in-message/991/8) or reply to this email to respond.