[ros-users] [Discourse.ros.org] [Next Generation ROS] IPC in ros2

Geoffrey Biggs ros.discourse at gmail.com
Sat Sep 30 05:52:25 UTC 2017



I attended the OMG meeting last week, so I took the opportunity to hear what PrismTech and RTI/TwinOaks/eProsima had to say about their competing submissions. The following are the notes I took during their presentations. Remember while reading this that it is based solely on the presentations. I haven't read the submissions themselves in any detail yet.

Prismtech presentation:

- They believe that their proposal is more efficient in the wire protocol (trying to save every single byte possible), and supports brokered as well as peer-to-peer communication.
- They have invited the competing submitters to join the PrismTech submission.
- PrismTech believes that they have presented their final submission in Brussels (June) and so were expecting to vote for acceptance in New Orleans (September), but because there are still two competing submissions, and no final submission document was received (only a presentation), this is not possible.
- PrismTech's reasons for going forward with their own submission:
  - XRCE tries to target the most wire/power/memory efficient protocol, targeting not just IP infrastructures but a whole range of infrastructures.
  - Their submission provides reliability and fragmentation.
  - Their current prototype runs on an 8-bit microprocessor with 1 KB of RAM and has a wire overhead of 4 bytes for data samples.
  - They got a review from the AB which was favourable (only editorial comments), and they believe that the AB review shows that they satisfy all the mandatory requirements.
- XRCE applications can be brokered into a DDS data space, or they can discover one another and communicate P2P.
- Their submission is only about the protocol, it does not say anything about the API.
- A DDS-XRCE Agent running on permanently connected hardware provides the access to the rest of the (non-XRCE) DDS domain.
- XRCE provides a data space abstraction in which applictions can read and write data autonomously and asynchronously.
- Data read and written by XRCE applications is associated with one or more resources identified by a URI.
  - An XRCE resource is a closed description for a set of named values.
  - Resource URIs allow for wildcards, which means that more than one resource can be targeted in one definition, which is useful for capturing collections of sub-namespaces.
  - A resource with a cardinality of one is called a Trivial Resource.
  - Resources also have properties, which are used to attach QoS settings to them, such as "this resource is transient" or "this resource is reliable".
  - An XRCE selection is the conjunction of a resource and a predicate over the resource content and properties, used to filter a selection. For example, finding all lights with a luminosity greater than zero. (It seems to provide functionality similar to a very basic subset of SQL.)
- There is a mapping from XRCE resources to DDS topics.
- Resource serialisation uses the same format as DDS.
- All XRCE messages are a single-byte header and a body.
  - There are flags in the header for reliability and synchronicity.
  - Messages can be "decorated" by prefixing them with a one-byte pre-header, which determines some of the properties for the following header byte, such as fragmentation.
  - The protocol is little-endian.
  - Variable-length encoding is used to save space.
- For addressing, the source address for each message is assumed to be a unique address of the sender.
- The protocol is modular, with a core profile (required for any communication), an optional query profile, and an optional discovery profile.
  - e.g. If using a communications system like Bluetooth, which already has discovery, the XRCE discovery profile can be removed, saving some space.
  - Discovery happens by scouting (sending a scout message to ask for types of nodes to reply, i.e. broker nodes or durability services or peers or clients).
  - Other nodes reply to a scout with a HELLO message.
- After discovery, two nodes need to open a session, which enables publishing and subscribing between those two nodes.
  - Authentication data is included in the session open message.
  - Locator information, telling the other node how it can be reached, is included in the open message.
  - The receiving node replies with an accept message or a reject message.
  - There is an FSM describing the states a session goes through.
- Resources and selections are uniquely identified by numerical IDs to save space on the wire.
- A declare message is sent to declare what resources, selections, etc. a node has or will publish or subscribe to.
- Subscriptions can be push, pull, periodic pull or periodic push.
- All data messages and declarations are transported over a conduit (for the session), which is a pair of a reliable and a best-effort channel. Multiple conduits may be used in parallel to avoid head-of-line blocking and allow multicasting.
- One decorator is used to select the conduit for the next message.
  - It is possible to make this decorator one byte instead of two if the number of conduits is less than five.
- There is a sync message available to set the next sequence number to expect (e.g. for when not starting at zero).
- The ACKNACK message can acknowledge multiple messages at once, usually up to the given sequence number. It has the option to optionally request retransmission of one or more messages after that point.
- There is a one-shot write function in the protocol, which can do a write of a resource without needing to do any prior registration or discovery.
- This proposal appears to be a very complex protocol with a lot of options in message header structures. An implementation would have a lot of choice points during the decoding of a stream of messages. The "decorator" idea especially may save bytes on the wire (and in message construction buffers) in some cases, but it complicates the protocol implementation.
- PrismTech are already trying to bring their submission to market as a product.

RTI/eProsima/TwinOaks presentation:

- This proposal also uses an XRCE agent to provide access from DDS-XRCE nodes and the DDS domain.
- An important use case for them is that devices will tend to mostly sleep, and only wake up occassionally to do some processing, and send and receive data. This means that two devices may never be awake at the same time. This is why they have the agent, which is permanently present and provides a way for XRCE nodes to communicate with each other.
- They say that their proposal focuses not just on the XRCE protocol, but also on the interaction between the XRCE protocol and the DDS domain.
- It is possible for an XRCE agent to behave as an XRCE client to another agent, allowing for a hierarchical structure of agents and clients.
- Their proposal is based on the web-enabled DDS specification, with a DDS-XRCE object model in the agent that has a one-to-one mapping to the DDS data model.
- eProsima have put up a demo on Youtube: https://www.youtube.com/watch?v=HJ5eBQ2tZNQ
- Their demo uses 43 KB of RAM (much bigger than the PrismTech proposal, which can fit in 1 KB).
- The XRCE Agent object model is very similar to the DDS object model, making the mapping very simple.
- XRCE objects are modeled as resources that are addressable by their name and have CRUD operations.
  - Each resource has a name to address it within the agent and a context; a representation describing the resource; and an ID.
  - Resources can be represented as a name, an XML description, or a binary representation.
- Authentication capability is built into the proposal.
- Types are typically pre-defined profiles in the XRCE Agent.
  - It is possible to transmit types as binary or XML representations.
- Message structure:
  - A message header is either 4 or 8 bytes, depending on if a clientKey is used.
  - There is a sub-message header, which is another 8 bytes.
  - It is possible to send data in a sequence, meaning the header only needs to be sent once for a bunch of samples.
- The transport can be message-oriented or packet-oriented.
- CDR and DDS-XML representations are reused.
- Message overhead is typically 12 bytes.
  - They consider that further reduction in overhead would increase complexity and reduce robustness. e.g. They do not need different code paths for multiple sessions, re-connections, handling variable-length encoding.
  - They say it compares favourably to TCP/IP overhead (40 bytes).
  - They think they could save 6 bytes from their message header, but the reduction is only 6% in the context of total overhead (when considering the use case of using TCP/IP as the underlying transport protocol) and so is not worth it in the face of increased complexity.
- They also received 3 AB reviews, which they claim were supportive and did not find any non-editorial problems.

My summary:

- The submissions are very different. PrismTech is aiming for cutting down the bytes used by the protocol to the absolute minimum at the expense of all else. RTI/eProsima/TwinOaks are favouring some overhead in order to achieve a simpler and more robust protocol and implementation.
- PrismTech's protocol is overly complex with too many choices during the decoding of a message.
- The extra overhead of the RTI/eProsima/TwinOaks submission is not likely to be a problem in the majority of embedded micro-processors used these days (although I'm not sure how many 8-byte, 1KB-of-RAM micros there are in use in new products). However, their consideration that TCP/IP is going to be the most common transport may not be accurate.





---
[Visit Topic](https://discourse.ros.org/t/ipc-in-ros2/2619/8) or reply to this email to respond.




More information about the ros-users mailing list