10 KiB
ROS 2 design notes for instrumentation
The goal is to document ROS 2's design/architecture in order to properly design the instrumentation for it.
Notes on client libraries
ROS 2 has changed the way it deals with client libraries. It offers a base ROS client library (rcl) written in C. This client library is the base for any language-specific implementation, such as rclcpp and rclpy.
However, rcl is obviously fairly basic, and still does leave a fair amount of implementation work up to the client libraries. For example, callbacks are not at all handled in rcl, and are left to the client library implementations.
This means that some instrumentation work might have to be re-done for every client library that we want to trace. We cannot simply instrument rcl, nor can we only instrument the base rmw interface if we want to dig into that.
This document will (for now) mainly discuss rcl and rclcpp, but rclpy should eventually be added and supported.
Flow description
Process creation
In the call to rclcpp::init(argc, argv), an rclcpp::Context object is created and CLI arguments are parsed. Much of the work is actually done by rcl through a call to rcl_init().
This has to be done once per process, and usually at the very beginning. The components that are then instanciated share this context.
sequenceDiagram
participant process
participant rclcpp
participant rcl
participant tracetools
process->>rclcpp: rclcpp::init()
Note over rclcpp: allocates <div></div> rclcpp::Context object
rclcpp->>rcl: rcl_init(out context)
Note over rcl: validates & processes context object
rcl->>tracetools: TP(rcl_init, &context)
Note/component creation
In ROS 2, a process can contain multiple nodes. These are sometimes referred to as "components."
These components are instanciated by the containing process. They are usually classes that extend rclcpp::Node, so that the node initialization work is done by the parent constructor.
This parent constructor will allocate its own rcl_node_t handle and call rcl_node_init(), which will validate the node name/namespace. rcl will also call rmw_create_node() to get the node's rmw handle (rmw_node_t). This will be used later by publishers and subscriptions.
sequenceDiagram
participant process
participant Component
participant rclcpp
participant rcl
participant rmw
participant tracetools
Note over rmw: (implementation)
process->>Component: Component()
Component->>rclcpp: : Node()
Note over rclcpp: allocates rcl_node_t handle
rclcpp->>rcl: rcl_node_init(out rcl_node_t, node_name, namespace)
Note over rcl: validates node name/namespace
Note over rcl: populates rcl_note_t
rcl->>rmw: rmw_create_node() : rmw_node_t
Note over rmw: creates rmw_node_t handle
rcl->>tracetools: TP(rcl_node_init, rcl_node_t *, rmw_node_t *, node_name, namespace)
Publisher creation
The component calls create_publisher(), a rclcpp::Node method for convenience. That ends up creating an rclcpp::Publisher object which extends rclcpp::PublisherBase. The latter allocates an rcl_publisher_t handle, fetches the corresponding rcl_node_t handle, and calls rcl_publisher_init() in its constructor. rcl does topic name expansion/remapping/validation. It creates an rmw_publisher_t handle by calling rmw_create_publisher() of the given rmw implementation and associates with the node's rmw_node_t handle and the publisher's rcl_publisher_t handle.
If intra-process publishing/subscription is enabled, it will be set up after creating the publisher object, through a call to PublisherBase::setup_intra_process(), which calls rcl_publisher_init().
sequenceDiagram
participant Component
participant rclcpp
participant Publisher
participant rcl
participant rmw
participant tracetools
Note over rmw: (implementation)
Component->>rclcpp: create_publisher(topic_name, options, use_intra_process)
Note over rclcpp: (...)
rclcpp->>Publisher: Publisher(topic_name, options)
Note over Publisher: allocates rcl_publisher_t handle
Publisher->>rcl: rcl_publisher_init(out rcl_publisher_t, rcl_node_t, topic_name, options)
Note over rcl: populates rcl_publisher_t
rcl->>rmw: rmw_create_publisher(rmw_node_t, topic_name, qos_options) : rmw_publisher_t
Note over rmw: creates rmw_publisher_t handle
rcl->>tracetools: TP(rcl_publisher_init, rcl_node_t *, rmw_node_t *, rcl_publisher_t *, topic_name, depth)
opt use_intra_process
rclcpp->>Publisher: setup_intra_process()
Publisher->>rcl: rcl_publisher_init(...)
end
Subscription creation
Subscription creation is done in a very similar manner.
The componenent calls create_publisher(), which ends up creating an rclcpp::Subscription object which extends rclcpp::SubscriptionBase. The latter allocates an rcl_subscription_t handle, fetches its rcl_node_t handle, and calls rcl_subscription_init() in its constructor. rcl does topic name expansion/remapping/validation. It creates an rmw_subscription_t handle by calling rmw_create_subscription() of the given rmw implementation and associates it with the node's rmw_node_t handle and the subscription's rcl_subscription_t handle.
If intra-process publishing/subscription is enabled, it will be set up after creating the subscription object, through a call to Subscription::setup_intra_process(), which calls rcl_subscription_init(). This is very similar to a normal (inter-process) subscription, but it sets some flags for later.
sequenceDiagram
participant Component
participant rclcpp
participant Subscription
participant rcl
participant rmw
participant tracetools
Note over rmw: (implementation)
Component->>rclcpp: create_subscription(topic_name, callback, options, use_intra_process)
Note over rclcpp: (...)
rclcpp->>Subscription: Subscription(topic_name, callback, options)
Note over Subscription: allocates rcl_subscription_t handle
Subscription->>rcl: rcl_subscription_init(out rcl_subscription_t, rcl_node_t, topic_name, options)
Note over rcl: populates rcl_subscription_t
rcl->>rmw: rmw_create_subscription(rmw_node_t, topic_name, qos_options) : rmw_subscription_t
Note over rmw: creates rmw_subscription_t handle
rcl->>tracetools: TP(rcl_subscription_init, rcl_node_t *, rmw_node_t *, rcl_subscription_t *, topic_name)
opt use_intra_process
rclcpp->>Subscription: setup_intra_process()
Subscription->>rcl: rcl_subscription_init(...)
end
rclcpp->>tracetools: TP(rclcpp_subscription_callback_added, rcl_subscription_t *, &any_callback)
Executors
An rclcpp::executor::Executor object is created for a given process. It can be a SingleThreadedExecutor or a MultiThreadedExecutor.
Components are instanciated, usually as a shared_ptr through std::make_shared<Component>(), then added to the executor with Executor::add_node().
After all the components have been added, Executor::spin() is called. SingleThreadedExecutor::spin() simply loops forever until the process' context isn't valid anymore. It fetches the next rclcpp::AnyExecutable (e.g. subscription, timer, service, client), and calls Executor::execute_any_executable() with it. This then calls the relevant execute*() method (e.g. execute_timer(), execute_subscription(), execute_intra_process_subscription(), execute_service(), execute_client()).
sequenceDiagram
participant process
participant Executor
participant tracetools
process->>Executor: Executor()
Note over process: instanciates components
process->>Executor: add_node(component)
process->>Executor: spin()
loop until shutdown
Executor->>tracetools: TP(?)
Note over Executor: get_next_executable()
Note over Executor: execute_any_executable()
Note over Executor: execute_*()
end
Subscription callbacks
Subscriptions are handled in the rclcpp layer. Callbacks are wrapped by an rclcpp::AnySubscriptionCallback object, which is registered when creating the rclcpp::Subscription object.
In execute_*subscription(), the Executor asks the Subscription to allocate a message though Subscription::create_message(). It then calls rcl_take*(). If that is successful, it then passes that on to the subscription through rclcpp::SubscriptionBase::handle_message(). This checks if it's the right type of subscription (i.e. inter vs. intra process), then it calls dispatch() on the rclcpp::AnySubscriptionCallback object with the message (cast to the actual type). This calls the actual std::function with the right signature.
Finally, it returns the message object through Subscription::return_message().
sequenceDiagram
participant Executor
participant Subscription
participant AnySubscriptionCallback
participant rcl
participant rmw
participant tracetools
Note over Executor: execute_subscription()
Executor->>Subscription: create_message(): std::shared_ptr<void>
Executor->>rcl: rcl_take*(rcl_subscription_t, out msg) : ret
rcl->>rmw: rmw_take_with_info(rmw_subscription_t, out msg, out taken)
Note over rmw: copies available message to msg if there is one
opt RCL_RET_OK == ret
Executor->>Subscription: handle_message(msg)
Note over Subscription: casts msgs to its actual type
Subscription->>AnySubscriptionCallback: dispatch(typed_msg)
AnySubscriptionCallback->>tracetools: TP(rclcpp_subscription_callback_start, this, is_intra_process)
Note over AnySubscriptionCallback: std::function::operator(...)
AnySubscriptionCallback->>tracetools: TP(rclcpp_subscription_callback_end, this)
end
Executor->>Subscription: return_message(msg)
Message publishing
To publish a message, an object is first allocated and then populated by the Component (or equivalent). Then, the message is sent to the Publisher through publish(). This then passes that on to rcl, which itself passes it to rmw.
TODO add inter- vs. intra-process execution flow TODO talk about IntraProcessManager stuff?
sequenceDiagram
participant Component
participant Publisher
participant rcl
participant rmw
participant tracetools
Note over Component: creates a msg
Component->>Publisher: publish(msg)
Note over Publisher: ...
Publisher->>rcl: rcl_publish(rcl_publisher_t, msg)
rcl->>rmw: rmw_publisher(rmw_publisher, msg)