Skip to main content
A ROS 2 node misbehaved once, in a run you cannot reproduce. Record what each process took in, in the order it took it, then replay that one process by itself, as often as you like, under a debugger or with extra logging, and get a verdict on whether it did the same thing again. This works for ROS 2 processes running on Cerulion’s transport (rmw_cerulion, see Bridge a ROS 2 system). The replayed process runs its own unmodified code. It gets every message, service request and response and /clock tick it got live, in the same order, and nothing else. The comparison checks each callback’s outputs against the recording.
cerulion ros2 run and ros2 launch need Linux: ROS 2 Jazzy publishes no macOS binaries. Replay is runtime-tested on ROS 2 Humble, with rclpy nodes, rclcpp nodes on ROS time and the stock demo_nodes_cpp executables; Foxy, Jazzy and Lyrical builds compile the same code.

​
Record

Put --record-deliveries <DIR> right after run or launch; everything after it goes to ros2 as usual:
Each process writes its own delivery log, ./logs/<exe>-<pid>.rdl.mcap. An existing file is never overwritten: a second process with the same name and pid writes <exe>-<pid>.1.rdl.mcap. To record a process launched some other way, set the variable instead:
A delivery log is not a topic recording. cerulion bag record records the frames on topics; a delivery log records what one process took from them, in its order. Both can run at once.
Recording costs time on the calling thread: each take copies its payload into the log and each output is hashed. The cost grows with message size, and the log grows by every payload taken. Record processes whose large topics run at a few Hz.

​
Replay

Replay one recorded process, alone, with the same command it was recorded with:
Nothing else needs to run: no talker, no clock, no robot. ros2 launch takes no replay flag, because a replay is one process. Setting CERULION_RMW_REPLAY=<log> does the same on any launch path. When the replay starts, <log>.verdict.json says running; the real verdict replaces it once the log is used up. Stop the process with Ctrl+C after the file no longer says running. A replay that crashes or is killed leaves running.
A replay really publishes. The replayed process sends its outputs on the transport as it did live, and through cerulion-netd they can reach other hosts. Never replay a controller or a driver where a live robot, or anything that acts on its topics, can receive them: replay on a machine of its own, or on one where no live node runs and cerulion-netd is stopped or started with CERULION_NETD_NETWORK=off.

​
Read the verdict

A divergence is also printed on stderr when the verdict is written, naming the process, node, topic and window. /rosout and the wall-clock stamp in a message header are never compared.

​
When a replay is exact

A replay reproduces the recorded run when all of these hold for the process:
  • Every thread that runs callbacks is a single-threaded executor thread. Several single-threaded executors on several threads are fine; a multi-threaded executor is refused by name.
  • No node logic reads the wall or steady clock. Use ROS time (use_sim_time, with /clock recorded); a wall timer can diverge, and when it changes an output the verdict names it.
  • Random numbers are seeded or absent.
  • No callback blocks on work another thread owns.
  • No input reaches the node outside the rmw: no sockets, devices, files it reads during the run, or shared memory of its own.
  • The same binaries, parameters and distro are used. A log recorded by another command line, ROS_DISTRO or build of librmw_cerulion.so is refused at start, naming the field.
What a pass does not prove: an identical verdict shows that this recording replays, not that the node is deterministic, and a race that did not fire while recording stays hidden. Each process replays alone against its own log, so processes that feed each other are not replayed together. EventsExecutor, rclcpp intra-process communication and loaned takes are not covered; a replay turns loans off.

​
Next steps

Bridge a ROS 2 system

Run ROS 2 nodes on Cerulion’s transport with cerulion ros2 run.

Record and replay a run

Record and re-execute a native Cerulion graph.