> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cerulion.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Replay a ROS 2 process

> Record what one ROS 2 process consumed over rmw_cerulion, replay that process alone against the recording, and read the verdict.

A ROS 2 node misbehaved once, in a run you cannot reproduce. Record what each process took in, in the order it took it, then replay that one process by itself, as often as you like, under a debugger or with extra logging, and get a verdict on whether it did the same thing again.

This works for ROS 2 processes running on Cerulion's transport (`rmw_cerulion`, see [Bridge a ROS 2 system](https://docs.cerulion.com/cerulion/guides/bridge-ros2.md)). The replayed process runs its own unmodified code. It gets every message, service request and response and `/clock` tick it got live, in the same order, and nothing else. The comparison checks each callback's outputs against the recording.

> [!NOTE]
>
> `cerulion ros2 run` and `ros2 launch` need Linux: ROS 2 Jazzy publishes no macOS binaries. Replay is runtime-tested on ROS 2 Humble, with rclpy nodes, rclcpp nodes on ROS time and the stock `demo_nodes_cpp` executables; Foxy, Jazzy and Lyrical builds compile the same code.

## Record

Put `--record-deliveries <DIR>` right after `run` or `launch`; everything after it goes to `ros2` as usual:

```bash theme={null}
cerulion ros2 run --record-deliveries ./logs my_pkg my_node --ros-args -p use_sim_time:=true
cerulion ros2 launch --record-deliveries ./logs my_robot bringup.launch.py
```

Each process writes its own delivery log, `./logs/<exe>-<pid>.rdl.mcap`. An existing file is never overwritten: a second process with the same name and pid writes `<exe>-<pid>.1.rdl.mcap`. To record a process launched some other way, set the variable instead:

```bash theme={null}
CERULION_RMW_DELIVERY_LOG=./logs RMW_IMPLEMENTATION=rmw_cerulion ros2 run my_pkg my_node
```

A delivery log is not a topic recording. `cerulion bag record` records the frames on topics; a delivery log records what one process took from them, in its order. Both can run at once.

> [!WARNING]
>
> Recording costs time on the calling thread: each take copies its payload into the log and each output is hashed. The cost grows with message size, and the log grows by every payload taken. Record processes whose large topics run at a few Hz.

## Replay

Replay one recorded process, alone, with the same command it was recorded with:

```bash theme={null}
cerulion ros2 run --replay-deliveries ./logs/my_node-4242.rdl.mcap my_pkg my_node --ros-args -p use_sim_time:=true
```

Nothing else needs to run: no talker, no clock, no robot. `ros2 launch` takes no replay flag, because a replay is one process. Setting `CERULION_RMW_REPLAY=<log>` does the same on any launch path.

When the replay starts, `<log>.verdict.json` says `running`; the real verdict replaces it once the log is used up. Stop the process with `Ctrl+C` after the file no longer says `running`. A replay that crashes or is killed leaves `running`.

> [!WARNING]
>
> A replay really publishes. The replayed process sends its outputs on the transport as it did live, and through `cerulion-netd` they can reach other hosts. Never replay a controller or a driver where a live robot, or anything that acts on its topics, can receive them: replay on a machine of its own, or on one where no live node runs and `cerulion-netd` is stopped or started with `CERULION_NETD_NETWORK=off`.

## Read the verdict

| `verdict` | Meaning |
| - | - |
| `identical` | Every compared output equals the recording, every recorded window was served and every recorded delivery was taken. |
| `diverged` | `culprit` names the first segment whose inputs matched and whose outputs differ: the node, topic and window. |
| `input_mismatch` | The replayed process asked for an input in an order the recording does not have. |
| `log_exhausted_early` | The log did not close cleanly (the recorded process was killed) and the replay matched up to where it ends. |
| `replay_stopped_early` | The process exited before it was served every recorded window. |
| `watchdog` | A callback held the replay for 5 s without progress, waiting on another thread the replay cannot run. |
| `refused` | The process is not replayable; `refused` says why. |
| `running` | The replay has not written its verdict yet, or it crashed or was killed. |

A divergence is also printed on stderr when the verdict is written, naming the process, node, topic and window. `/rosout` and the wall-clock stamp in a message header are never compared.

## When a replay is exact

A replay reproduces the recorded run when all of these hold for the process:

* Every thread that runs callbacks is a single-threaded executor thread. Several single-threaded executors on several threads are fine; a multi-threaded executor is refused by name.
* No node logic reads the wall or steady clock. Use ROS time (`use_sim_time`, with `/clock` recorded); a wall timer can diverge, and when it changes an output the verdict names it.
* Random numbers are seeded or absent.
* No callback blocks on work another thread owns.
* No input reaches the node outside the rmw: no sockets, devices, files it reads during the run, or shared memory of its own.
* The same binaries, parameters and distro are used. A log recorded by another command line, `ROS_DISTRO` or build of `librmw_cerulion.so` is refused at start, naming the field.

What a pass does not prove: an `identical` verdict shows that this recording replays, not that the node is deterministic, and a race that did not fire while recording stays hidden. Each process replays alone against its own log, so processes that feed each other are not replayed together. `EventsExecutor`, rclcpp intra-process communication and loaned takes are not covered; a replay turns loans off.

## Next steps

> [card] [Bridge a ROS 2 system](https://docs.cerulion.com/cerulion/guides/bridge-ros2.md)
>
> Run ROS 2 nodes on Cerulion's transport with `cerulion ros2 run`.

> [card] [Record and replay a run](https://docs.cerulion.com/cerulion/guides/record-and-replay.md)
>
> Record and re-execute a native Cerulion graph.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.