Skip to main content

​
Troubleshooting

This guide helps you diagnose and fix common issues when using Cerulion RCL Hooks. If you encounter problems, start here before diving into deeper debugging.

​
Verifying Library Loading

The first step in troubleshooting is confirming that the hooks library is actually being loaded by your ROS process.
1

Check with lsof

Use lsof to see if the library is loaded in your process:
lsof -p $(pgrep -f "ros2 run") | grep cerulion_rcl_hooks
You should see a line showing the library file is open. If you don’t see it, the library isn’t loading.
2

Use LD_DEBUG (advanced)

For detailed library loading information:
LD_DEBUG=libs LD_PRELOAD=/path/to/libcerulion_rcl_hooks.so \
ros2 run <package> <node> 2>&1 | grep cerulion
This shows all library loading activity, which can help identify loading issues.
3

Verify file path

Make sure the path in LD_PRELOAD is correct:
ls -lh /home/cerulion/cerulion-dev/target/debug/libcerulion_rcl_hooks.so
If the file doesn’t exist, you need to build it first (see Installation Guide).
4

Check file permissions

Ensure the library file is readable:
chmod +r /path/to/libcerulion_rcl_hooks.so

​
Common Issues and Solutions

Symptoms: ROS node runs, but you don’t see any [Cerulion Hook] messages.Possible causes:
  • Library not loading
  • Wrong path in LD_PRELOAD
  • Library built incorrectly
Solutions:
  1. Verify the library path is absolute and correct
  2. Check that the library file exists and is readable
  3. Try rebuilding the library: cargo build -p cerulion_rcl_hooks
  4. Use lsof to confirm the library is loaded
Symptoms: You see [Cerulion Hook] Serialization failed... in the logs (note: this refers to schema processing, not actual serialization since Cerulion doesn’t serialize).Possible causes:
  • Missing schema for the message type
  • Schema doesn’t match message structure
  • Incompatible field types
Solutions:
  1. Check if a schema exists in schemas_ros2/ for your message type
  2. Verify the schema matches your ROS message definition
  3. Check the error message for specific field that failed
  4. Create or update the schema file if needed
When processing fails, messages are hijacked and do not continue to ROS subscribers. Make sure you have the correct schema configured for your message type to ensure messages are properly published to Cerulion.
Symptoms: Error messages about missing libraries or symbols.Possible causes:
  • ROS 2 libraries not in library path
  • Missing dependencies
  • Architecture mismatch (32-bit vs 64-bit)
Solutions:
  1. Source your ROS environment: source /opt/ros/<distro>/setup.bash
  2. Check library dependencies: ldd libcerulion_rcl_hooks.so
  3. Ensure ROS 2 development packages are installed
  4. Verify you’re using the correct architecture build
Symptoms: Hook messages appear, but topics don’t show up in Cerulion.Possible causes:
  • Cerulion transport not running
  • Network connectivity issues
  • Topic filtering in hooks
  • Schema issues preventing publication
Solutions:
  1. Verify Cerulion transport is running and accessible
  2. Check network connectivity between ROS node and Cerulion
  3. Review hook logs for schema processing errors
  4. Confirm the topic has a matching schema
  5. Check Cerulion logs for connection or reception errors
Symptoms: ROS node runs slower with hooks enabled.Possible causes:
  • Using debug build in production
  • Large message sizes
  • Network latency to Cerulion
  • Note: Cerulion doesn’t serialize (unlike ROS 2 which continues to serialize locally), so serialization overhead is not a factor
Solutions:
  1. Use release build: target/release/libcerulion_rcl_hooks.so
  2. Check network latency to Cerulion endpoint
  3. Monitor system resources (CPU, memory, network)
  4. Consider filtering topics if you don’t need all of them
Debug builds are 2-10x slower than release builds. Always use release builds for production.
Symptoms: ROS node crashes when hooks are enabled.Possible causes:
  • Library version mismatch
  • Incompatible ROS version
  • Memory corruption
  • Stack overflow
Solutions:
  1. Verify ROS 2 version compatibility
  2. Rebuild hooks library from clean state: cargo clean && cargo build
  3. Check for core dumps: dmesg | tail or journalctl -xe
  4. Try with a different ROS node to isolate the issue
  5. Report the issue with stack traces if possible
Crashes are serious - make sure to report them with full error messages and stack traces if possible.

​
Debugging Techniques

​
Enable Verbose Logging

The hooks use eprintln! for all diagnostics, which goes to stderr. To capture everything: 
LD_PRELOAD=/path/to/libcerulion_rcl_hooks.so \
ros2 run <package> <node> 2>debug.log
Then examine debug.log for detailed information.

​
Verify Library SHA

To confirm you’re using the correct library version: 
sha256sum /path/to/libcerulion_rcl_hooks.so
Compare this with the expected hash from your build.

​
Test with Simple Node

If you’re having issues, test with a simple ROS node first: 
LD_PRELOAD=/path/to/libcerulion_rcl_hooks.so \
ros2 run demo_nodes_cpp talker
This helps isolate whether the issue is with your specific node or the hooks in general.

​
Check ROS Environment

Ensure your ROS environment is properly set up: 
echo $ROS_DISTRO
ros2 --help
If these don’t work, source your ROS setup file.

​
Understanding Error Messages

​
Schema Processing Errors

When you see processing failures (the log message says “Serialization failed” but Cerulion doesn’t actually serialize), the message usually indicates:
  • Missing field: A field in the schema doesn’t exist in the message
  • Type mismatch: Field type doesn’t match between schema and message
  • Nested message issue: Problem with a nested message structure
Look at the specific field mentioned in the error to fix the schema.

​
Library Loading Errors

Common library loading issues:
  • “cannot open shared object file”: File path is wrong or file doesn’t exist
  • “undefined symbol”: Missing dependency or version mismatch
  • “wrong ELF class”: Architecture mismatch (32-bit vs 64-bit)

​
Network Errors

If Cerulion can’t be reached:
  • Check Cerulion transport is running
  • Verify network connectivity
  • Check firewall rules
  • Confirm endpoint configuration

​
Getting Help

If you’ve tried the solutions above and still have issues:
  1. Collect information:
    • Full error messages
    • Hook log output
    • ROS version and distribution
    • Library build information
    • Steps to reproduce
  2. Check existing issues: Look for similar problems in issue trackers or forums
  3. Report the issue: Include all collected information when reporting

​
Prevention Tips

Use Release Builds

Always use release builds in production for better performance

Test First

Test with simple nodes before using with complex applications

Monitor Logs

Regularly check hook logs to catch issues early

Keep Updated

Rebuild hooks when updating ROS or Cerulion versions

​
Next Steps

If you’ve resolved your issue or want to learn more:
  • Review the Overview for conceptual understanding
  • Check the Usage Guide for best practices
  • Learn about adding custom schemas for your message types