Skip to main content

​
Troubleshooting

This guide helps you diagnose and fix common issues when using the Cerulion Visualization bridge. If you encounter problems, start here before diving into deeper debugging.

​
Verifying the Bridge is Running

The first step in troubleshooting is confirming that the bridge server is actually running and accessible.
1

Check if the server is listening

Verify the WebSocket server is listening on the expected port:
netstat -an | grep 9090
# or on Linux
ss -tlnp | grep 9090
# or using lsof
lsof -i :9090
You should see the port is in LISTEN state. If not, the server isn’t running.
2

Check bridge process

Verify the bridge process is running:
ps aux | grep cerulion | grep bridge
You should see the bridge process listed.
3

Test WebSocket connection

Test the WebSocket endpoint with a simple client:
# Using wscat (if installed)
wscat -c ws://localhost:9090

# Or using curl (for basic HTTP check)
curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" http://localhost:9090
4

Check bridge logs

Look at the terminal where you started the bridge. You should see:
  • Server startup messages
  • Topic discovery output
  • Client connection logs

​
Common Issues and Solutions

Symptoms: Error message like “Address already in use” or “Port 9090 is already bound”.Possible causes:
  • Another process is using port 9090
  • Previous bridge instance didn’t shut down cleanly
  • Another WebSocket server is running
Solutions:
  1. Find what’s using the port: 
    lsof -i :9090
# or
netstat -tulpn | grep 9090
  2. Kill the process using the port, or use a different port: 
    cargo run --bin cerulion --features bridge -- bridge --port 9091 --transport iox2
  3. Then connect Lichtblick to ws://localhost:9091 instead
Symptoms: Bridge starts but shows “Found 0 topics” or no topics appear in Lichtblick.Possible causes:
  • No topics are actually publishing
  • Wrong transport mode selected
  • Topics published via different transport than bridge mode
  • Transport not properly configured
Solutions:
  1. Verify topics are actually publishing:
    • For iox2: Check iceoryx2 service registry
    • For zenoh: Check zenoh network for topics
  2. Ensure transport mode matches:
    • If topics use iox2, bridge must use --transport iox2
    • If topics use zenoh, bridge must use --transport zenoh
  3. Check that your Cerulion nodes are running and publishing
  4. Verify transport configuration (iox2 running, zenoh network accessible)
The bridge can only discover topics from the transport mode you’ve selected. Make sure it matches how your topics are published.
Symptoms: Lichtblick shows “Connection refused” or can’t connect to the bridge.Possible causes:
  • Bridge server isn’t running
  • Wrong port number
  • Firewall blocking connection
  • Wrong hostname/IP address
Solutions:
  1. Verify bridge is running (see “Verifying the Bridge is Running” above)
  2. Check the port matches:
    • Bridge: --port 9090 (or whatever you specified)
    • Lichtblick: ws://localhost:9090 (same port)
  3. If connecting remotely, check:
    • Bridge is bound to 0.0.0.0 (not just 127.0.0.1)
    • Firewall allows connections on the port
    • Correct IP address or hostname
  4. Test with a simple WebSocket client first
Symptoms: Topics show up in Lichtblick, but subscribing shows no data or messages.Possible causes:
  • Topics aren’t actually publishing data
  • Schema conversion issues
  • Message format problems
  • Subscription not working
Solutions:
  1. Verify topics are actively publishing:
    • Check your Cerulion node logs
    • Verify data is flowing in the transport
  2. Check bridge logs for errors:
    • Look for serialization/conversion errors
    • Check for subscription failures
  3. Verify schemas exist for message types:
    • Without schemas, messages may not convert properly
    • Check schema registry configuration
  4. Test with a simple topic first to isolate the issue
Symptoms: Bridge starts with iox2 mode but finds no topics or can’t connect.Possible causes:
  • Iceoryx2 not running
  • Iceoryx2 service registry not accessible
  • Topics not published via iox2
  • Permission issues
Solutions:
  1. Verify iceoryx2 is running: 
    # Check iceoryx2 processes
ps aux | grep iceoryx
  2. Check iceoryx2 service registry:
    • Ensure iceoryx2 daemon is running
    • Verify service registry is accessible
  3. Confirm topics are published via iox2:
    • Check your Cerulion node configuration
    • Verify transport mode in node setup
  4. Check permissions:
    • Ensure bridge has access to iceoryx2 shared memory
    • Verify user permissions
Symptoms: Bridge starts with zenoh mode but can’t connect or find topics.Possible causes:
  • Zenoh network not accessible
  • Zenoh configuration issues
  • Network connectivity problems
  • Zenoh routers not running
Solutions:
  1. Verify zenoh network connectivity:
    • Check zenoh configuration
    • Ensure zenoh routers are accessible
    • Test zenoh connectivity separately
  2. Check network settings:
    • Firewall rules
    • Network routing
    • DNS/hostname resolution
  3. Verify zenoh session:
    • Check zenoh session configuration
    • Ensure zenoh libraries are properly installed
  4. Test zenoh independently:
    • Use zenoh tools to verify connectivity
    • Check if topics are visible via zenoh directly
Zenoh topic discovery is currently a placeholder. You may need to manually subscribe to known topics in Lichtblick.
Symptoms: Topics work but messages appear as hex-encoded data instead of structured JSON.Possible causes:
  • No schema exists for the message type
  • Schema not found or not loaded
  • Schema format incorrect
Solutions:
  1. Check if schema exists for your message type:
    • Look in schema registry
    • Verify schema file exists
  2. Verify schema is loaded:
    • Check bridge logs for schema loading errors
    • Ensure schema registry path is correct
  3. Create or update schema:
    • Add schema for your message type
    • Ensure schema format matches Cerulion schema specification
Without schemas, the bridge can’t convert messages to JSON, so it returns hex-encoded data. This is expected behavior when schemas are missing.
Symptoms: Bridge starts but then crashes or exits without error message.Possible causes:
  • Panic in Rust code
  • Missing dependencies
  • Transport connection failure
  • Resource exhaustion
Solutions:
  1. Check for panic messages:
    • Look for “thread panicked” in output
    • Check system logs: dmesg | tail or journalctl -xe
  2. Run with debug output: 
    RUST_BACKTRACE=1 cargo run --bin cerulion --features bridge -- bridge --transport iox2
  3. Check system resources:
    • Memory usage
    • File descriptor limits
    • Network connection limits
  4. Verify dependencies:
    • All required libraries installed
    • Transport libraries (iox2/zenoh) working
  5. Try with release build: 
    target/release/cerulion bridge --transport iox2

​
Diagnostic Commands

​
Check Bridge Status

# Is the bridge running?
ps aux | grep "cerulion.*bridge"

# Is the port listening?
lsof -i :9090
# or
netstat -tulpn | grep 9090

​
Test WebSocket Connection

# Using wscat
wscat -c ws://localhost:9090

# Using curl (basic test)
curl -i -N \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Key: test" \
  http://localhost:9090

​
Check Transport Status

For iox2: 
# Check iceoryx2 processes
ps aux | grep iceoryx

# Check iceoryx2 shared memory
ipcs -m | grep iceoryx
For zenoh: 
# Check zenoh connectivity (if zenoh CLI available)
zenoh-bench --help

​
Verify Topics Exist

Check if topics are actually publishing in your transport:
  • For iox2: Use iceoryx2 tools or check service registry
  • For zenoh: Use zenoh tools to list topics

​
Understanding Error Messages

​
”Address already in use”

The port is already bound by another process. Use a different port or stop the conflicting process.

​
”Failed to bind”

The bridge couldn’t bind to the port. Check:
  • Port permissions
  • Another process using the port
  • Firewall blocking the port

​
”No topics discovered”

No topics found in the selected transport. Verify:
  • Topics are actually publishing
  • Transport mode matches topic transport
  • Transport is properly configured

​
”Connection refused”

Client can’t connect to bridge. Check:
  • Bridge is running
  • Correct port number
  • Network connectivity
  • Firewall rules

​
”Serialization failed”

Message couldn’t be converted to JSON. Usually means:
  • Missing schema
  • Schema format incorrect
  • Message structure mismatch

​
Performance Issues

If the bridge is slow or using too many resources:
  1. Use release build: Debug builds are slower 
    target/release/cerulion bridge --transport iox2
  2. Limit topics: If possible, only subscribe to needed topics
  3. Check system resources: Monitor CPU, memory, network
  4. Reduce message frequency: If topics publish very frequently, consider rate limiting

​
Getting Help

If you’ve tried the solutions above and still have issues:
  1. Collect information:
    • Bridge startup command and output
    • Full error messages
    • Transport mode and configuration
    • System information (OS, Rust version)
    • Steps to reproduce
  2. Check logs:
    • Bridge server output
    • System logs
    • Transport-specific logs (iox2/zenoh)
  3. Report the issue: Include all collected information when reporting

​
Prevention Tips

Use Release Builds

Always use release builds in production for better performance

Match Transport Modes

Ensure bridge transport mode matches how topics are published

Verify Before Connecting

Check topics are publishing before starting the bridge

Keep Schemas Updated

Ensure schemas exist for message types you want to visualize

​
Next Steps

If you’ve resolved your issue or want to learn more: