DevoxxFR 2024 Reproducible Builds with Apache Maven
ROS 2 Foxy with Eclipse Cyclone DDS | Philly ROS Meetup July 20th 2020
1. ROS 2 Foxy with Eclipse Cyclone DDS
makes robots Easier, Smaller, Faster
Philly ROS Meetup July 20th 2020
bit.ly/eclipsecyclone for this presentation and links
Joe Speed
@JoeSpeeds
joespeed
jspeed12
Erik Boasson
too busy for twitter
erik-boasson-21344912
eboasson
@EclipseCyclone
3. Leading Edge COMPUTING adlinktech.com
ROS1 created in 2007 by Willow Garage as a platform for robotics
research
• Moved into stewardship by the Open Source Robotics Foundation
• Vibrant ecosystem of tools, algorithms and drivers for constructing robots
• Much used for research, education, prototypes
ROS2 is a rewrite of ROS1 driven by Open Robotics, first released 2017
• Addresses the weaknesses in the architecture of ROS1
• Maturing nicely, ecosystem is migrating to it
• Already the choice for mobile robotics, gaining ground in industrial robotics
The briefest of recaps: ROS1 & ROS2
4. Leading Edge COMPUTING adlinktech.com
- 115 engineers, equivalent to $8M
- Completeness
- Quality
- Performance
- Security
- Navigation
- No more releases of ROS “classic”
- aws.amazon.com/blogs/robotics/ros-2-foxy-fitzroy-robot-development
ROS 2 Foxy - Now is the time to switch
7. Leading Edge COMPUTING adlinktech.com
“Eclipse Cyclone DDS is a superior architecture”
- Automotive CTO
• This open source contributed DDS is very special
• Makes ROS 2 robots fast, small, reliable
• Tier-1 ROS middleware built into ROS 2 Foxy
• Open Robotics is evaluating for ROS 2 default
• Built into Autoware autonomous driving
• Industry is helping it become real-time & safe
bit.ly/eclipsecyclone
10. Leading Edge COMPUTING adlinktech.com
a) Eclipse IoT open source
project
b) very performant DDS
implementation
c) Easy, Small and Fast
d) Loves github issues & PRs!
e) Led by IoT mad scientist &
2nd gen DDS inventor Erik
Boasson eboasson
What is Eclipse Cyclone DDS?
11. Leading Edge COMPUTING adlinktech.com
Pure C code
Run-time library has minimal dependencies
Compact
• code size can go down to ~0.5MB
• latency testing needs ~0.5MB of heap memory
(macOS)
• Configuring for minimal footprint does reduce
throughput (~0.5x) & increase latency a bit (~5µs)
Platforms
• Supported: Linux, Windows, macOS, FreeRTOS,
x86, Arm, Jetson
• Planned: QNX, possibly VXWorks and others
• Works: Solaris 2.6, OpenIndiana (Erik & Morgan are sentimental)
Footprint of Eclipse Cyclone DDS
12. Leading Edge COMPUTING adlinktech.com
tested the RMWs,
selected Eclipse Cyclone DDS
“The (ROS 2) RAM usage goes from 150MB to 15MB and we basically
had no late and no lost messages. This is a huge improvement in
terms of performance simply changing the DDS implementation”
13. Problem Statement
~1000 robots in the same network
~35 topics from each robot
~5 different subnetworks across the globe
We want to be able to individually access any robot's
messages from a remote debugging machine
14. Addressing Scalability
Proposed solution 1:
Use default DDS configuration and a different namespace for
each robot
Problems:
ALL robots discover each other!
Slow/unreliable discovery, network saturation, multicast requires
forwarding between subnets
15. Addressing Scalability
Proposed solution 2:
Solution 1 + ROS_DOMAIN_ID
Problems:
Limited numbers of domain IDs. What about assignment?
Still considerable network usage and multicast limitations
16. Addressing Scalability
Proposed solution 3:
Disable multicast discovery.
When a remote machine wants to connect to a robot, add the
robot’s IP to the initial peers list (unicast)
Problems:
The user needs direct access to the DDS configuration. This can
be solved by new APIs.
✓
17. Solution 3 with CycloneDDS
ROBOT_IP=10.22.22.90 ros2 topic list
✓
18. Leading Edge COMPUTING adlinktech.com
Erik eboasson added domainTag to DDSI 2.3 spec
and implemented iRobot’s use case in
CycloneDDS: Find Roomba by serial # among
~1,000 robots on network
DDSI domainTag things talk if domain + tag match
//CycloneDDS/Domain/Discovery/Tag
String extension for domain id that
remote participants must match to be
discovered. The default value is: "”
Solution 4 with Eclipse Cyclone DDS
XYZ270011101230600001
✓
19. Leading Edge COMPUTING adlinktech.com
github.com/irobot-ros/ros2-performance
on your RPi2B or other machine
on The Construct’s ROS Dev Studio http://www.rosject.io/l/ebd6221/
Try iRobot’s test with Eclipse Cyclone DDS
26. Suspension Analysis / Tire Modeling
CAN Bus over LTE
Intel RealSense
Embedded ML
ADLINK ROScube Jetson Xavier
27. Demo - Vehicle Dynamic (AI Driver Model)
trained driver model, can currently
uses a combination of neural
networks to predict with 85%
accuracy:
• Steering Angle
• Brake Position
• Throttle Position
• Gear and Speed
28. Leading Edge COMPUTING adlinktech.com
AMRs with Eclipse Cyclone DDS
mobile vision
“camera on wheels”
Education
Inspection
100kg payloads
31. Leading Edge COMPUTING adlinktech.com
“Eclipse Cyclone DDS has given the fast
& precise control of our ROS 2 baseball
pitching robot to accurately recreate
any throw from any pitcher”
- Rowan Ferrabee
CTO Trajekt Sports
32. Leading Edge COMPUTING adlinktech.com
Eclipse Cyclone DDS is built into
components of military vehicles with
32-bit microcontrollers
Intra-vehicle communication!
Inter-vehicle communication?
Couldn’t say who, so here is a
picture of an old jeep
microcontrollers running Eclipse Cyclone DDS
source: Franklin D. Roosevelt Presidential Library
and Museum, Hyde Park, New York.
34. Leading Edge COMPUTING adlinktech.com
”multicast autotune” – not
really, but sort of !"
Sped Nav2 WiFi bringup
while others learned how
“just works” in hospital with
terrible multipath issues, no
config needed
WiFi happy Eclipse Cyclone DDS
image source: edmtips.com/how-to-autotune-properly
35. Leading Edge COMPUTING adlinktech.com
<images from Pack Expo>
Worked reliably in Last Vegas Convention Center
despite 666 WiFi APs “a devilish number” !
36. Leading Edge COMPUTING adlinktech.com
Vendor benchmarks?!?
Source dharmapunxnyc.blogspot.com/2013/08/dont-believe-hype.html
37. Leading Edge COMPUTING adlinktech.com
apt install ros-foxy-rmw-cyclonedds-cpp
(or eloquent)
(or dashing)
RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
bit.ly/eclipsecyclone for this presentation and links
iRobot ROS 2 performance ROScon `19 slides
iRobot ROS 2 performance test tool for mobile robots
ApexAI/performance_test RMW and DDS evaluation tool
ros2/rmw_cyclonedds try it
39. Eclipse Cyclone DDS
• Full-featured DDS implementation
• QoS support: nearly complete
• Content filtering/querying: planned
• Durable data: work-in-progress, partial support later this year
• DDS Security: available in master, to be released shortly
• DDS XTypes: underway, support for key features expected in the fall
40. Eclipse Cyclone DDS
• Fast(est) and small(est) DDS
• it was the fastest(*) by a good margin, then the others woke up
• (*) disputed by the other DDS implementors, certainly no longer true in all scenarios
• The only C-based open-source DDS implementation
• Smaller run-time footprint, fewer platform dependencies
• People are using Cyclone on FreeRTOS on a ARM Cortex-M4 with a only few MB of memory
• ADLINK’s OpenSplice DDS is in C, but support for it has been dropped in Foxy
41. Cyclone DDS in ROS 2
• Tier 1 since Foxy release — Yay!
• DDS Security and some performance improvements to be released shortly in
Foxy via rolling update
• Actively tracks ROS 2 development
• e.g., first RMW to support incompatible QoS notifications
• More often than not, test failures turn out to be wrong assumptions in ROS test
code
• Some designs prompted by Fast-RTPS limitations are bad for Cyclone DDS
46. Challenges for DDS in robotics
• A very different environment than where DDS grew up
• more dynamic (roving robots entering a cage of Faraday such as an elevator
— things like that)
• dependence on really large messages: point clouds of 10MB are not uncommon,
and DDS people used to think 100kB was large!
• people focussed on developing the robots, rather than on pleasing the middleware
• often running everything inside a single machine, rather than over the network
• … and if there’s a network, it is often Wi-Fi
47. Moving forward: throughput & latency
• Good progress on transparent use of shared memory
• integrating Eclipse Iceoryx
• we’re not interested in a “shared memory transport”: that only gives you maybe
20%
• we’re interested in using the shared memory as a store of messages
• allows zero-copy communication
• pure shared memory within the machine for many cases
• automatic fall-back to loopback if QoS cannot otherwise be supported
48. Moving forward: scalability
• DDS discovery can be a burden, especially over Wi-Fi
• Only real solution is to not discover everything
• We’re developing a discovery service that allows discovering only matching endpoints
• This is the second version, building on Eclipse Zenoh (zenoh.io)
• The first version was in Java and used, e.g., in Singapore for connecting smart lampposts
• Will also provide transparent bridging of DDS across networks
• Disclosure: I am one of the originators of Zenoh, this work is being done by friends in Paris
49. Moving forward: multiple networks
• Cyclone DDS today operates over a single network interface
• Will be adding support for operating over multiple interfaces simultaneously
• easier out-of-the-box use with containers, multiple interface, &c.
• support for load-balancing
• support for redundant networking
51. Maximising throughput
• Default config doesn’t max out UDP datagram sizes
• more overhead, less to retransmit on loss
• loopback benefits from larger messages, GbE doesn’t
• ≤ 0.6 default: 4kB, current master: 15kB
• Tweak: General/MaxMessageSize
• limit to 65500B or it won’t always fit in a UDP datagram
• (macOS, master, Linux is qualitatively similar; some retransmitting because of buffer overruns)
10MB sample
100kB sample
1kB sample
10 20 30 40 50
msg size [kB]
5
10
15
20
throughput [Gb/s]
52. Maximising throughput
• UDP socket receive buffer sizes do matter
• overrunning the buffer means retransmits (reliable) / lost messages (best-effort)
• Defaults
• Linux defaults to limiting to ~400kB, macOS to ~800kB; Cyclone to 1MB
• even without tweaking, it works well (some retransmits in preceding graph)
• Rule of thumb: make it large enough to easily contain the largest sample
• echo 33554432 > /proc/sys/net/core/rmem_max
• Internal/SocketReceiveBufferSize = 30M
• Better yet: don't send such large samples!
53. Multicast vs unicast
• If multicast enabled, Cyclone uses it when > 1 receiver IP address
• This is not configurable, but is something under investigation
• 1 receiver IP address: always unicast
• Multicast is enabled by default, except when a Wi-Fi interface is selected
• On Wi-Fi multicast well enough for initial node discovery, but not much else
• Ubuntu loopback interface claims to not support multicast (“ifconfig lo multicast”
enables it)
• Topics can be mapped to multicast addresses in configuration
54. Multicast (2)
• General/AllowMulticast configures whether it is allowed:
• false: disallowed
• spdp: allowed only for initial participant/node discovery
• true: enabled
• Discovery/SPDPMulticastAddress is discovery address (default: 239.255.0.1)
• also default data multicast address, override with Discovery/DefaultMulticastAddress
• if IGMP is used, multicasts only reach nodes subscribed to that IP address
• if you want no interference, pick a different multicast address: port numbers are ignored in
routing
55. Multicast (3)
• Topics (with wildcards) can be mapped to different multicast addresses in config:
<Partitioning>
<NetworkPartitions>
<NetworkPartition Name="ssm" Address="232.3.1.4"/>
<NetworkPartition Name="x" Address=“224.225.1.4"/>
<NetworkPartition Name="y" Address="224.226.0.4;8;4;${NODEIDX:-0}"/>
</NetworkPartitions>
<PartitionMappings>
<PartitionMapping DCPSPartitionTopic=“rt/camera*" NetworkPartition="ssm"/>
<PartitionMapping DCPSPartitionTopic=“*pointcloud*” NetworkPartition="y"/>
<PartitionMapping DCPSPartitionTopic="*.*" NetworkPartition="x"/>
</PartitionMappings>
</Partitioning>
• source-specific multicast (SSM) is supported, the “y” one needs a bit more explanation …
56. Multicast (4)
• 224.226.0.4;8;4;${NODEIDX:-0} means:
• use 224.226.0.4 as a base address
• use the 4 bits at positions 8 .. 11 for identifying individual nodes
• expand NODEIDX environment variable to get the index of this node (0 if not set)
(environment variable expansion is performed throughout the configuration)
• gets dynamically expanded to address just the subscribed nodes (O(2n) groups!)
• switches can typically handle up to 7 or 8 nodes this way before flooding the network
• e.g.:
• if 0 and 1 are subscribed, a multicast is sent to 224.226.3.4
• if 1 and 3 are subscribed, a multicast is sent to 224.226.6.4
57. Unicast discovery
• If multicast disabled for initial node discovery
• set Discovery/ParticipantIndex = auto
• add machines to ping in Discovery/Peers, e.g.,:
<Peer address=“apple”/>
<Peer address=“banana”/>
<Peer address=“cherry”/>
<Peer address=“durian”/>
• The “Participant index” bit is not very user-friendly …
58. Retransmit on slow networks
• Taken from PR#558
• Macbook Pro to a (cheap) AP then via 100Mb/s Ethernet to an RPi3, out-of-the-box
• Packet loss at 100kB and 1MB messages but negligible received duplicates (brown)
• Retransmits explode at 10MB messages, with significant duplication
• caused by retransmits often taking more than 100ms to fill the gap
• increasing Internal/NackDelay helps (it never delays the 1st retransmit)
59. Retransmit on slow networks (2)
• For laughs:
• same setup but good AP
• RPi3 can’t cope, Fast Ethernet is too slow for the WiFi
• that is, a very large part of the packets gets dropped in the network
• publisher uses more bandwidth for retransmits than data!
• I don’t know how to tune Cyclone for this …
100 1k 10k 100k 1M 10M
0 100 200 300 400 500 600
0.10
1
10
100
1000
Mb/s
new ap
60. Updating Cyclone in your ROS 2 build
• A full source build always works, but updating a binary install is possible, too
• use an “overlay workspace”, i.e., create a workspace, clone cyclonedds and
rmw_cyclonedds packages
• do colcon build —cmake-args -DCMAKE_BUILD_TYPE=Release
• “. install/local_setup.bash” to point to the newly built ones
• For “cyclonedds”: use the master branch
• For “rmw_cyclonedds” use the branch matching the ROS 2 release you are using
• for Dashing & Eloquent, that’s the “dashing-eloquent” branch
62. No communication at all
• Usually caused by any of:
• Wrong network interface
• No/flakey multicast support (see a few slides back)
• Firewall blocking traffic
63. No communication: network interface
• General/NetworkInterfaceAddress selects the interface
• specify an interface name
• IP address of the machine
• IP address of the network (i.e., mask out the host part)
• e.g.: with vmnet8: […] inet 172.16.93.1 netmask 0xffffff00, any of
• General/NetworkInterfaceAddress = vmnet8
• General/NetworkInterfaceAddress = 172.16.93.1
• General/NetworkInterfaceAddress = 172.16.93.0
64. Firewall blocking traffic
• There is a rule for computing the port numbers, but it takes a ton of parameters
• https://github.com/eclipse-cyclonedds/cyclonedds/blob/master/docs/manual/
config.rst#controlling-port-numbers
• short version: domain id 0 with multicast:
• 7400 and 7401 are used for multicast, random port numbers used for receiving unicasts
• if you set Discovery/ParticipantIndex to “auto” (same as with unicast discovery!)
• 7410 + 10k and 7411 + 10k used for receiving unicast data, k is the “participant index”
• data is currently always sent from a random port number
65. Sanity checking
• “ddsperf” tool is quite a practical way
• included in the packages installed by ROS 2
• “ddsperf -h” explains the options
“ddsperf sanity” set ups many-to-many ping at 1Hz and is an easy way to check
• “ddsperf pub” publishes small messages as fast as possible
“ddsperf pub 10Hz size 10k burst 100” sends bursts of 100 messages of 1kB each,
at 10 bursts/s
• “ddsperf sub” subscribes and shows the throughput
• several more modes and options …
66. Interference from other users in the network
• Use a different domain id
• supported directly in ROS 2
• doesn’t separate out multicast traffic
• only a limited range of domain ids available (32 bits in theory, but they also map to UDP port numbers)
• Use a different domain tag
• a configurable string included in initial discovery, nodes only communicate if they have the same tag
• doesn’t separate out multicast traffic
• e.g., Discovery/Tag = fruit
• Use a different set of multicast addresses
• be sure to only do that in combination with either (or both) of the above
67. How to change Cyclone settings?
• Set CYCLONEDDS_URI environment variable
• point it to XML config files
• embed configuration snippets in “abbreviated” XML
• e.g.: “CYCLONEDDS_URI=‘<Tr><V>config</><Out>stdout</></>’”
to dump the configuration to stdout on startup
• comma-separated
• Configuration files can specify settings that apply to all domain ids or just to a single one
• this can be useful if you want to use multiple domains in your system
• XSD with descriptions:
• https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd
69. If all else fails …
• Wireshark … but wireshark is not so pleasant …
• Traces
• Crazy level of detail in the traces, but this slows stuff down
• I’ve been told no-one knows how to read them but me …
• Ping me on GitHub :)
70. ROS 2 Foxy with Eclipse Cyclone DDS
makes robots Easier, Smaller, Faster
Philly ROS Meetup July 20th 2020
bit.ly/eclipsecyclone for this presentation and links
Joe Speed
@JoeSpeeds
joespeed
jspeed12
Erik Boasson
too busy for twitter
erik-boasson-21344912
eboasson
@EclipseCyclone