Paolo

general recent conversations

Asimov's Corner

General discussion for robotics.stackexchange.com

Paolo

yst 08:19

@Chuck - many thanks for the insights

Paolo

Mon 07:22

@Chuck Do you have any experience with Nvidia Isaac sim? I couldn't find any multi-robot (or multi-lidar) use-case that give me some guessing about performances of that simualtor

Paolo

Mon 07:19

Hi @Chuck many thanks for the explanation and for sharing your expertise. It will help me a lot to make future decisions

Paolo

Mon 07:18

Hi @ch

Paolo

Mon 07:18

Hi @c

Paolo

Jul 9 06:46

@CroCo, thanks for the reply. Unfortunately, I'm using Gazebo right now, and the problem is not with the visualization itself, but with the rendering of sensors like LiDAR, which requires a GPU (as opposed to Gazebo Classic, which uses the CPU). This is why the simulation slows down significantly when spawning multiple robots. I will check CoppeliaSim, many thanks

Paolo

Jul 9 06:44

@cro

Paolo

Jul 8 07:44

Simulator for multi-robot system or swarm robotics.
Hi everyone!
I’m writing to gather information and opinions on simulators used for multi-robot systems or swarm robotics. Currently, I’m using Gazebo (Harmonic) to emulate TurtleBot 4; however, I can only simulate a few robots before the system crashes. Furthermore, if I want to run learning algorithms (e.g., evolutionary algorithms or reinforcement learning), I need a simulator that is significantly faster than real time and highly parallelizable. I would like to know what other possible solutions exist to address the above-mentioned issu…

Discussion on question by Paolo: Igni…

Imported from a comment discussion on robotics.stackexchange.c...

Paolo

Mar 25 14:09

Hi Martin, I tried to run an apptainer image with Ubuntu24, ROS2 jazzy and Gazebo harmonic (should be the latest LTS release) with the example simulation of the TurlteBot4 robot and the rtf seems way better overall, around ~1.3 (even with some drops to 0.2). The world that I spawned was even more complicated that the one that I'm planning to use for my experiments

Paolo

Mar 24 22:25

Ok Martin. Thank you very much for your help!

Paolo

Mar 24 08:45

So basically we can conclude that everything is working right now.

Paolo

Mar 24 08:45

Hi Martin thanks again for your feedback. On my machine have a RTF that is ~1.0 in my local machine. On the cluster, I have something around ~0.3. In the simulation I have the basic sensors that TurtleBot4 is running: lidar, camera, and infrared (that as far as I can tell from the source code is just an adaptation of a lidar sensor). I tried also to disable the camera but the improvement is not that huge to be fair, like from 0.3 to ~0.4.

Paolo

Mar 22 09:05

ON the cluster there is this one Tesla V100-SXM2-32GB (cluster) and locally I have this NVIDIA GeForce RTX 4070

Paolo

Mar 22 09:00

maybe disable the camera can help?

Paolo

Mar 22 08:49

Anyway, I still don't understand how is it possibile that in the cluster the rtf is slower than in my local machine...

Paolo

Mar 22 08:44

Hi Martin, I checked also the `nvidia-smi` output by running a simulation locally and when I run something inside a container I cannot see `gz-server` but just something like
| 0 N/A N/A 557667 G ...er-engine ogre2 visualize_lidar.sdf 130MiB |
as I do in the cluster. So I guess, the simulation on the cluster is running properly with the GPU.

Paolo

Mar 21 17:58

So if I edit the launch file resposibile to starting gazebo by adding the variable needed I hope I can fix it

Paolo

Mar 21 17:57

And that's why I guess my rtf is low

Paolo

Mar 21 17:57

I see it in the full example with ros2 launch etc. But I cannot see that gz-server is running on nvidia-smi

Paolo

Mar 21 17:53

Anyway, I'm editing the ros_gz package so I can include the commands that I need to run gz in the GPU. Let's see if it works

Paolo

Mar 21 17:52

I'm running a simulation with a TurtleBot4 robot and the scene it's really simple, 4 walls, floor and 7 of simple objects. On my local machine I'm around 1 rtf, sounds strange that here I just have 0.2

Paolo

Mar 21 17:21

I can also see `*** Starting GLX Subsystem ***` and this that seems quite encouraging `17:37:38: **************************************
17:37:38: *** OpenGL 3+ Renderer Started ***
17:37:38: **************************************
17:37:38: Registering ResourceManager for type GpuProgram
17:37:38: RenderSystem capabilities
17:37:38: -------------------------
17:37:38: RenderSystem Name: OpenGL 3+ Rendering Subsystem
17:37:38: GPU Vendor: nvidia
17:37:38: Device Name: Tesla V100-SXM2-32GB/PCIe/SSE2`

Paolo

Mar 21 16:33

I understand

Paolo

Mar 21 16:31

too many comments ahahah

Paolo

Mar 21 16:30

The low rtf happens because the gz-server is not executed in the GPU. I checked with nvidia-smi, if I run xvfb-run -a vglrun +v -d /dev/dri/card1 ign gazebo -v 4 -r --render-engine ogre2 visualize_lidar.sdf I can see gz-server with nvidia-smi

Paolo

Mar 21 16:30

I run the simulation with the commands listed in Update 6, but the rtf is still too low. How can I be ensure that Gazebo uses GPU now that the container can 'see' it?

Paolo

Mar 21 16:30

Just to check if I understood, if I run

apptainer shell --bind /usr/lib64/libEGL_nvidia.so.0 --bind /usr/lib/libEGL_nvidia.so.0 --bind /usr/share/glvnd/egl_vendor.d/ --nv  apptainer_images/ROS2Humble_GazeboFortress_GPU

and I can see the output of xvfb-run -a vglrun +v -d /dev/dri/card1 glxgears -infolike this

[VGL] WARNING: Could not set WM_DELETE_WINDOW on window 0x00200002 GL_RENDERER   = Tesla V100-SXM2-32GB/PCIe/SSE2 GL_VERSION    = 4.6.0 NVIDIA 550.54.15 GL_VENDOR     = NVIDIA Corporation ...

means that I can run my simulation or there is something else that I need to check?

Paolo

Mar 21 16:30

My bad, if I use

apptainer shell --bind /usr/lib64/libEGL_nvidia.so.0 --bind /usr/lib/libEGL_nvidia.so.0 --bind /usr/share/glvnd/egl_vendor.d/ --nv  apptainer_images/ROS2Humble_GazeboFortress_GPU

gives me the same output as outside

Paolo

Mar 21 16:30

Anyway, inside the container, xvfb-run -a vglrun +v -d /dev/dri/card1 glxgears -info gives me the usual error, [VGL] ERROR: in init3D-- [VGL] 245: Invalid EGL device [VGL] Shared memory segment ID for vglconfig: 6553607

Paolo

Mar 21 16:30

Ok I managed to fix it. If I run the command (on the GPU node, outside the container) xvfb-run -a vglrun +v -d /dev/dri/card0 glxgears -info I get the usual error. If I switch to /dev/dri/card1 I can see GL_RENDERER = Tesla V100-SXM2-32GB/PCIe/SSE2 GL_VERSION = 4.6.0 NVIDIA 550.54.15 GL_VENDOR = NVIDIA Corporation

Paolo

Mar 21 16:30

If I run apptainer shell with the binding of the folder that I get when I execute strace -f -e trace=open,openat nvidia-smi 2>&1 | grep nvidia outside the container and, inside the container I run xvfb-run -a vglrun +v -d /dev/dri/card0 glxgears -info, I get the following error

[VGL] Shared memory segment ID for vglconfig: 6520841 [VGL] VirtualGL v3.1.2 64-bit (Build 20241219) Error: couldn't open display :116

. Don't know if might help

Paolo

Mar 21 16:30

ok I did it and I can see the usual output from glxgears -info by running xvfb-run -a vglrun +v -d /dev/dri/card0 glxgears -info: 10283 frames in 5.0 seconds = 2056.458 FPS and same info before like GL_RENDERER, GL_VERSION, etc.. I get the following errors but I guess due to some misconfiguration and nothing related to my problem: ERROR: ld.so: object 'libdlfaker.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. , ERROR: ld.so: object 'libvglfaker.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Paolo

Mar 21 16:30

I can rim xvfb outside of apptainer and seems to work, since I get something like 11422 frames in 5.0 seconds = 2284.254 FPS and some other info before. For virtualgl can you explain how can I run it outside?

Paolo

Mar 21 16:30

@MartinPecka thanks for the help again. I added Update 4 with the output of the commands

Paolo

Mar 21 16:30

I checked by printing the files in the binded folder and it works as well, I can see the files 0_nvidia.json and libEGL_nvidia.so.0. I was asking if I have anyway to run the simulation with the xvfb-run -a vglrun +v -d /dev/dri/card0, remove the --nv flag or something else. Sorry, but the possible configuration are becoming quite a lot and I'm scared that things won't work just because I mess up something :)

Paolo

Mar 21 16:30

Sorry, I mean this command to bind the folders

apptainer shell --nv --bind /usr/lib64/libEGL_nvidia.so.0,/usr/lib/libEGL_nvidia.so.0,/usr/share/glvnd/egl_vendor.d/  apptainer_image

Output from the terminal: $ find /usr -name "libEGL_nvidia.so.0" 2>/dev/null /usr/lib/libEGL_nvidia.so.0 /usr/lib64/libEGL_nvidia.so.0

Paolo

Mar 21 16:30

@MartinPecka thanks for all the information. So the command to run is apptainer shell --nv --bind /usr/share/glvnd/egl_vendor.d/ apptainer_image? And after that launch the simulation with xvfb-run -a vglrun +v -d /dev/dri/card0 ros2 launch package_name launch_file.py?

Paolo

Mar 21 16:30

@MartinPecka thanks for the suggestions. The statement you're referring to (Update 1) was made on a GPU node, but the issue is that sometimes the command glxinfo | grep -i opengl produces no output, and in this case I cannot run the simulation. Even when it does run, the RTF is low. I have added an Update 3 section, where you can find the output of the command xvfb-run -a vglrun +v -d /dev/dri/card0 ros2 launch package_name launch_file.py

Paolo

Mar 21 16:30

@MartinPecka If I use a node without GPU the simulation works (with a low real-time factor)

Paolo

Mar 21 16:30

@MartinPecka thanks for the reply. It doesn't work (same error) regardless of whether I run it in interactive or non-interactive mode

Paolo

Mar 21 16:30

Hi @MartinPecka thanks for the reply. I added Update 2 to my question. dev_gpu_4 is a development partition in the HPC with GPU nodes (4x NVIDIA Tesla V100), intended for testing and debugging, with a job time limit of 30 minutes.

Paolo

Mar 21 16:30

@ntfs.hard good point. Indeed, if I just run the world without spawning the TurtleBot 4 robot the real time factor is around 3. Anyway, in my local machine I can run the simulation with the robot with a rtf of ~ 1 (in the cluster the resources should be even better than mine). Also, I noticed that if I run just the gazebo world ign gazebo arena.sdf -s -v 4 -r I don't need to setup the DISPLAY=:0. I still think that there is something going on with ogre or ogre2

Paolo

Mar 21 16:30

@ntfs.hard thanks for the suggestion, I added Update 1 to my original question

Paolo

Mar 21 16:30

@ntfs.hard thanks for the reply. Apptainer should work fine since I can run the simulation locally using the GPU on my workstation. Interesting fact is, if I run the simulation on a CPU-only node of the cluster, I don't have this error (but the real-time factor is really low), when I move to GPU-equipped node, I get the error mentioned above. My question is why in the first case it doesn't require a backend rendering.