Connect Herelink to SITL simulation

I am trying to make our software compatible with the Herelink. Currently we listen on a UDP port, 14540, using MAVSDK. When pointing the PX4 simulator to the IP of the Herelink controller, the connection works.

I tried to connect QGroundControl to it and it also connects. However, the joysticks don’t do anything, whereas I would have expected them to. Is there configuration I need to do? Do the joysticks communicate directly over the radio, is there a way to make the MAVLink messages go to the simulator over IP?

I feel like I’m missing major pieces of the puzzle, I haven’t found any relevant info in the docs

I saw this post: Herelink simulation, but it doesn’t really answer my question.

Is there maybe a way to connect my computer to a the Herelink Airunit with the ethernet cable and have a simulation run on the computer and connect through ethernet?

Or would I need to buy a cube and run HITL simulation on there?