UAV drifts away using HereFlow

Dear community,
my problem (short form) is the following: the UAV does not hold the position properly, but drifts away when relying on HereFlow optical flow sensor. Attempts to trim out the drone to compensate the drift were also unsatisfactory.

Setup:

Situation & Issues:
The sensors all work perfectly fine, required topics (e. g. OPTICAL_FLOW_RAD & DISTANCE_SENSOR) can be monitored and show reasonable values, and UAV seems to hold position after takeoff. Altitude is held perfectly over entire flight duration. The problem is, that it still has a slight drift (roughly 1 meter per minute), which is not sufficient for mission target. The log files show that the estimator assumes the drone to oscillate with minor amplitudes around the initial position while the GPS sensor measures a significant drift. The drift direction is fixed in body frame, it is independent of wind direction. The log files also show that the estimator is good when the drone is moving: when manually flown back to its initial position the position shift looks similar recognized by estimator and by GPS (flight path has same shapes, only a GPS offset is given).

Internet research leads to the following results for Troubleshooting:

  1. “Calibrate the sensor.” (usually leads to here: Optical Flow Sensor Testing and Setup — Copter documentation).
    As I am using PX4, not Ardupilot, I could not find a way to perform the calibrations – I could not find a PX4 equivalent for parameters like FLOW_FXSCALER etc. Also, HereFlow setup does not include any calibrations (see link above)
  2. “Exact values for Offset are very important.”
    I used very exact values, but also tried to strongly detune via setting big offset values – the effects of high offsets were too neglegtible to explain the drift by having not exact enough offset values
  3. “The ground surface plays an important role.”
    I performed the tests above an asphalt surface with many features and no elements moving due to downwash / wind in an adequate altitude (ca. 5 m above ground).

At testing I found the following:
a. Position offset parameters do barely affect the drift
b. (Not) including Multi rotor drag fusion in the EKF has no effect on the drift
c. Enabling UAVCAN_SUB_IMU (using the IMU of the HereFlow Sensor) leads to an instable flight behavior (strong swings / oscillations of the drone in xy-plane around setpoint position).
d. Recalibration of level horizon had no effect on the drift
e. PX4Flow roation parameter has no effect when HereFlow is used (as expected)
f. Slight adjustments to the orientation / attachment of the sensor on the drone show significant effects on the drift
Changing the pitch angle and the yaw angle of the sensor relatively to the Pixhawk by beveling by clamping thin objects under the sensor’s circuit board affected the drift direction and magnitude. The problem here is that I could not find parameters or calibration methods to compensate for this. Also, even the smallest changes create visible effects on the drift, which is why I can’t imagine that perfect physical alignment is achievable when mounting.
g. Attempts to trim out the drone to compensate the drift deliver unsatisfactory results.

My questions:
I. Is there a way to manually calibrate the sensor / compensate for sensor orientation offsets when used on a PX4 setup?
II. Is there a tutorial or best practice for perfectly mounting the Sensor that has proven to be successful?
III. Can this be solved by adapting the EKF and how can this be achieved?
IV. What is another potential source of error?

Please find some additional information below.
Many thanks in advance for your contribution,
knolle

Add-on: More detailed information:
Here is some more information on the test flights and the log files.
As I can not post enough links in this post, I put the descriptions of setup changes and the links to the logfiles in the feedback section of this flight log:
https://logs.px4.io/plot_app?log=2dc5f9a6-ffe3-41c2-aa91-d591789ddd08
Sorry for the inconvencience.

Here is some additional plots of flight data. If not stated otherwise, the x-axis shows the frames. As the optical flow (OF) sensor has double the frame rate as the GPS sensor, those value ranges resemble the same timespan for both sensors. Positions are given in meters, velocities in m/s. Major position and velocity changes are due to manual inputs to compensate the drift. Unfortunately, number of embedded files to post is limited, but further plots can be provided on request.

Here you can see the drift, and how drift direction turns after the 90 deg yaw:
grafik

Shows the drift:
grafik

Shows, how the OF-based estimator brings a velocity which results in drift:
grafik

Here’s an update of all what I have tried:

  • (de-)tuning of optical flow sensor’s and rangefinder’s pitch offset (EKF2_RNG_PITCH)

  • (de-)tuning of optical flow sensor’s and rangefinder’s x, y, z offsets (EKF2_RNG_POS_, EKF2OF_POS_)

  • (de-)tuning of optical flow sensor’s and rangefinder’s mounting roll, pitch and yaw

  • (de-)tuning of optical flow sensor’s and rangefinder’s mounting x, y, z

  • (de-)tuning of optical flow’s and rangefinder’s noise, delays, gate size (EKF2_RNG_NOISE, EKF2_RNG_DELAY. EKF2_RNG_GATE)

  • increase and reduction of sensors’ mounting stiffness

  • (un-)tick “optical flow” in EKF2_TERR_MASK

  • (de-)activating COM_HOME_IN_AIR

  • use barometer instead rangefinder as primary height source (EKF2_HGT_MODE)

  • enable & disable EKF2_RNG_AID

  • auto PID tuning

  • manual PID tuning (Velocity, Position)

  • (un-)tick “inhibit IMU bias estimation” and “multi-rotor drag fusion” in EKF2_AID_MASK

  • use “simple control mode” or “acceleration based input” in MPC_POS_MODE

  • setting unused sensor’s EKF2 Delay parameters to zero (EKF2_GPS_DELAY etc.)

  • changed values of EKF2_BCOEF_

  • several sensor recalibrations

  • flying indoor and outdoor

  • different propellers

  • different batteries

  • different lightings

  • use optical flow sensor and rangefinder on different drones of the same model

  • use optical flow sensor and rangefinder on different drone models

None of the actions above brought significant improvement to reduce drift. By replacing the optical flow sensor with another optical flow sensor of the same model the drift direction and intensity changed. This leads to the conclusion that the drift is inherent in the sensor and possibly a lack of production quality.

I found the following related issues:

Possibly these are solved when you read this searching for solutions.

Best regards,
knolle