Hello CubePilot community. Your help is always invaluable. This incident happened yesterday, while flying at 5000 mts above sea level. In a routine loiter test, the 1400mm coaxilal quad (26" propellers and 22mAh battery) suffered a sudden drop of alitude which ended up breaking the landing gear.
The following was the message that I received from telemetry:
In the image, it can be seen that the drone looses confidence on its vertical measurement, and tries to change EKF mid-flight. Also, the following can be seen in the barometric altitude vs the EKF estimated altitude:
It appears as if the main barometer spikes and causes the drone to descend, but id did so in an uncontrolled matter.
The internal barometers show that indeed something happened, but I dont understand why the EKF did not reject that measurement. Is there anything else I am missing?
(possible miss-tuning, compass, power,…)
The log is the following:
Thank you for your time
A sudden spike is something that the system should respond too… so it shouldn’t be rejected.
If both barometers see it then it is indeed there…
He actually has two GPS units and 3 barometers - probably two Here2s on CAN.
He’s using the noisiest one as his primary.
And that spike threw EKF completely bonkers. Here’s Pitch vs DesPitch and Roll vs. DesRoll:
Phillip, DePara, thank you for your reply. It happens in this case that the spike is only reported by the barometer labeled as the first, which indeed I suspect came from a here2.
After the incident, I went and disabled both barometer units from the here2 modules (leaving the 2 internal ones in the cube), but I am still using both GPS units as well as both external compass units from both here2 modules using CAN connection. Should I switch to serial connections if I want to be able to use both here2 GPS (GPS blend) and compass capabilities? I also just upgraded to copter 4.
To clarify, was the spike in the noisy barometer the cause of the incident? Why did the ekf not switch to a different barometer?
Thank you all for your time and experienxe
The worrying fact is that your trivial baro glitch threw your EKF into an unstable state.
What you’re feeding your FC with, basically, is a whooping 5 altitude estimates. Three from baros and two from GPSes. One glitched, and the system as a whole didn’t discard it, and instead went haywire.
It’s hard to get devs attention when running “older stable firmware” as opposed to latest stable ore latest beta, but I’ll poke @rmackay9 here, as there maybe a lesson to learn.
May be a short? Here is a spike of 113 Amps
Thanks for all the responses, that spike in current and drop in voltage is interesting, however the drone was fine electrically after we fixed the landing gear (no more of such issues). However, I still have other possible causes of why the spike resulted in a highly unstable response from the drone:
The drone was tuned (roll/pitch/yaw pids) at 3600m MSL, and flew at 5050 MSL, could also be a tuning problem in the sense that the deviation from the equilibrium state was so high that hiddenunstable modes became important?
In this particularly harsh weather (-4 celsius low atm pressure) the IMUs struggle to heat up (flying at 30/60 celsius ) and had an important mismatch between them (can be seen in the Mission Planner auto analysis), so perhaps that made the EKF get crazy?
Thank you again for your time.
For my two cents it looks like more a overtuned PID issue, than EKF. According to the EKF log, EKF nicely recovered from the Alt spike (IPD went back to normal). But if you look for the Des Pitch vs. Real Pitch it looks like a PID feedback which just started to wear out when the copter hit ground.
Hello EsosBandi, thank you for your input. The issue with the overtuning is the fact that all three degrees of freedom, pitch roll and yaw, got severely affected after the ekf glitch, which was not induced by some sudden input on a single degree of freedom and instead induced by the glitch. In our previous mistuning issues, a small pilot command induced a big instability in one of the degrees of freedom, but this one in particular drove the entire drone bananas and uncontrollable.
Also, the drone had finished 3 successful missions, which show very good correlation between expected and actual input. The only difference in the present mission was the addition of 1.5 kg of weight and the glitch, which affected the EKF.
Perhaps the tuning, which was done at 3600m MSL was overcompensating at 5000 with the weight on?
Could have the glitch induced a virtual input in the drone that caused instability? many questions. This is fascinating.
In more news, the landing gear was repaired and the drone is now updated to copter 4.0 and performing well.
EKF is only responsible to determine the actual attitude of the aircraft. Responses to attitude changes are provided by the PID controllers.
I based the overtuning theory on the data that shows that EKF was perfectly aware of the copters attitude, if you compare AHRS roll/pitch/yaw with EKF roll/pitch/yaw they are in sync. SO the PID controllers were getting correct attitude information all the time.