I wanted to post on this thread because I would like to show a plot from a recent log that occurred on a drone running two Here4 units. I know that it was mentioned that flight with Dual Here4 is acceptable, but based on this, I’m not sure I can agree. I have seen multiple Here4 failures and all have generally developed in the same way. First, the values of one or multiple magnetometer axes start to get noisy and grow very large, which does cause lots of EKF switching during flight, especially if it happens to the primary compass. Usually, after this failure happens in flight, the subsequent power cycle of the drone results in the compass being “offline” (i.e. the mag data is not showing anymore on the DroneCAN node). If it were such a rare occurrence, I would agree that a failure mode is predictable and can generally be handled without too much issue by having a backup. However, this particular log shows that there is a failing Z axis magnetometer on the secondary mag (MAG[1]) and towards the end of the flight, the Z axis of the primary mag (MAG[0]) starts to show this same failure mode:
One person there mentions at least 15 of them having failed, which seems extreme. With multiple reports of this type of failure, I also can’t imagine we are all doing something fundamentally wrong with the installation or build.
Mostly I am simply hoping to gain some understanding about what is going on here. I have sent multiple failed units to our reseller who communicates that they’ve been sent to CubePilot, but I have so far not been able to gain any insight into the mode of failure.
Please let me know if this information is helpful, or if there have been additional learnings about this issue.
I recently ordered 20 pieces here4. Already in 5 of them I have RM3100: BIST. The problem occurs very quickly because after about 2h of flight in the worst case. I also see unanswered threads on the forum about cases of up to 15 damaged modules this way in one user. Since discovering that the error occurs, do we have any news as to the cause and solution?
Yes, I have already come to the same conclusion. After examining my damaged modules, I see that the individual coils in the compass circuits lose resistance so it is pure mechanical damage. What is puzzling is that they are not the same coils, some are Z coils in others are Y coils. Now I am thinking more about how to counteract this problem. Currently, after the occurrence of damage in one module the machine is grounded, it must be sent back for a module change and calibration. The fact that in recent cases the occurrence was after just a few hours of flight practically excludes our machines from production flights. I plan to add external compass modules, however, I cannot leave the current problem unresolved. Is the problem related to a specific batch of equipment or does it occur completely at random and we can’t guard against it in any way? Another problem for me is that when occurring in flight, the problem is not clearly reported so that the pilot is aware of the problem with one compass and can land as soon as possible.
The recent issues seem more like major issue. Devices not showing up seems bad. Bad firmware should put in to maintenance/bootloader. But device not showing seems something major. Clock issues? Bad crystal oscillators?
I’m just presenting the data I see from the logs. In most logs I’ve seen with this failure, the jump from normal data to wildly out of range data is quite sudden, which seems in line with the copper wire cracking root cause. This log is one of the few that show this slightly different trend for MAG[0].MagZ. Either way, it still looks like the same end result and potentially the same failure mode.
As per my other comment, this topic is here4.. it is extremely obvious that here4 is not alone in this issue, but that’s again not relevant to this topic
we will have instructions for the update for bulk working units, and we will show advanced users how to update white leed units, or we will let you know where to send them to get a firmware update if you do not have the tools for a JTAG update. (JTAG update will only be for the white LED units, all other units will get a much more simple proceedure that requires no tools.. )
Once we are finished the full investigation and fix, I’ll chat about details, but until then, I will let our engineering team get to a solution as fast as possible without distracting them too much.
basically there is an issue where the bootloader gets into a bad state, so we will need to update the bootloader.
as we need everyone to update their systems, we need to make some tools to ensure people can do this safely so we don’t cause people to brick their units and make a bad situation worse.
Hi @Dawid_Zielonka
Can you share pictures of the damaged coils you found in your modules? To compare to what we’ll see in our modules. Thank you very much.
