Bunch of orange cubes start to fail

Dear,

we are using the cube black and oranges in our fleet for drone shows. They are very reliable! Kudos. Though lately we have been plagued with an issue:

[MAV 174:1] Config Error: fix problem then reboot
[MAV 174:1] Config error: Board Validation $CHECK_BARO1_PRESEN
[MAV 174:1] T Failed
[MAV 174:1] Config Error: fix problem then reboot
[MAV 174:1] Config error: Board Validation $CHECK_BARO1_PRESEN
[MAV 174:1] T Failed

We have already 4 that are failing like that. And we have no idea why. They fly happely and correctly do shows, but suddenly we try to use them again and are greeted with this error message. We started with one such cube. That could be a fluke, so we didn’t think a lot about it. We investigated and for some reason the barometer is bad. So we voided warranty and replaced it, which worked.

Though now we have already 4 of those and we don’t want to replace all those barometers and voiding warranty. It starts to look systematic and therefor we are now investigating what the reason is.

Does CubePilot has experience with this error? Is this maybe a problem in a specific batch of orange cubes? Or do you have any idea how this can be triggered, so we can make sure we don’t have more cubes dying on us.

Thanks!

What firmware did you install?
It looks like you loaded an old parameter file from one of your Cube Blacks.
That is not a supported function.

Currently we have ardupilot 4.1.5 on it. But I can try stable.
So you think there is a wrong setting in the parameter file? Interesting, I have not though about that.
Do you know which? Or should it be solved by resetting all parameters? (How can I do that?)

Thanks!

I don’t think the parameters are the problem.

I updated to 4.4.0 and tried all parameter reset options in
https://ardupilot.org/copter/docs/common-parameter-reset.html

But sadly without avail; I’m still getting the error:

[MAV 001:1] ArduCopter V4.4.0 (502702df)
[MAV 001:1] ChibiOS: 1ec9f168
[MAV 001:1] CubeOrange 0032004B 3230510B 34373234
[MAV 001:1] RCOut: Initialising
[MAV 001:1] motors not allocated
[MAV 001:1] Config Error: fix problem then reboot
[MAV 001:1] Config Error: Board Validation $CHECK_BARO1_PRESEN
[MAV 001:1] T Failed
[MAV 001:1] Config Error: fix problem then reboot
[MAV 001:1] Config Error: Board Validation $CHECK_BARO1_PRESEN
[MAV 001:1] T Failed

@Hannes_Verschore Have these Cubes been opened?
Is this issue on CubeOranges or CubeBlacks or both?
Can you enable LOG_DISARMED and share the logs?

Sorry about the delay.

The cubes have not been opened and the issue is present on Cube Orange and blacks.

I’ve tried running with LOG_DISARMED, but wasn’t able to get a log out of it. Could it be that the logging didn’t start because of the error. I’ll let one of my engineers try next week again.

@Hannes_Verschore Can you load ArduPlane 4.4 on the Cubes and check if you still see this error.

Hi @Hannes_Verschore, I have run into very similar behaviour this past week with two different brand-new Cube Orange Plus units. Did you end up solving this issue?

Dear Franco,

we did not solve the issue. We can only assume that the BARO just dies. And apparently there are higher quality BARO and lower quality. We currently replace the BARO and we don’t see the issue reappear. Sad that we must do this, but it works for us.

Are you talking about a lot of units? Or just 2-3?

Best Hannes

Hey Hannes, thank you for your response and sorry for the belated reply. I have made a post detailing my findings in the meantime.

This has happened only on two units so far but these are also the only two units that we have tested from the new batch. The Baros are also failing intermittently on our side and I suspect this also causes the internal SPI lines to fail. I am aware that ICs have performance tolerances but it is wild that these Cubes passed QC processes - they are not reliable.

So to confirm, you fixed the issue by opening up the Cubes and dropping in replacement ICs for the MS5611 Barometer? This is good to know but I will likely be forced to return my faulty units as we can’t amend our production process to fix broken parts :sweat_smile:

We have just experienced this for the first time.
A barometer failed midflight, and resulted in an altitude drop since the air pressure is used to calc the height off the ground. After landing, the cube failed to boot giving the same error as described in the title post.

This cube was purchased a few months ago, and it seems like the cube failed at an arbitrary point with only a few hours of operation.

@Hannes_Verschore thank you for your documentation on this issue, it seems to be related to mfg and this thread sped up our troubleshooting quite a bit

Hi @Hannes_Verschore I really don’t recommend modifying the product like that. It will be really unsafe to fly with modification like that.

The sensor check at boot is something that is done only on Cubes, no other hardware running ardupilot has this sanity check. ardupilot/libraries/AP_HAL_ChibiOS/hwdef/CubeOrangePlus/hwdef.dat at master · ArduPilot/ardupilot · GitHub

The reason is that we seriously care about safe use of our products, that is why we have multiple sensors to cope with inflight failures and boot time check to ensure all sensors are healthy before flight. Although we have a really strong QC process, there’s still a chance of sensors getting broken on the way to our customers and during operation. If you have a broken Cube, please initiate RMA process through your reseller, you do not need to modify the Cube and continue using it.

@CjF Can you share your flight log, There are two Barometers on the Cube, so even if one fails it should have switched to the second EKF with BARO. Please ensure you have correctly set EKF Affinity settings: EKF3 Affinity and Lane Switching — Dev documentation

@sidbh The file is too large to share here, is there an email I can send it to?

There was an event while descending in auto mode, which we suspect is related to the cube failing. Barometer health reports fine on both throughtout the flight, but the baro failed message comes up whenever the cube has been rebooted since this flight.

Incident:

RC signals from that time period seem to show one of the motors commanded to go low, which would cause the yaw.
image

The drone handled fine and all motors appeared to respond normally once the drone was switched to loiter mode, out of auto.

We are preparing the cube+ for RMA return