[Solved] Peripheral power / brownout / bootloader / PSM

I have asked you to contact your reseller, because your reseller can assist me to get to the bottom of this faster!

From what I can see from your test results, this “difference” between blue and black meets to have a root cause that is not Hardware related.

Please post the name of the serial port on mission planner that is used to connect to the Blue cube, and please post the same for the black.

Under the messages tab on mission planner, please post the startup information for the blue and black.

As stated before, you can continue to loose your cool, and just deal with me, and get one or two bits of information a day due to time zones, or, you can deal with your distributor, who is one of the best experts in these matters.

The startup information is very interesting.

Logs hopefully will tell us if ardupilot is detecting a fault, which is important here.

2 Likes

What code is running on each cube?

ArduCopter 3.6.0 first stable release with NuttX. Will change over to ChibiOS during next few months.

The only significant difference we have found between black and blue cube is following. Blue cube will pull-up randomly CE pin on BQ24315 to 3V3 shortly 500…1000us at bootloader time. Too short <=800us pull-up on CE pin causes undefined fault on BQ24315. This fault with 600-800mA load causes the BQ24315 to go into a current regulating mode. It will continue to regulate current until power rebooted or reset properly with >1ms pull-up on CE pin.

The question is this pull-up caused by hardware on cube blue or does it have different bootloader software that has this issue?

A proper pull-up during bootloader or during ArduCopter boot up can solve this issue. For example a
2ms pull-up on CE (reset) will clear the fault on BQ24315 100% times on our tests.

We use Serial 0 (aka usb port) on both black and blue cubes.

_UPDATE: Before you get to comment on the IO thread heartbeat. We didn’t have SD cards on the cube at the time. However this issue happens regardless of the SD card installed or not.

Black cube
Blackcube
Blue cube
bluecube

In short autopilot won’t detect power fault directly.

Here are the logs from the autopilot. You will find that GPS/compass (HERE2) will die suddenly, when the issue begins. BQ24315 IC is not giving Over Current fault during this problem, unless the current really goes shortly over 1A. The autopilot could make a guess about the issue indirectly if it sees multiple peripherals stopping to work.

Logs @Google drive

I would say that you have pretty much all information you need to replicate this issue.

We have already made changes to power supply infrastructure as this is a critical failure in our view and we cannot use the standard carrier board as part of the power bus for peripherals. We are no longer in a urgent hurry due changes, but in my opinion this should be investigated quickly, so that other users could make actions if necessary.

what you are describing is a software issue if anything.

I will chase up if there is anything in the bootloader that may cause this.

I have asked quite politely for you to contact your reseller, and you still refuse.

At this point, I am demanding that you ground your vehicles till you are willing to update to Chibios!

And I refuse to continue this conversation while you refuse to use SD cards, and contact your distributor

Conversation over till you comply

Thanks for being so sensitive and understanding.

I have begun the conversation with the distributor, with same kind of conversation delays as we have here. Distributor will try to reproduce the issue. I will still update our findings here.

Can’t do that now. Will do that during next few months. Thanks for the caution.

As I have said on my last post, we didn’t have SD card attached during taking those images.

We use them always when flying and this should be quite obvious as we had the LOGS there for you to download.

Even tough the software is not most up to date and SD card was not used while taking prints screen of startup info, it has nothing to do with the issue.

Feels like this issue is a joke for you. Thanks for your efforts anyway.

NONE OF THIS IS A JOKE TO US!
You are running code that has known serious bugs that directly affects the safety of your flight!

I have said we are looking into the issues we have raised, and I thank you for finally contacting your reseller, who is actively assisting in the matter.

Our team is looking into the code, and neither Nuttx or Chibios versions of ardupilot do anything but set the pin low.

It is possible that there is a change of timing of when that happens due to the new systems being more efficient, so we are looking at that.

If you could chill with the narrative around the issue, and stick to the exact known facts, we will get to the bottom of this.

I do thank you for bringing it up, it has highlighted that ardupilot is missing some key code on how to safely reset accessories for a transient OC in flight

Thanks for that bit.

Let’s get back to the engineering on this issue, and get some results.

Thanks for taking this under work :+1:
I think I have nothing more to add to this for now.
Distributor has my email and I will check this forum frequently if you need something like testing from me.

Hopefully this will further improve the autopilot and it’s resilience to external error sources.

1 Like

I’d like to add that we may be experiencing the same issue, although we use Cube Black. I described our experience here, but the tl;dr is that our peripheral supply seems to be getting stuck in a current limiting state, even though our draw is well under the limit (<400 mA) and no overcurrent state is logged.

Any tests we can/should do to determine if it’s the same issue?
In any case, we will likewise be contacting our supplier with our test data.

1 Like

Yep,

I have read your description and that sounds similar. During the current limiting the yellow flashing turned to orange.

You could verify this issue by having resistor or other constant load of about 800mA on peripheral output. Remove all other peripherals and connect 6 ohm power resistor or other suitable load across the peripheral power supply (Note: Power 4W). Measure voltage and or current on the resistor with multimeter. Voltage should be about 5V at begin and current about 0.8A, after a while voltage will drop to about 1,5V across the resistor and current to ~250mA.

1 Like

If you don’t mind, I would like to change the title of this to **peripheral power / bootloader/ brownout **
Or something similar, need to get some more software types onto this

Does it behave differently if you apply the load AFTER power up?

In my case, no. It can happen regardless of when the load is applied.

I haven’t been able to correlate the time-to-failure with any factors, such as hot start vs. cold start. The issue often appears between 5 and 15 minutes after boot/load applied, but it seems quite random; I’ve had it happen immediately after applying the load, or never (left it overnight).

Is it possible this is not a new issue? Here is a thread in the archive over at ardupilot.org, referencing low/no voltage on peripherals using a similar bq2431x component, but i assume back on legacy pixhawk and the user reports a crash, so maybe it was physically damaged:

… And there’s another current thread over there in the /hardware/pixhawk section with no power:

For what it’s worth, we have never experienced this issue before (100+ units). It only started happening recently, and on multiple units, so it seems to be a new issue to us.

The PSM seems to start regulating even on low currents if the reset was not done properly.
A 300mA load is enough on long time periods (15-20mins).

Do you have varying load on the power bus? Using constant resistive load makes this issue quite repetitive and easy to replicate.

Issue found, Tridge is doing a patch for this as we speak :slight_smile:

Once he has a firmware, are any of you guys willing to bench test it?

1 Like

PR here: https://github.com/ArduPilot/ardupilot/pull/11319

Great, thanks. I will start testing it.

Not yet, but soon I will have the equipment to start making more scientific measurements. Will test old and new firmware with various loads.

1 Like

PSM is reset properly now on every reboot. The Power enable is pulled high for 107 ms, should be 100 ms? However that should not be a problem?

PSM works now like it is specified to work.

2 Likes

Awesome!
How ironic is it, that making code more efficient can cause an issue like this!

Glad it’s better