SB_0000002 Critical service bulletin for Cubes Purchased between January 2019 to July 2019. DO NOT FLY

Mandatory Safety Bulletins
1

**Edit 3 April 2020
This issue is fully resolved by 3.6.12 but many updates have come since. Please keep up to date as this SB only relates to the IMU matter.
Please see Ardupilot release notes for other Ardupilot matters covered in each release

EDIT 11th December 2019
Due to SB_0000005 I’m updating this to ensure that you use 3.6.11 as a minimum
No other changes

Edit 28th July 2019
Please ensure you are using at least Arducopter 3.6.10
SB_0000002 Critical service bulletin for Cubes Purchased between January 2019 to present.
flight is possible under the following conditions :

  1. You MUST run Latest Ardupilot RELEASE CUBE BLACK code.
  2. You must use latest Mission Planner or Latest QGC
  3. Set the following parameters
    CUBE_BLACK.param (98 Bytes)
BRD_TYPE == 3
EK2_IMU_MASK == 7
(or EK3_IMU_MASK == 7  if using EKF3)
INS_USE == 1
INS_USE2 == 1
INS_USE3 == 1
  1. Save the parameters
  2. Reboot the cube
  3. reconnect mission planner
  4. If Mission planner flags your code, contact your local re-seller, and log the issue. DO NOT FLY TILL I HAVE PERSONALLY CHECKED YOUR LOGS, and either given the go ahead, or, organised what we will do next in your case.
    8.Further flight is only to be conducted where you have carried out a full risk assessment on the implications of a failure occurring, the new code is good… but it should not be relied upon as your only means of protection. You must not fly over people, you must not fly over critical infrastructure, you must also take responsibility for your payload and vehicle.

Failure to comply with these requirements may lead to serious damage, injury, or death.

EDIT: 02nd of May 2019:
In addition to following the instructions on the 27th of April, ensure that the following parameters are set.

EK2_IMU_MASK == 7 (or EK3 if using EKF3)
INS_USE == 1
INS_USE2 == 1
INS_USE3 == 1

EDIT: 27th of April 2019: Ardupilot has now added detection, prearm checking, and inflight checking and failover to the redundant sensors. This is now in Release Copter and Release Plane.
Detection of the fault has also been added to Mission Planner, and QGC latest. Mission planner now collects information on this fault if you are willing to send it to us.

As such, we are reducing this service bulletin to flight is possible under the following conditions:
1. You MUST run Latest Ardupilot RELEASE CUBE BLACK code.
2. You must use latest Mission Planner or Latest QGC
3. If QGC or Mission planner flags your code, contact your local re-seller, and log the issue. DO NOT FLY TILL I HAVE PERSONALLY CHECKED YOUR LOGS, and either given the go ahead, or, organised what we will do next in your case.
4.Further flight is only to be conducted where you have carried out a full risk assessment on the implications of a failure occurring, the new code is good… but it should not be relied upon as your only means of protection. You must not fly over people, you must not fly over critical infrastructure, you must also take responsibility for your payload and vehicle.

Failure to comply with these requirements may lead to serious damage, injury, or death.

Original notice:

It has come to our attention, that there is an anomaly with sensor readings on some recent cubes. This notification will be modified multiple times a day!!! Please check often.

This is an unfolding investigation, more details will be added as they come to light.

If your vehicle is fitted with a CUBE BLACK, DO NOT FLY. Wait for further instructions.

Your Cube is fitted with Three Gyros, Three Accelerometers, two Barometers, two internal Compasses.

to tell what is fitted…

Open mission planner and connect to your cube.
go to the full parameter list.

Please keep an eye on this notice for further instructions. There will be an update to mission planner, and to Ardupilot coming to assist in determining if your Cube is affected by this fault.

We are sorry about this situation, and will endeavour to follow up with solutions as soon as possible
Thank you

NEW MISSION PLANNER BETA RELEASED TODAY! PLEASE UPDATE MISSION PLANNER TO ADD a test of AUTO DETECTION OF THIS ISSUE . please let us know if you get the following error…

“Your board has a Critical service bulletin please see [link;https://discuss.cubepilot.org/t/sb-0000002-critical-service-bulletin-for-cubes-purchased-between-january-2019-to-present;Click here]”

Failure to get the mission planner notification is NOT an indication that it is safe to fly.

All advice given will assume your device is running MASTER ARDUPILOT FOR CUBE BLACK, unless otherwise stated.

Philip Rowse

10 Likes
2

Match in the sense how IMU 1&3 should have same value of first 2 digit?

3

Match in the sense IMU 1 & 3 should have same digit at first 2 nos.?

4

What about Baro and compass? You mentioned them, but did not provide value in the example. Please clear. Thank Philip.

PS: is this related to this issue? https://discuss.ardupilot.org/t/ekf-have-a-glitch-in-pos-alt-estimation/41018

5

Anyone with this issue, please update mission planner to latest beta, and please verify if the issue is caught.

Thanks

6

Perhaps a daft question but where does one find the latest Mission Planner Beta.

If I go here: http://firmware.ardupilot.org/Tools/MissionPlanner/ the Beta folder does not seem to have been updated for some weeks.

7

if you have installed the latest msi from ardupilot firmware page. Then go to help at bottom check beta update. Or planner in configuration there is a check box for check beta update (if using this option you will need to open and close MP twice for update notification)

8

Related Patches being worked on to resolve this issue for users of Ardupilot

2 Likes
9

In the Crash that triggered this Grounding a few key points became apparent.

  1. That the startup method of ardupilot if set as BRD_TYPE = 0 can fail to find some sensors depending on the status of other sensors.
  2. Unfortunatly, the User had Prearm checks set to “disable”
  3. An issue has been found in the firmware that caused the bad IMU to continue to be used for flight control.
  4. We have identified a group of cubes that have failed sensors due to a failure of the MOSI signal line in a limited number of PCB’s we are investigating this issue.
  5. in normal conditions, if the PreArm was NOT disabled, and with the patch applied to the EKF, we believe this crash was avoidable. Unfortunately hindsight is of little comfort for the owner of the aircraft.
  6. Once we fully determine the diagnosis of this issue, we will assist all who are affected to get replacement hardware ASAP.
  7. that the affected cubes are confirmed to have left the factory in good working order, and that all tests had passed
  8. we will be adding additional testing in the factory to further attempt to catch any issues like this in the future

Thankyou
Philip CTO

1 Like
10

Hi Philip, thanks for the very fast update on this.

Can you just explain a little on how you think the line is failing as from what your saying it’s sounds like a physical track failure in the Pcb after leaving the factory ?

Is is a connection issue at a later change ect or is the track burning out.

Also as a matter of interest is there something that changed that lead this to be 2019 cubes only? So last back of PCB’s or a minor design change if you get my drift.

Sorry for the 10 questions I’m interested in the failure as it appears it’s a physical one and it’s intrigued me .

Again I think fair to say you guys have been all over this and while no one ever wants to see a craft go down as a result of a HW issue there is never a single cause and as point out several learning points to take forward.

11

After much investigation, it appears that the through via plating process has not successfully plated the through bias that are between layer 2-3 in a very limited number of boards.

What is super strange. Is that the failure is the same area of the pcb for ail failures. But the rest of the pcb is unaffected.

The other thing that’s frustrating, is that the fault appears to show after the units have successfully passed flying probe testing, and the full function test jig.

The cubes that have been returned had all IMU’s functional at time of factory testing.

So now that we have located the exact fault, we have also had our internal team, and the Ardupilot team looking at the secondary aspects of this… ie, how did this single fault lead to a crash?

It turns out that the failover logic was not performing as optimally as it should, and Tridge and Jonathan have been working on a solution to get everyone flying again.

The Cube is designed to be fault tolerant, and should survive multiple IMU failures.

Testing today of their work is promising, we hope to find a short term and longer term solution that will make all boards much safer moving forward.

3 Likes
12

Some video of todays testing of the patches for Ardupilot.

4 Likes
13

Good job guys! That was a swift solution from the Dev team. :clap:

14

Thanks that’s fantastic and explains it, strange one for sure as you say, especially the change after leaving the test rig but then then there is vibrations in flight and transit to contend with.

I’m guessing this is on the lines either on or to the isolated board IMUs.

The only concerning thing really is we would have expected Ardu to use the remaining IMU as it was available but as you say it did not perform as expected.

Overall though failure will always happen and what’s important is how it’s delt with, I guess the next issue for you is is how do you deal with what’s good and what could fail in channel/people’s hands as it’s not something you can easily see before hand or test for before failure. Hopefully most will show failure before flight.

1 Like
15

The fault is on the isolated rigid PCB. same location for all affected boards.

While the investigation continues, (at the PCB manufacturing facility), it has been a good oppotunity to Audit our full supply chain, and current testing capabilities.

The bug does show up in a way that we can protect the vehicle in flight once the new changes are brought in, which is awesome! We believe we will catch 99.99% on the ground, and the 0.1% is a number that we expect in the air which is why we have triple redundancy in the first place.

However… we clearly missed a very important step in the chain here…

running a regular test of the firmware, in which we purposefully kill the IMU’s and see how it behaves.

you can see the behaviour in the video

3 Likes
16

Hello Philip !

Thank you for your fast answers !

I’d like to know what could be expected about Px4 and QGroundControl users ?
Are we subject to the same possible critical failures ? If it’s the case, what do we have to do ?

About the matching parameters, do they have to be this exact values ? For exemple, I do have the following parameters on one of my Cubes:

ACC_ID 1442850
ACC2_ID 1114658
ACC3_ID 1442826

GYR_ID 2360354
GYR1_ID 2228514
GYR2_ID 2360330

If it a problem ?

Sorry for asking so many questions, our drone production rely on your answers !

Best regards,

Florian Viguier

17

Good day, thanks for the update.

When do we expect the following

  1. ground station update that is not beta which will Gov the warning for the issue

  2. firmware that corrects the improper switchover of IMU

thanks again

18

Any further announcements will be here.

All released code will be Beta, and will only progress to full release if the full Ardupilot team is confident in this.

I am sorry I cannot give any further helpful answers, but this is an unfolding situation, and we are still trying to figure out which batches are affected by this issue

What we can tell you, is that the issue is unrelated to the design, assembly or testing of the cubes, and that once the patch is through with ardupilot, should not by itself lead to catestrophic failure of the control system. The issue is isolated to the vibration isolated pcb. I will not go into details yet on this. But will report as soon as I can.

However, the system would revert to a single non isolated IMU.

As such, I have asked for a prearm check to be added to enforce the three IMUs be enabled, and three EKF cores to be running on all cube Autopilots.

We designed in the redundancy in hardware for situations like this, so setting parameters to anything other than using all of these features is counterproductive.

Thanks

Philip

19

Thanks Philip for detailed notification, appreciated.

20

Does that mean there is no problem when running PX4 on the Cube? Or will you, or someone else, make a similar patch for PX4?