Here4 loses many satellites and positional accuracy ~8:20min after being switched on

[wolke (michael wolkstein)]

Test Systems:

• Holybro 6x & Here4 on Px4 1.15.4 and Px4 1.16.1
• CubeOrange & Here4 on Ardupilot 4.6.3
• Here4 Hardware Version 4.19
• Here4 Firmware Version 1.15.C71AF5…..

What happens:

Approximately 8:20minutes after system power-up, the Here4 loses about half of its satellites for a second or less. As a result, the position becomes too inaccurate, and depending on the configuration, the system switches to failsafe land mode or drops out of position modes like Position (Px4) and back to, for example, Althold. After this single dropout occurs, it does not happen again. I was able to continue a test flight for about 30 minutes after this dropout without further problems.

This dropout can also be observed while on the ground. In fact, any flight controller with a connected Here4 module is sufficient to reproduce this error. It happens consistently, regardless of the flight controller hardware (in our tests, CubeOrange and Holybro 6x) or the time of day (satellite position).

Px4 Logfiles (6x FC + Here4 setup - no flight):

https://review.px4.io/plot_app?log=ede73e3e-040e-4b26-b931-267b9b0d7549

https://review.px4.io/plot_app?log=24198eb7-7074-480b-818a-2a7d464e33cb

https://review.px4.io/plot_app?log=1af8be9a-f329-4465-8f44-0719e6e7c712

Ardupilot Logfile (CubeOrange + Here4 setup - no flight):

ArdupilotLog.zip (5.5 MB)

Dataplot Image:

[Michael_Oborne]

Hi, I assume I spoke to you at xponential?

Is it possible to get the raw ublox log as well, this will need to be on ardupilot, with the passthrough function and I enter.

Also the firmware version on the here 4, just so we know where to start looking

Thanks Michael

[Jannis292 (Jannis)]

Hey @Michael_Oborne ,

this is Jannis. I spoke to you at the Xponential. Michael is one of my colleagues and it seems that he was faster posting this then me Thanks for your help thought!

Here is the FW-Version of the Here4. It should be the newest:

Here is a ublox log I recorded three days ago. But i noticed, since i enabled all debug msgs, there are some glitches in the communication, where no data is displayed (?).

tcp___127_0_0_1_500_260324_134136.zip (3.4 MB)

If you need something else just let me know!

Best regards

Jannis

[Michael_Oborne]

is it possible to get a new log with a faster baudrate?

i can see packet loss in this log

what im trying to do is determine which part is actually the cause.

the default is 460800

also during this log, did you notice the problem?

[Jannis292 (Jannis)]

Hey @Michael_Oborne

Sure, i will record a Log with the higher baudrate.

Since there were a lot of glitches in the u-blox center where no data was displayed, I could not see the problem occurring.

[Jannis292 (Jannis)]

Here are two logs, with and without debug-msgs enabled.

I set the baudrate to 460800 on the SL can port on the top right in mission planner and also on the popup that appears when setting the Here4 in passtrought mode.

Hope that helps @Michael_Oborne

tcp___127_0_0_1_500_260414_113111andmore.zip (8.8 MB)

[BayloReed (Baylor Reed)]

I think we are having a very similar issue posted here GPS Drops in Satellites at 4:12 elapsed time

[RoboMotic (AMV)]

…could be jamming? maybe?

[Oliver_Papak]

I am chiming in to confirm that we are experiencing the exact same critical failures with our HERE4 GPS units.

We operate a drone light show fleet, and this specific glitch has already cost us 5 drones. Like you, we see a sudden drop to 0 satellites, a massive HDOP spike, and altitude instability. The most frustrating part is that there is no pattern—it happens randomly mid-flight or even while the drones are stationary on the ground.

In our case, this glitch is catastrophic because it triggers a safety feature where the drone enters Land Mode (believing it’s in the final 5 meters of the mission). Since we fly mostly over the sea, the drones simply land in the water and are lost.

A few points to add to the discussion:

• We have tested all v1.15 firmware iterations, and the issue persists on every single one.

• We’ve been forced to ground our entire fleet because the GPS is the core of our business and we can no longer trust the hardware.

• I’ve attached a preview of our logs showing the instantaneous satellite drop and EKF variance.

This is a professional-grade module, but right now it’s performing like a liability. We need an official word from the developers—is this a known batch issue or a fundamental flaw in the current firmware’s GNSS processing?

Over 50% of our fleet is experiencing the glitch some after 9 min, the example below is at 51 as you can see so we cant predict it

[wolke (michael wolkstein)]

Hello

1. Today I conducted a few more tests. Specifically, I equipped a Holybro 6X FC with a Here4 CANbus GNSS module, and set up a second system with a Holybro 6X and a Holybro DroneCan F9P. First off, with the Holybro DroneCan F9P, I didn’t experience any dropouts—as described in this thread—during any of my test runs today. The longest log file with the Holybro DroneCan F9P is about 33 minutes long and has no dropouts.

In the comparison test, both systems were started simultaneously. The Here4 dropped out after ~8 minutes, while the DroneCan F9P continued running at the same time. ( In Image Orange, the System with Here4, Gren the system with Holybro Droncan F9P)

2. Regarding PX4 1.15 vs. 1.13, it seems something has changed here. There is also a thread on the PX4 forum: Failsafe Triggered: "Invalid Setpoints" and Blind Land with Here4 GPS - PX4 Autopilot - Dronecode Forum | Open Source Drone Development . However, it confuses more than it clarifies. The commit on GitHub, linked below, doesn’t provide much insight either. What is certain is that in PX4 1.15, EKF2_NOAID_TOUT set to 5 seconds by default does not work the way a human would expect based on the description. Instead, the 5 seconds are used to wait after the EKF position error occurs until the failsafe is triggered. However, the parameter suggests that the position does not have to be incorrect over a period of XX. I have included an image showing when the dropout generates a horizontal error. Incidentally, this is well below the threshold value of COM_POS_FS_EPH, which defaults to 5 meters. See the image for what Px4 1.15.4 does instead.
It has a minor positional error due to the brief satellite drop and triggers failsafe AltHold 5 seconds later. In my opinion, that’s completely wrong! (failsafe to AltHold in Yellow)

I have no idea if it’s possible to configure PX4 1.15 to have a slightly softer EKF positional error decision without jeopardizing the entire EKF system or making it too insensitive.

/g

wolke

Flight Review

[guyzoler]

That is very interesting since I had a similar issue a couple of weeks ago as well checking the log for that flight, I get many GPS errors. See post at: GPS 1: error changed

I replaced the Here4 unit with another and the errors in the log went away.

wonder if it is the same issue.

[César_Eizo]

Hello everyone, I am facing the same issue, with both the Here4 Blue and the Here4 Black. The problem is not happening with just one unit, but with several. On top of that, there are the units that present the compass issue. I see many other people reporting the same (very serious) issue and seeing very little action from CubePilot’s side. This isn’t something that just started; I have seen reports of this for a while now. This is happend to me today (again). Two Here4 Blue installed

[HeliEngadin]

Hello,
in our experience after distributing thousands of units, there are some unfortunate situations that push the GPS chip inside the Here4 to the limits of CPU processing capabilities.

We noticed first hand that units used as RTK rovers, sometime fail this way due to CPU being overwhelmed with correction data and position fix calculation, and the CPU just resets.
Usually they recover fine in a couple of seconds like shown here above.

We traced this to a configuration where these conditions happened simultaneously:

• all GNSS constellations were enabled
• RTCM correction data for both L1 and L5 were provided
• message rate was set to 10 Hz
• lots of satellites were visible

Reducing to using only two constellations allowed us to maintain a 10Hz RTK rate thereafter.

Maybe there is some message that can be enabled in the uBlox module in order to monitor the CPU/RAM usage and look for peaks.

[César_Eizo]

Ok, but is this official?
Is there any official notice from CubePilot about this?
Are they going to release a Service Bulletin to correct this?
Are they going to replace any components?
I say this because they always release something, ask us to change parameters or update the firmware, but a new problem always arises. And they are always serious issue…

I know this is a delicate subject, but I believe that when there are very recurrent problems, there should be an official statement informing the users. This is not news; it is something that the Here4 has been having problems with for years, and new users unfortunately end up finding out about it in the worst way. I’m sorry to say all this, but I’ve been through so many problems because of the Here4. I just wish it would work properly…

[HeliEngadin]

This is not part of any official statement from CubePilot, it is just our own experience.
And honestly for this particular situation, we can’t really blame CubePilot; problem happens with non-standard rate setting (default is 5Hz), it’s more like we’re hitting a limitation of the chip itself than an issue with the CubePilot product in and of itself.

We understand your frustration. In many cases, we rely on our own real-world experience to support customers after sales while working with CubePilot behind the scenes to better identify issues.

A large number of support requests end up being configuration-related, so we first try to rule those out before escalating to the manufacturer. It is also understandable that, due to past incidents, they prefer to fully validate solutions internally before releasing procedures or recommendations publicly.

[César_Eizo]

Thank you mate !

[Fares (Dr. Fares Al Dhaheri)]

This is a very interesting pattern.

Looking at the plots, the most important point seems to be the repeatability of the event around the ~8:20 mark after startup. The satellite count drops very sharply for a short period, HDOP/accuracy spikes at the same time, and then the system recovers and remains stable afterwards.

Because it happens on the ground, across different flight controllers, and apparently only once after startup, it does not look like typical random RF interference or normal antenna placement behavior. It feels more like a receiver-side state transition, firmware timing event, ephemeris/constellation update process, or communication/output congestion during initialization.

One screenshot that caught my attention is the UBX MON-COMMS view showing UART TX usage at 100% with peak usage above 100%. If many raw/debug messages are enabled, it may be worth testing with a reduced UBX output message set to see whether the dropout changes or disappears.

A few tests that may help isolate it:

• Repeat the same test with minimum GNSS output messages enabled
• Repeat with raw/debug messages disabled
• Compare cold start vs warm start behavior
• Test whether the event timing changes after clearing receiver data / GNSS reset
• Test the module with active cooling or shaded mounting to rule out thermal transition effects
• Log raw UBX during the exact dropout window if possible

The repeatable timing is the most interesting part here. Very curious to see whether this turns out to be firmware-related or output-stream/load related.

Dr. Fares Al DHaheri

Al-Etihad Industrials. UAE

[Jesus_Cervantes]

Hi all,

I too have experienced the exact same issues on Here4, and have done a lot of testing on the ground to confirm that it’s happening consistently and independently of other interference sources such as RF and high power.

At first I saw it happening on my drone, so I thought maybe it was radio or ethernet interference knocking out the GPS momentarily. But after investigating prior logs, I saw that it was definitely not that issue, and definitely not “random”.

In nearly all of the logs collected the issue is happening around the 8.2 minute mark. A transient drop of sat count typically from around 30 down to 12 or 13. In some cases, the EKF handles it fine, but in others, GPS is rejected for a few seconds and the drone does a toilet bowl.

I did some tests with ONLY a carrier board, a cube, and a Here4 GPS, powered by only USB - no radios, no high current, no switching regulators, etc. The issue still happens nearly 100% of the time, at either ~4 minutes or around 8.2 minutes.

The tests were done with the latest Here4 firmware, and no other settings have been modified, aside from the CAN node ID.

I also did an extended test (>30 minutes) to see if it would reoccur, but it does seem, as others have reported, that it only happens once in a power cycle. Here’s a log from the event happening around 4 minutes:

@sidbh I know you’ve previously helped me identify another bug with the 71 minute wraparound issue - is this reported bug something you guys are already investigating?

It feels like it must be something firmware related, and possibly on the ublox side of things.

I have a Here4 from a prior batch (without AKM compass) that I am happy to test with different things for the sake of learning. Anything we can try changing from within u-center? I know that the Ublox firmware on the Here4 is custom to permit moving baseline, but in my case right now, I’m working on a drone with only 1 GPS, so I don’t care about moving baseline functionality. Would it be safe to use the Ublox production firmware as a comparison test?

At the risk of stating something obvious (but hoping it will help spur some thoughts), the 4 minute-ish time marks seems to correlate with a byte rollover (~255 seconds).

Thanks and hoping we can collectively understand and resolve the root cause of these problems.

[Jesus_Cervantes]

Hello again,

After spending significant time chasing the same ~4-min and ~8-min sat dropout bug across multiple Here4 units (including a brand-new unit straight from the shelf), I want to share a workaround that has so far held up across three consecutive bench tests on a unit that previously failed 100% of the time. I am hoping others affected can try this and confirm whether it also works for them. And I would also love for the CubePilot team to weigh in on potential side effects of this if possible.

Background

I observed the bug on three separate Here4 units, including the latest firmware (GNSSPeriph 1.15.7). The bug fires once per power cycle at either ~4 min or ~8 min after boot, dropping sat count from ~28-30 to ~10 (sometimes lower) for about a second. The EKF reacts to this, and in my flights it has caused toilet-bowling behavior in Loiter mode, so this is operationally significant, not cosmetic.

Things I ruled out through extensive testing:

• Magnetometer interference (mag innovations were stable)
• Power-system EMI from motors (no current correlation across full flight)
• RF interference from onboard payload (bug fires on bare carrier board with only USB power)
• Constellation overload (bug fires with GPS-only, GPS+Galileo, GPS+GLONASS)
• L5 health flag toggling (GPS_DRV_OPTIONS bit 5 set, override active)
• HSITrimUsingPPS (GPS_DRV_OPTIONS=288 default on both working and broken units)
• QZSS signal disable alone

I also did try collecting the .ubx log to see if I could capture the issue, but ran into several problems. The first one being that I could not figure out how to increase the baud rate beyond 230400, and it seemed that the UART link was saturated. I tried changing it on the Mission Planner passthrough, but it failed unless I did 230400. I collected a ubx log anyway, but from the brief analysis that was done on it, it doesn’t seem like the bug happened while collecting data in this state. The system may be working differently when connected to u-center and debugging.

The Workaround

The thing that worked, on the unit that previously failed deterministically, was performing u-center’s “Revert to default configuration” followed by “Save current configuration” on the receiver, via Mission Planner’s CAN passthrough.

Step-by-step:

I’m unsure if the first 3 steps are truly needed. This was an artifact of trying to reproduce something from yesterday, so I followed the same steps exactly. Keep that in mind.

1. Connect to the Here4 in Mission Planner’s DroneCAN GUI. Set `GPS1_GNSS_MODE = 5` (or any non-zero value). Write and commit. This causes AP_Periph to push a CFG-GNSS message to the u-blox.

2. Power-cycle the airframe. Confirm sat count drops (constellation reduction took effect).

3. Set `GPS1_GNSS_MODE = 0` via DroneCAN parameters. Write to flash. This stops AP_Periph from continuously re-pushing CFG-GNSS on subsequent boots.

4. Right-click the Here4 node in DroneCAN GUI and select CANPassThrough Here3+/4. Default TCP port, baud 230400.

5. Open u-center → File → Receiver → Connection → Network → `tcp://127.0.0.1:port`.

6. In u-center: View → Configuration View → CFG → select “Revert to default configuration” → check both 0-BBR and 1-FLASH in the device list → Send.

7. Without closing the dialog, select “Save current configuration” → BBR + FLASH again → Send.

8. Cold start the receiver from u-center (Receiver → Action → Cold Start).

9. Disconnect u-center. Close CAN passthrough in MP.

10. Full power cycle (drain backup capacitor for 30+ seconds).

After this, on next boot the receiver takes a bit longer than usual to acquire (cold start), and sat count settles around 24-26 (vs ~28-30 before). The periodic 4-min and 8-min dropouts have not fired across 3 test sessions of 10-12 minutes each (i.e., past both typical trigger windows).

Some side-effects to note:

- Sat count drops 4-6 sats below pre-fix levels.

- The procedure probably resets ALL u-blox CFG values to factory defaults, then commits them to non-volatile memory. AP_Periph’s normal CFG-MSGOUT and CFG-RATE pushes still happen at boot.

Open Questions for CubePilot

If anyone from CubePilot is reading: would you be willing to comment on the following?

1. Is u-center’s “Revert to default configuration” safe to apply to a Here4? Specifically, does it affect the Moving Baseline customization that you’ve added to the u-blox firmware? My understanding is that “Revert” only affects CFG-VALSET items, not the firmware itself, so Moving Baseline capability should remain, but I’d appreciate confirmation.
2. Are there any other side effects I should be aware of? Settings that get reset to defaults that might be important for specific use cases?
3. Could this be incorporated as a one-time provisioning step on production Here4 units to ship them in a state that doesn’t exhibit the bug?
4. Any insight from your side on what specifically gets cleared by the revert that suppresses the bug? That information would help us understand whether the workaround is durable across firmware updates.

For Other Affected Users

If you’re seeing the same ~4 or ~8-min sat dropout on your Here4, please consider trying this procedure on a bench setup first (not in flight) and report back whether it stops the dropout on your unit. The more datapoints we collect, the better we can characterize whether this is a universal workaround or specific to my unit’s history.

[Fares (Dr. Fares Al Dhaheri)]

This is a very useful update, thank you for documenting the procedure and the side effects clearly.

The fact that the issue disappeared after a u-center “Revert to default configuration” followed by saving to BBR/FLASH is a strong clue. It makes the behavior look less like external RF interference, antenna placement, or flight-controller interaction, and more like a receiver-side persisted configuration/state issue.

The repeatable timing around the ~4 min / ~8 min window was already pointing away from random environmental causes. If resetting the u-blox configuration suppresses the event on a unit that previously reproduced it 100% of the time, that seems like an important datapoint.

The only part I would be cautious about is treating this as a general field fix before CubePilot confirms the side effects. A full revert may clear more receiver configuration than intended, and the reported reduction of around 4–6 satellites after the procedure is worth understanding.

It would be very interesting to know exactly which CFG items differ before and after the revert/save process. A before/after u-center config dump could help identify whether one specific persisted value is triggering the behavior, rather than requiring a full reset.

For affected users, I agree this should first be tested only on the bench, not treated as a flight-ready fix until CubePilot confirms the implications for Here4-specific configuration and moving baseline behavior.

Dr. Fares Al Dhaheri

Al-Etihad Industrials. UAE