ESP32 MQTT funktioniert nicht 24/7 - solved

[eRacoon]

Tried Board Version 1.0.6... took 2min until the first crash

Next try with 2.0.11 running right now...

Update: Took 9min until the first crash with 2.0.11...

 

[WhatATest]

Hmm, is arduinoJson up to date?

 

[eRacoon]

Nope... it's 6.18.5 like the config.h notes are recommending.
I will try the latest 6.21.5 and if it doesn't change anything will go to 7.0.4.

I also removed mqttClient.loop function from the loop, can't see that this line has any use. HTTP and MQTT still working fine but also had a crash reboot after 30min.

void mqtt_loop() {
  mqtt_poll_mower(millis());
  mqtt_connect();
  // mqttClient.loop();
  mqtt_publish_props();
  mqtt_publish_state();
  mqtt_publish_stats();
}

 

[eRacoon]

6.21.5 -> same crashes
7.0.4 -> instant crash loop

So I was able to check where exactly it crashes with some console output.
The crash happens always in the mqtt_publish_state(); part of the mqtt code.

This is the section with AT+S where the voltage, ampere, gps position etc. is requested.

Maaaybe this problem is related to a sunray version where sunray is showing some unexpected output for this command which will kill the esp32 when trying to sent this over mqtt. Maybe the convertion to json or string is not working for that output "mower.state.toJson().c_str()".....

Now I have the mqtt_publish_state function disabled and will watch if the crashes are gone.
Maybe this is also the reason why the problem only happens when charging and not while mowing.

 

[eRacoon]

Not a single crash over night with the state mqtt function disabled.
Doesn't help, because that is the important stuff I want to be send over mqtt.

 

[WhatATest]

Yeah, I remember something about Json crashing.
The weird thing is that my file is the same as master, so not sure how I dealt with it...

Can you try printing mower.state.toJson().c_str() and see if it crashes on 7.0.4 and what get printed?

 

[eRacoon]

With 7.0.4 and also 7.0.3:
CONSOLE.println(mower.state.toJson().c_str()); -> still instant loop crashes on that line
CONSOLE.println(mower.state.c_str()); -> not compiling
CONSOLE.println(mower.state); -> not compiling

When I do the output with the latest 6.x version it looks fine... until it's crashing around every 30min.

Example output:

{"battery_voltage":28.59000015,"position":{"x":8.430000305,"y":-20.55999947,"delta":-2.880000114,"solution":0,"age":230.5,"accuracy":607400.0625,"visible_satellites":0,"visible_satellites_dgps":0,"mow_point_index":1142},"target":{"x":0,"y":0},"job":2,"sensor":0,"amps":-0.01,"map_crc":-3820500}

The values looks like kind of the same like cassandra is requesting over http... I also looked an the cassandra data which was pulled right before the crashes but can't see anything that looks different than the other 1000 times which doesn't crash.

Here are two examples right before the crash:

10:59:31.704 -> 1101949 UART rx:S,28.59,8.43,-20.56,1.32,0,2,1142,1116.86,0,0.00,0.00,607400.06,0,-0.01,0,-3820500,0.00,-1,0,0x7d

12:24:38.568 -> 5106496 UART rx:S,28.58,8.43,-20.56,1.76,0,2,1142,6217.75,0,0.00,0.00,607400.06,0,-0.01,0,-3820500,0.00,-1,0,0x89

And here some that worked fine:

13:10:16.052 -> 133738 UART rx:S,28.59,8.43,-20.56,-2.88,0,2,1142,279.21,0,0.00,0.00,607400.06,0,-0.01,0,-3820500,0.00,-1,0,0x84

13:10:24.929 -> 142651 UART rx:S,28.59,8.43,-20.56,-2.88,0,2,1142,288.11,0,0.00,0.00,607400.06,0,-0.01,0,-3820500,0.00,-1,0,0x83

13:10:40.113 -> 157803 UART rx:S,28.59,8.43,-20.56,-2.88,0,2,1142,303.25,0,0.00,0.00,607400.06,0,-0.01,0,-3820500,0.00,-1,0,0x7c

13:10:54.850 -> 172551 UART rx:S,28.59,8.43,-20.56,-2.88,0,2,1142,317.98,0,0.00,0.00,607400.06,0,-0.01,0,-3820500,0.00,-1,0,0x8b

 

[WhatATest]

Weird, does this crash?
mower.state.toJson()

 

[eRacoon]

Yeah, sorry I guess I tried this already but forgot to mention....

My code:

void mqtt_publish_state() {
  if (!mqttPendingPublishState) return;
  CONSOLE.println("Before state Json");
  CONSOLE.println(mower.state.toJson());
  if (!mqttClient.publish(mqtt_topic("/state").c_str(), mower.state.toJson().c_str())) return;

  mqttPendingPublishState = false;
}

The output:

13:35:19.043 -> Before state Json
13:35:19.043 -> Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
13:35:19.043 ->
13:35:19.043 -> Core  1 register dump:
13:35:19.043 -> PC      : 0x400d6582  PS      : 0x00060330  A0      : 0x800d6e07  A1      : 0x3ffcdfe0
13:35:19.043 -> A2      : 0x0000d0d0  A3      : 0x3ffce050  A4      : 0x0000ffff  A5      : 0x0000ffff
13:35:19.043 -> A6      : 0x00000000  A7      : 0x3f40091a  A8      : 0x0000ffff  A9      : 0x3ffcdfc0
13:35:19.043 -> A10     : 0xa5a5a6a5  A11     : 0x00000100  A12     : 0x3ffdd0d0  A13     : 0x3ffce0dc
13:35:19.091 -> A14     : 0x00000000  A15     : 0x00000001  SAR     : 0x00000013  EXCCAUSE: 0x0000001c
13:35:19.091 -> EXCVADDR: 0xa5a5a6af  LBEG    : 0x4008a871  LEND    : 0x4008a881  LCOUNT  : 0xfffffffc

The other requests from cassandra over http are still working, so there should be some valid data...

13:39:57.438 -> UART tx:S,28.59,8.43,-20.56,-2.73,0,2,1142,317.40,0,0.00,0.00,607400.06,0,-0.01,0,-3820500,0.00,-1,0,0x78

 

[WhatATest]

Well, my best idea is to go here

Sunray/esp32_ble/src/adapter_tojson.cpp at master · Ardumower/Sunray

Ardumower Sunray. Contribute to Ardumower/Sunray development by creating an account on GitHub.

github.com

and comment stuff out in State::toJson() to see where it is crashing, I dont know if the ESP 32 supports debugging, that could be faster.

 

[eRacoon]

Ah good idea, I was wondering where the convert is defined...

Comment those two lines and the code working fine with that output.

String ArduMower::State::State::toJson() {
  String result = "";
  DynamicJsonDocument doc(1024);

  doc["battery_voltage"] = batteryVoltage;
  //doc["position"] = position.toJsonObject();
  //doc["target"] = target.toJsonObject();
  doc["job"] = job;
  doc["sensor"] = sensor;
  doc["amps"] = amps;
  doc["map_crc"] = mapCrc;

  serializeJson(doc, result);
 
  return result;
}

Output:

14:17:53.683 -> Before state Json
14:17:53.683 -> {"battery_voltage":28.45999908,"job":0,"sensor":0,"amps":0,"map_crc":-3820500}

 

[WhatATest]

So both lines crash?
Those are defined in Position::toJsonObject() and Point::toJsonObject()

You could try and do the same and see what crashes in those.
If both crash it cold be a problem with the casting (??), try "to<>" instead of "as<>" and see if that does anything.
Here you can read about DynamicJsonDocument, might help you debug this.

Another possibility is that it crashes at "serializeJson(doc, result)" if the json objects are present.

 

[eRacoon]

I tried a bit around but couldn't find any idea how to fix/debug this.
Whatever I try to change the code gets invalid... not deep enough into coding.

So I just give it a day and reverted to the old version and hope my wlan timeout will help that the mower will reconnect after crashes.
When someone find a solution I would be happy to update, I assume all ESP's have the same problem with newer sunray releases but maybe never noticed that.
I only found this because of my bad wlan.

@WhatATest
Thanks for your massive help!