Help debugging "Disconnect reason 7" problem

[sude22]

Hello Bert,
It's me again. Your MQTT client is really an excellent thing, years of annoyance about MQTT connections not reconnecting are a thing of the past :)
Because it worked so well for a new project, I have written a HOMIE replacement for all my home automation devices. It only contains what I need, but also what I personally was missing in HOMIE.
The basis of the thing is of course your MQTT client.

Maybe you can help me with a problem: Although the reconnect always works reliably, regardless of whether WiFi or the broker fails, I have a problem.
The broker is a current Mosquitto 2.0.15 in a Docker container on the HomeServer. The WiFi is now running quite stable with several AccessPoints from ExtremNetworks under OpenWRT and B.A.T.M.A.N Mesh.

I keep noticing that some devices have to reconnect to the broker after a while. The disconnect reason is 7.

KeepAlive is set to 30, but I send a status message every 10 seconds (more for testing).
Changing that values did not help.
Do you have any tips on how to debug the problem?

Best regards,
sude22
P.S. happens with ESP8266 devices...and espMqttClientAsync

[sude22]

These graphs are not entirely correct, as the broker was restarted more often at the end. However, it gives an impression of the magnitude of the disconnects we are talking about

[bertmelis]

Would you be able to get the broker log? Often it logs why it kicks a client.

[sude22]

Just turned on the debug logs on Mosquitto...
Well, there comes a lot.. I will try to find some parts that are relevant.

[sude22]

OK, first I changed some things...using the "normal" sync client now, removed some 3rd-party OTA stuff.
What I see now is this:
2023-12-16T09:19:06: Client esp8266c5b8d8 has exceeded timeout, disconnecting.
2023-12-16T09:19:22: New connection from 192.168.1.32:62316 on port 1883.
2023-12-16T09:19:22: New client connected from 192.168.1.32:62316 as esp8266c5b8d8 (p2, c1, k20).

One can clearly see that before the connection is interrupted, the heap memory rapidly decreases. The question is whether this is due to my remaining code or the espMqttClient. I will set up a test project with only the client and observe that.

[Pablo2048]

@sude22 check the RSSI or wifi disconnect/reconnect, check if you have some guard to handle MQTT disconnect, or set EMC_ALLOW_NOT_CONNECTED_PUBLISH flag to 0 (https://www.emelis.net/espMqttClient/#compile-time-configuration) to avoid the memory exhaustion when disconnected.

[sude22]

This my publish function, it checks if the client is connected before publishing:

uint16_t publish(PGM_P topic, uint8_t qos, bool retain, const char *payload)
{
if (!mqttClient.connected())
return INT16_MIN;

return mqttClient.publish(prefixMqttTopic(topic), qos, retain, payload);
}

[Pablo2048]

Well, then you have to check the wifi connection status also. As I wrote - start debugging with the RSSI value and the wifi connection status.

[sude22]

I will try.
My assumption was that no matter if MQTT or WiFi is not connected, the client returns connected()==false as status

[proddy]

not sure if this is related but in one of my projects we had many issues of reconnection issues with MQTT and it turned it to be a problem with espressif32's WiFi connectivity on Mesh networks (specifically FritzBoxes) due to channel roaming. We solved this by using BSSID or fixing IPs in the client code.

[sude22]

Thank you, you've given me some good food for thought.
I have noticed that there is indeed something wrong with the mesh, some ESPs have been wildly "pushed" from one ap to another. Then it's no wonder that the MQTT connection breaks off. I am now converting everything to unique SSIDs without mesh, the ESPs are fixed to one location anyway.
Let's see what the long-term measurements show.
Thank you all for your help so far.

[sude22]

A small interim result:
The use of unique SSIDs on the APs has significantly improved the situation. Almost all ESPs are now running for several hours with a stable WiFi AND MQTT connection.
Nevertheless, one out of 8 test devices showed a different behavior again. Same software version, same WiFi as the other device.

I have now compiled the espmqttclient with debug output and hope to gain further insights.

Theoretically, I could live with this status, as the MQTT connection is always re-established properly and there is no ESP reset. But now my curiosity has been aroused to clarify the matter.

[bertmelis]

Oddly, I'm happy with the situation. But this indeed sparks also my curiosity.

[sude22]

Unfortunately, another case has now occurred in which an ESP cannot reconnect to the broker.
Debugging is not switched on for this ESP.

You can see in the broker that it establishes a new connection, but then the broker times out after 30 seconds.
This then repeats itself again and again.

I use the reconnect code from the basic example.
Hopefully the case will also occur once on the device with debugging activated...

[bertmelis]

probably related to #106. That user could switch to the sync version. On esp8266 this isn't always the case. ESPAsyncTCP doesn't seem to be the most stable library...

[sude22]

Well, i did that yesterday..
This is really strange, but i guess without the debug logs there is no way to find out.

[sude22]

Ah, OK. I use VS Code / PlatformIO.
The debug device is up and running, let's see what happens over night.

[sude22]

It is really strange. Nearly all devices of 8 run 24h with only 2 MQTT reconnects over all. Pretty stable, I guess it must have something to do with my WiFi equipment.

But, at least, the debug device lost it's connection and I can provide the broker and ESP-Logs.
Please don't mind, in the ESP debug log the "MqttConnected" millis are always the same as "WifiConnected", c&p mistake.
disconnect_logs_20231218.zip

[sude22]

It is in a state that it could not reconnect to the broker again....
no_reconnect20231218.txt
[EDIT]
Not correct, after a while it connected again. Maybe you can see something in the logs....

[sude22]

After all, I really think it is the problem of my special WiFi setup, that sometimes blocks the traffic, even if the WiFi state is connected.

Anyway, thanks for all your time and help with that. I move on with my project, and in case I find out something new about this I let you know.