We have seen a few cases when the MQTT messages between the forge app and broker appear to stop flowing. No errors are logged on either side - but the MQTT messages just stop.
This is most apparent with Device Checkin messages. We have ops alerting in place for when Device status update log messages drop as a stop-gap solution. A restart of the forge app 'fixes' the connection.
We haven't been able to identify the root cause as it happens very rarely with no event to correlate it against.
The purpose of this task is to introduce an application-level heartbeat to the Comms connection.
It should:
- publish to a dedicated heartbeat topic every 30seconds with an incrementing payload.
- subscribe to the same topic and validate the heartbeats arrive
- if 2 heartbeats are missed, trigger a comms client reconnect.
The code should log these events clearly so we can monitor their occurrences.
We have seen a few cases when the MQTT messages between the forge app and broker appear to stop flowing. No errors are logged on either side - but the MQTT messages just stop.
This is most apparent with Device Checkin messages. We have ops alerting in place for when Device status update log messages drop as a stop-gap solution. A restart of the forge app 'fixes' the connection.
We haven't been able to identify the root cause as it happens very rarely with no event to correlate it against.
The purpose of this task is to introduce an application-level heartbeat to the Comms connection.
It should:
The code should log these events clearly so we can monitor their occurrences.