IoT client dies permanently after extended network outage

[cameronsjo]

Problem

After an extended network outage (hours to days), the plugin can go permanently silent: HomeKit shows the devices but no commands or state updates flow, and it never recovers without a Homebridge restart. In my case the plugin sat dead for five days after a multi-hour internet outage until I restarted it.

Root cause (four compounding failure modes)

1. createNewIotClient ends the only client before creating its replacement. If creation then fails mid-outage (token fetch with no internet), no MQTT client exists at all, the AWS SDK's reconnect loop died with the old client, and nothing ever retries — this is the permanent-death path.
2. The debounceTime-based credential refresh disarms itself on failure. The refresh cycle re-arms only when a new client is emitted onto the subject; a single failed refresh emits nothing, so the periodic refresh silently stops forever.
3. The SDK's built-in reconnect reuses the original signed wss URL. Its Cognito credentials expire (~1 hour), so once an outage outlives the signature, the SDK can retry forever without ever succeeding — recovery needs a new client signed with fresh credentials, which nothing triggers.
4. After an SDK-internal reconnect, the device shadow is never re-fetched (registering the same thing twice errors), so state can be stale even when the connection does come back.

There is also a related boot-time race: the connect listener is attached behind previousUpdatePromise, so if the MQTT client connects before the listener lands, the connect event is missed and device commands block.

Suggested fix

I have a working overhaul of the client lifecycle and will open a PR referencing this issue:

• Create the replacement client before ending the previous one, so a failed creation leaves the old client (and the SDK's own reconnect loop) alive.
• A re-entrant recreateIotClient that retries failed creation with 30 s-doubling backoff (5-minute cap), forever — creation failure is never terminal.
• A dead-connection watchdog: close/offline arm a 10-minute timer (cleared on connect); if the SDK's built-in reconnect can't recover in that window, recreate the client with fresh credentials. Events from ended, replaced clients are guarded out of driving recovery.
• Credential rotation on a plain interval instead of debounceTime, so a failed refresh retries instead of ending the cycle.
• Shadow re-fetch on SDK-internal reconnects (register once per client; later connects get() the shadow fresh).
• Attach the connect listener synchronously to fix the boot-time race.

Verification

The boot path (client creation, registration, commands, state updates) is live-verified on a physical Hatch Restore. The outage paths are design-verified — they are hard to reproduce on demand, but each failure mode above is directly traceable in the current source.