Handle null sys.status values from old firmware

Gen2 devices whose clock never synced report "unixtime": null in
sys.status. SystemComponent.from_raw_data passed that None to a
non-optional int field, Pydantic raised, the gateway fell back to the
Gen1 path, and the device became a 404 on the detail page. Local
firmware updates failed on the same read.

Null-guard the numeric and dict fields in SystemComponent the way
InputComponent already does, and make ComponentFactory fall back to the
base Component with a warning when a typed model rejects its data, so
one malformed component can no longer hide a reachable device.

Closes #88

Author: jfmlima

docs/plans/issue-88-null-sys-status-values.md

@@ -0,0 +1,79 @@
+# Issue 88: Null values in sys.status break device detail and local updates
+
+GitHub issue: https://github.com/jfmlima/shelly-manager/issues/88
+
+## Problem
+
+Gen2 devices on firmware 1.4.0 show two symptoms:
+
+1. The device detail page fails with "Device not found or unreachable" even though the device answers RPC calls.
+2. Firmware updates through the manager fail for the same devices.
+
+Devices on firmware 1.7.5 work. The firmware version is an indirect signal. The real trigger is that these devices report literal nulls in `sys.status` when their clock never synced (common on isolated IoT VLANs without NTP or internet access):
+
+```json
+"sys": {
+  "status": { "time": null, "unixtime": null, "uptime": 3539695, ... }
+}
+```
+
+## Root cause (verified with the payload attached to the issue)
+
+`SystemComponent.from_raw_data` uses `status.get("unixtime", 0)`. When the key is present with a null value, `get` returns `None`, not the default. The field is declared `unixtime: int` (non optional), so Pydantic raises `ValidationError`.
+
+- Crash site: `packages/core/src/core/domain/entities/components/system.py:45` (field declared at line 21).
+- The same pattern applies to `uptime`, `ram_size`, `ram_free`, `fs_size`, `fs_free`, `restart_required`, and `available_updates` at lines 38 to 44. Any of them can be null on old firmware.
+- `InputComponent.from_raw_data` already guards against this with `or <default>` (`packages/core/src/core/domain/entities/components/input.py:23-27`). `SystemComponent` does not.
+
+### Failure chain
+
+1. `DeviceStatus.from_raw_response` calls `ComponentFactory.create_component` per component and propagates the `ValidationError` (`packages/core/src/core/domain/entities/device_status.py:69`).
+2. `ShellyDeviceGateway.get_device_status` catches the generic exception and falls back to the legacy Gen1 gateway (`packages/core/src/core/gateways/device/shelly_device_gateway.py:166-181`). A Gen2 device does not serve the Gen1 `/status` endpoint, so the fallback returns `None`.
+3. The devices controller raises `DeviceNotFoundError`, which the API maps to a 404 with the message "Device not found or unreachable" (`packages/api/src/api/presentation/handlers.py`). The web detail page renders that message verbatim.
+4. The update path fails through the same read: `UpdateDeviceFromLocal.execute` raises `DeviceNotFoundError` when `get_device_status` returns `None` (`packages/core/src/core/use_cases/update_device_from_local.py:62-64`), and `GetLocalFirmwareReleases` fails the same way (`packages/core/src/core/use_cases/get_local_firmware_releases.py:41-43`).
+
+One bug therefore explains both symptoms.
+
+## Fix
+
+Both changes are in the core domain layer.
+
+### 1. Null-guard the fields in `SystemComponent.from_raw_data`
+
+In `packages/core/src/core/domain/entities/components/system.py`, apply the same guard `InputComponent` uses:
+
+```python
+uptime=status.get("uptime", 0) or 0,
+restart_required=status.get("restart_required", False) or False,
+ram_total=status.get("ram_size", 0) or 0,
+ram_free=status.get("ram_free", 0) or 0,
+fs_total=status.get("fs_size", 0) or 0,
+fs_free=status.get("fs_free", 0) or 0,
+available_updates=status.get("available_updates", {}) or {},
+unixtime=status.get("unixtime", 0) or 0,
+```
+
+### 2. Make `ComponentFactory.create_component` degrade instead of raise
+
+In `packages/core/src/core/domain/entities/components/factory.py:13`, catch `pydantic.ValidationError` from the typed model and fall back to `Component.from_raw_data(component_data)`, with a warning log. One malformed component must never turn a reachable device into a 404 and a dead update path. Without this, the next firmware quirk reproduces the whole issue.
+
+Note: `Component.from_raw_data` itself can raise if `key` is missing, but the factory is only called with dicts that carry a `key`, so guarding the typed-model call is enough.
+
+## Out of scope (file separately)
+
+- `Shelly.GetComponents` pagination: the gateway always requests `{"offset": 0}` and never loops to `total` (`packages/core/src/core/gateways/device/shelly_device_gateway.py:375-380` and `get_component_keys` at 184-198). Masked today because `DeviceStatus.from_raw_response` backfills missing components from `Shelly.GetStatus` keys, but backfilled components have empty config and `get_component_keys` silently misses page-two components. Separate issue.
+- Device-side internet updates failing on these devices: their own `available_updates` is empty because they cannot reach the Shelly cloud. Environmental, not a manager bug. The via-manager local update path is the designated answer, and this fix restores it.
+
+## Tests
+
+All in `packages/core/tests/unit/domain/entities/`:
+
+1. `test_components.py`: `SystemComponent.from_raw_data` with the exact `sys` component payload from the issue (`"time": null, "unixtime": null`) parses and yields `unixtime == 0`. Add a variant with nulls in the other guarded fields.
+2. `test_device_status.py`: `DeviceStatus.from_raw_response` with the issue's full `GetComponents` capture returns a `DeviceStatus` instead of raising. This is the regression test for the 404 chain.
+3. `test_components.py` (factory): a component dict whose status payload makes the typed model raise `ValidationError` returns a base `Component` with the right `key` instead of raising.
+
+## Verification
+
+- `make test-core`
+- `make lint`
+- Manual, if a 1.4.0 device is available: open the device detail page and run a via-manager update.

packages/core/src/core/domain/entities/components/factory.py

@@ -1,16 +1,29 @@
+import logging
 from typing import Any
 
+from pydantic import ValidationError
+
 from .base import Component
 from .registry import model_for
 
+logger = logging.getLogger(__name__)
+
 
 class ComponentFactory:
     @staticmethod
     def create_component(component_data: dict[str, Any]) -> Component:
         key = component_data.get("key", "")
         component_type = key.split(":")[0] if ":" in key else key
 
-        return model_for(component_type).from_raw_data(component_data)
+        try:
+            return model_for(component_type).from_raw_data(component_data)
+        except ValidationError as e:
+            logger.warning(
+                "Component %s did not match its typed model, keeping raw data: %s",
+                key,
+                e,
+            )
+            return Component.from_raw_data(component_data)
 
     @staticmethod
     def create_component_from_status(

packages/core/src/core/domain/entities/components/system.py

@@ -35,13 +35,13 @@ def from_raw_data(cls, component_data: dict[str, Any]) -> "SystemComponent":
             device_name=device_config.get("name"),
             mac_address=status.get("mac"),
             firmware_version=device_config.get("fw_id"),
-            uptime=status.get("uptime", 0),
-            restart_required=status.get("restart_required", False),
-            ram_total=status.get("ram_size", 0),
-            ram_free=status.get("ram_free", 0),
-            fs_total=status.get("fs_size", 0),
-            fs_free=status.get("fs_free", 0),
-            available_updates=status.get("available_updates", {}),
-            unixtime=status.get("unixtime", 0),
+            uptime=status.get("uptime", 0) or 0,
+            restart_required=status.get("restart_required", False) or False,
+            ram_total=status.get("ram_size", 0) or 0,
+            ram_free=status.get("ram_free", 0) or 0,
+            fs_total=status.get("fs_size", 0) or 0,
+            fs_free=status.get("fs_free", 0) or 0,
+            available_updates=status.get("available_updates", {}) or {},
+            unixtime=status.get("unixtime", 0) or 0,
             timezone=location_config.get("tz"),
         )

packages/core/tests/unit/domain/entities/test_components.py

@@ -7,6 +7,7 @@
     EMDataComponent,
     InputComponent,
     SwitchComponent,
+    SystemComponent,
     WebSocketComponent,
     WifiComponent,
     ZigbeeComponent,
@@ -856,3 +857,100 @@ def test_it_creates_em_component_from_status_only(self):
         assert isinstance(component, EMComponent)
         assert component.total_act_power == 500.0
         assert component.a_act_power == 200.0
+
+
+class TestSystemComponent:
+    def test_it_parses_sys_status_with_null_time_fields(self):
+        raw_data = {
+            "key": "sys",
+            "status": {
+                "mac": "E86BEAE5F208",
+                "restart_required": False,
+                "time": None,
+                "unixtime": None,
+                "uptime": 3539695,
+                "ram_size": 252308,
+                "ram_free": 132132,
+                "fs_size": 393216,
+                "fs_free": 98304,
+                "cfg_rev": 11,
+                "kvs_rev": 0,
+                "schedule_rev": 1,
+                "webhook_rev": 1,
+                "available_updates": {},
+                "reset_reason": 1,
+            },
+            "config": {
+                "device": {
+                    "name": "Flur SZ Rollo links",
+                    "mac": "xxxx",
+                    "fw_id": "20240726-114505/1.4.0-gb2aeadb",
+                    "discoverable": True,
+                    "eco_mode": False,
+                    "profile": "cover",
+                    "addon_type": None,
+                },
+                "location": {"tz": None, "lat": None, "lon": None},
+                "sntp": {"server": "xxxx"},
+                "cfg_rev": 11,
+            },
+            "attrs": {},
+        }
+
+        component = SystemComponent.from_raw_data(raw_data)
+
+        assert component.unixtime == 0
+        assert component.uptime == 3539695
+        assert component.mac_address == "E86BEAE5F208"
+        assert component.device_name == "Flur SZ Rollo links"
+        assert component.firmware_version == "20240726-114505/1.4.0-gb2aeadb"
+        assert component.timezone is None
+
+    def test_it_parses_sys_status_with_nulls_in_every_guarded_field(self):
+        raw_data = {
+            "key": "sys",
+            "status": {
+                "restart_required": None,
+                "unixtime": None,
+                "uptime": None,
+                "ram_size": None,
+                "ram_free": None,
+                "fs_size": None,
+                "fs_free": None,
+                "available_updates": None,
+            },
+            "config": {},
+            "attrs": {},
+        }
+
+        component = SystemComponent.from_raw_data(raw_data)
+
+        assert component.unixtime == 0
+        assert component.uptime == 0
+        assert component.restart_required is False
+        assert component.ram_total == 0
+        assert component.ram_free == 0
+        assert component.fs_total == 0
+        assert component.fs_free == 0
+        assert component.available_updates == {}
+
+
+class TestComponentFactoryFallback:
+    def test_it_falls_back_to_base_component_when_typed_model_rejects_data(
+        self, caplog
+    ):
+        raw_data = {
+            "key": "sys",
+            "status": {"uptime": "not-a-number"},
+            "config": {},
+            "attrs": {},
+        }
+
+        with caplog.at_level("WARNING"):
+            component = ComponentFactory.create_component(raw_data)
+
+        assert type(component) is Component
+        assert component.key == "sys"
+        assert component.component_type == "sys"
+        assert component.status == {"uptime": "not-a-number"}
+        assert any("sys" in record.message for record in caplog.records)