Give the database and firmware cache a real home (#73)

* feat: Give the images an explicit data directory

The database and the firmware cache both defaulted to a path relative to
the working directory, so they landed at /app/data inside the image tree,
which no deployment surface mounts. Point both at /data instead, and
create it owned by the app user so a freshly provisioned volume is
writable after the image drops to uid 10001.

The firmware cache has its own setting and is not derived from the data
directory, so it needs its own variable; setting only SHELLY_DATA_DIR
would move the database and leave the cache behind.

The dev images get the same two variables so every image in the repo
declares the same contract.

* fix: Keep the compose database across a teardown

The api service mounted only the two source trees, so the database lived
in the container's own filesystem and every docker compose down took it
with it. Give it a named volume at the data directory the image now
declares, which a plain down keeps and only down -v removes.

The dev image already sets SHELLY_DATA_DIR and SHELLY_FIRMWARE_DIR, so
the service needs the volume and nothing else.

* docs: Say where the data lives

The deployment examples showed no volume and the configuration section
described only browser localStorage, so nothing told a reader that the
database and the firmware cache need somewhere to live. Name the
directory and the two variables that set it, and mount it in every
docker run and compose example, including the quick start printed into
every GitHub release.

The API reference gave SHELLY_FIRMWARE_DIR the package default, which
stopped being what the image uses, and the wrong default host; both are
corrected. A bind mounted host directory is called out separately, since
it keeps its own ownership and the API runs as uid 10001.

Anyone who mounted the old /app/data path by hand gets a note on where
their database went, and the examples that were missing SHELLY_SECRET_KEY
now set it, since without it the container exits at startup.

Author: jfmlima

.github/workflows/release.yml

@@ -38,8 +38,12 @@ jobs:
 
             ## 🚀 Quick Start
             ```bash
-            # Run API server
-            docker run -p 8000:8000 ${{ env.IMAGE_PREFIX }}-api:${{ env.IMAGE_TAG }}
+            # Run API server. The -v keeps the database and firmware cache;
+            # generate the key with: openssl rand -base64 32 | tr '+/' '-_'
+            docker run -p 8000:8000 \
+              -e SHELLY_SECRET_KEY="your-generated-key" \
+              -v shelly-manager-data:/data \
+              ${{ env.IMAGE_PREFIX }}-api:${{ env.IMAGE_TAG }}
 
             # Run CLI tool
             docker run --rm ${{ env.IMAGE_PREFIX }}-cli:${{ env.IMAGE_TAG }} --help

DEVELOPMENT.md

@@ -41,6 +41,8 @@ docker compose down cli
 
 Both source trees are bind-mounted into that container, so edits on the host take effect without a rebuild.
 
+The api service keeps its database and firmware cache in a named volume mounted at `/data`, so a plain `down` leaves both in place. `docker compose down -v` removes the volume too, which is how you get back to an empty database.
+
 ### Option 2: Local Development
 
 1. **Install uv:**

README.md

@@ -106,6 +106,8 @@ services:
       # Defaults shown; set SHELLY_BACKUP_SCHEDULER_ENABLED=false to disable.
       - SHELLY_BACKUP_SCHEDULER_ENABLED=true
       - SHELLY_BACKUP_POLL_INTERVAL_SECONDS=60
+    volumes:
+      - shelly-manager-data:/data
 
   shelly-manager-web:
     image: ghcr.io/jfmlima/shelly-manager-web:latest
@@ -115,6 +117,9 @@ services:
       - VITE_BASE_API_URL=http://localhost:8000
     depends_on:
       - shelly-manager-api
+
+volumes:
+  shelly-manager-data:
 ```
 
 **With Traefik**:
@@ -132,6 +137,8 @@ services:
       # Defaults shown; set SHELLY_BACKUP_SCHEDULER_ENABLED=false to disable.
       - SHELLY_BACKUP_SCHEDULER_ENABLED=true
       - SHELLY_BACKUP_POLL_INTERVAL_SECONDS=60
+    volumes:
+      - shelly-manager-data:/data
     labels:
       - "traefik.enable=true"
       - "traefik.http.routers.shelly-manager-api.rule=Host(`shelly-manager-api.your.domain`)"
@@ -152,6 +159,9 @@ services:
       - "traefik.http.routers.shelly-manager-web.service=shelly-manager-web"
       - "traefik.http.routers.shelly-manager-web.entrypoints=web"
       - "traefik.http.services.shelly-manager-web.loadbalancer.server.port=8080"
+
+volumes:
+  shelly-manager-data:
 ```
 
 **Home Assistant Add-on**
@@ -202,15 +212,22 @@ docker run --rm -it \
 ```bash
 docker run -p 8000:8000 \
   -e SHELLY_SECRET_KEY="your-generated-key" \
+  -v shelly-manager-data:/data \
   ghcr.io/jfmlima/shelly-manager-api:latest
 ```
 
+The `-v` is what keeps your data. The image writes everything it stores to `/data`, and without a volume there that directory belongs to the container and goes away with it. A named volume like the one above picks up the right ownership from the image; if you bind mount a host directory instead, run `chown 10001:10001` on it first, because the API runs as that user and cannot write to a root owned directory.
+
 ### Configuration
 
 Shelly Manager is zero-configuration by default. All scan and management parameters are provided at runtime via the Web UI, CLI flags, API parameters or ENV variables. This ensures flexibility and removes the need for managing static configuration files.
 
 For persistent storage of discovered devices in the Web UI, the application leverages browser localStorage. For API-based integrations, the client is responsible for maintaining device lists.
 
+**Server-side state lives in `/data`.** Device credentials, configuration backups, backup schedules and provisioning profiles are kept in a SQLite database at `/data/data.db`, and downloaded firmware bundles under `/data/firmware`. The API and CLI images set `SHELLY_DATA_DIR=/data` and `SHELLY_FIRMWARE_DIR=/data/firmware`; mount a volume at `/data` and everything survives a restart, or point the two variables somewhere else if you prefer another path. Outside a container both default to `./data`, relative to the working directory. The Unraid image and the Home Assistant add-on set their own paths and handle this for you.
+
+Earlier images wrote to `/app/data` instead. If you carried that over with a mount of your own, such as `-v ./data:/app/data`, your database will look reset after this upgrade: it is still at the old path, and moving `data.db` and `firmware/` into the new volume brings it back.
+
 **Scheduled backups** run on the API server itself. When `SHELLY_BACKUP_SCHEDULER_ENABLED` is `true` (the default), an in-process poller captures backups for any due schedules every `SHELLY_BACKUP_POLL_INTERVAL_SECONDS` (default 60). Because the timer lives in-process, run the API as a single worker (the default); see the [API README](packages/api/README.md) for the full setting reference.
 
 **Local firmware updates** let a device that cannot reach the internet still be updated. Ask for one with `"source": "local"` on the update endpoint, or `--source local` from the CLI: the manager downloads the official firmware from Shelly once, keeps it, and tells the device to fetch it from the manager instead. The same copy serves every device running that model, so only the manager needs internet access.

docker-compose.yml

@@ -21,6 +21,7 @@ services:
       # internet update source works.
       SHELLY_FIRMWARE_ADVERTISED_BASE_URL: "${SHELLY_FIRMWARE_ADVERTISED_BASE_URL:-}"
     volumes:
+      - shelly-data:/data
       - ./packages/api/src:/app/packages/api/src:rw
       - ./packages/core/src:/app/packages/core/src:rw
     ports:
@@ -66,3 +67,6 @@ services:
       - cli
     entrypoint: ["tail", "-f", "/dev/null"]
     restart: unless-stopped
+
+volumes:
+  shelly-data:

packages/api/Dockerfile

@@ -27,6 +27,9 @@ COPY --from=builder --chown=app:app /app/packages/api /app/packages/api
 ENV PATH="/app/.venv/bin:$PATH"
 ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
 ENV HOST=0.0.0.0 PORT=8000 DEBUG=false
+ENV SHELLY_DATA_DIR=/data SHELLY_FIRMWARE_DIR=/data/firmware
+
+RUN mkdir -p /data/firmware && chown -R app:app /data
 
 WORKDIR /app
 USER app

packages/api/README.md

@@ -443,7 +443,8 @@ curl -X POST "http://localhost:8000/api/backups/1/restore" \
 | `SHELLY_BACKUP_SCHEDULER_ENABLED` | `true` | Run the in-process scheduled-backup poller |
 | `SHELLY_BACKUP_POLL_INTERVAL_SECONDS` | `60` | How often the scheduler checks for due backups |
 | `SHELLY_FIRMWARE_ADVERTISED_BASE_URL` | (none) | URL devices use to reach this API, e.g. `http://192.168.1.50:8000`. Required for local updates; it cannot be guessed |
-| `SHELLY_FIRMWARE_DIR` | `./data/firmware` | Where downloaded firmware bundles are kept |
+| `SHELLY_DATA_DIR` | `/data` in the image, `./data` otherwise | Directory holding the SQLite database (`data.db`) |
+| `SHELLY_FIRMWARE_DIR` | `/data/firmware` in the image, `./data/firmware` otherwise | Where downloaded firmware bundles are kept |
 | `SHELLY_FIRMWARE_INDEX_URL` | `https://updates.shelly.cloud/update` | Where published firmware is looked up, by the app name a device reports |
 | `SHELLY_FIRMWARE_ALLOWED_DOWNLOAD_HOSTS` | `shelly.cloud` | Comma separated hosts firmware may be downloaded from, matched exactly or as a parent domain and re-checked on every redirect. `*` accepts any host |
 | `SHELLY_FIRMWARE_VERIFY_SSL` | `false` | Verify TLS when talking to the firmware index and CDN. Off by default because Shelly signs those hosts with a private CA absent from public trust stores; devices verify firmware signatures themselves |
@@ -453,6 +454,14 @@ stored in the local database (`{data_dir}/data.db`); rotating the key makes exis
 undecryptable. Set `SHELLY_BACKUP_SCHEDULER_ENABLED=false` to turn off automated backups while
 still managing schedules through the API.
 
+Both directories are created on first use and both need to survive a restart, so mount a
+volume at `/data` when running this image. Without one they belong to the container, and
+replacing it resets the database and empties the firmware cache. A named volume inherits the
+image's ownership; a bind mounted host directory does not, so `chown 10001:10001` it before
+first start or the API cannot write to it. Packaged builds set their own path: the Unraid
+image uses `/config`, the Home Assistant add-on uses the add-on data volume, and neither
+needs a mount configured by hand.
+
 ## Docker Deployment
 
 ### Basic Deployment
@@ -463,6 +472,8 @@ docker run -d \
   -p 8000:8000 \
   -e HOST=0.0.0.0 \
   -e PORT=8000 \
+  -e SHELLY_SECRET_KEY="your-generated-key" \
+  -v shelly-manager-data:/data \
   ghcr.io/jfmlima/shelly-manager-api:latest
 ```
 
@@ -478,7 +489,13 @@ services:
       - HOST=0.0.0.0
       - PORT=8000
       - DEBUG=false
+      - SHELLY_SECRET_KEY=your-generated-key
+    volumes:
+      - shelly-manager-data:/data
     restart: unless-stopped
+
+volumes:
+  shelly-manager-data:
 ```
 
 ### Health Check

packages/api/dev.Dockerfile

@@ -15,6 +15,7 @@ RUN uv sync --frozen --package shelly-manager-api
 ENV PATH="/app/.venv/bin:$PATH"
 ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
 ENV HOST=0.0.0.0 PORT=8000 DEBUG=true
+ENV SHELLY_DATA_DIR=/data SHELLY_FIRMWARE_DIR=/data/firmware
 
 EXPOSE 8000
 

packages/cli/Dockerfile

@@ -24,6 +24,9 @@ COPY --from=builder --chown=app:app /app/packages/cli /app/packages/cli
 
 ENV PATH="/app/.venv/bin:$PATH"
 ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
+ENV SHELLY_DATA_DIR=/data SHELLY_FIRMWARE_DIR=/data/firmware
+
+RUN mkdir -p /data/firmware && chown -R app:app /data
 
 WORKDIR /app
 USER app

packages/cli/dev.Dockerfile

@@ -13,5 +13,6 @@ RUN uv sync --frozen --package shelly-manager-cli
 
 ENV PATH="/app/.venv/bin:$PATH"
 ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
+ENV SHELLY_DATA_DIR=/data SHELLY_FIRMWARE_DIR=/data/firmware
 
 ENTRYPOINT ["shelly-manager"]