Sender Health, Updates, and Recovery

The sender-backend dev baseline includes health snapshots, automatic update state, watchdog recovery, token refresh reporting, and device-side alerting. These checks help distinguish a stopped container, a stale image, an expired token, and a broken network path.

Health endpoints

From the Raspberry Shake, the sender backend listens on port 5001:

curl http://localhost:5001/health/network
curl http://localhost:5001/health/time
curl http://localhost:5001/health/resources
curl http://localhost:5001/health/sender-state
curl http://localhost:5001/health/metrics

/health/sender-state reports persisted operational state when available, including token refresh, update, watchdog, disk-alert, and remote-tunnel state. /health/metrics reports request counters, alert outcomes, and health trend history.

Update behavior

The host updater uses SENDER_BUNDLE_TAG (latest by default) and resolves image tags to digests before recreating the sender containers. A published image tag alone does not prove that a device has updated. Confirm the device’s update-state file and running container/image digest after the update cycle.

Existing devices may need the one-time bootstrap command:

bash <(curl -fsSL "https://raw.githubusercontent.com/UPRI-earthquake/sender-backend/dev/update.sh")

Normal operator commands remain available through sender-backend UPDATE_STACK and the installed update timer. Backend and frontend images should use the same bundle version when publishing a release.

Watchdog and alerts

sender-backend INSTALL_SERVICE installs the daily sender-backend-update.timer and periodic sender-stack-watchdog.timer. The watchdog checks container state and can restart stopped services. Sender alerts cover disk pressure, token refresh outcomes, and watchdog actions; they are sent to the restricted backend alert endpoint and are intended for administrators.

Recovery guidance

  1. Check docker ps -a and the health endpoints.

  2. Check sender-backend REMOTE_TUNNEL_STATUS if remote access is involved.

  3. Inspect /var/lib/upri-sender/update-state.json for the last update result.

  4. If the first update cycle only refreshed host scripts, allow the next cycle to run or perform the documented second-cycle recovery before reinstalling.

  5. Reinstall only when the service state or local files cannot be recovered; preserve device identity and link information where possible.

Do not report a station as updated merely because GHCR contains a new latest image.