Vistralio — DevOps / Internal Runbook

Internal-facing operational documentation. Not intended for end users.

Service topology

Unit	Purpose
vistralio-connect.service	Main API, auth, settings, admin/business logic (:8000)
vistralio-media.service	Stream proxy, snapshots, edge WebSocket, bridge tunnel lifecycle (:8200)
vistralio-core.service	Built SPA shell and static frontend assets (:8300)
vistralio-worker.service	Detection loops, recorder processes, event creation, retention work
vistralio-ls.service	Standalone license server (:8100)
vistralio-mqtt.service	Embedded MQTT broker
vistralio-watchdog.service / .timer	Health watchdog and storage cleanup pass
sentinel-update-check.service / .timer	Automatic update discovery

nginx routing

saas.vistralio.co.uk (UI + API — browser access)
  /api/streams/* /api/edge/ws  → 127.0.0.1:8200  (vistralio-media)
  /api/*                       → 127.0.0.1:8000  (vistralio-connect)
  /                            → 127.0.0.1:8300  (vistralio-core)

connect.vistralio.co.uk (primary external API entry point)
  /                            → 302 /api/
  /api/streams/* /api/edge/ws  → 127.0.0.1:8200
  /api/*                       → 127.0.0.1:8000
  everything else              → 404

updates.vistralio.co.uk        → /api/updates/* only → 127.0.0.1:8000
license-server.vistralio.co.uk → 127.0.0.1:8100

Filesystem layout

/opt/vistralio/
├── backend/          # Python source
├── frontend/         # React source + dist/
├── docs/             # Documentation
├── edge-bridge/      # Edge bridge agent and installers
└── venv/             # Python virtualenv

/etc/vistralio/vistralio.yaml  # Boot-time config (secrets, paths, JWT)

/var/lib/vistralio/
├── recordings/       # Segmented MP4 files
├── snapshots/        # Event snapshots
├── clips/            # Event clips
├── exports/          # Download archives
├── updates/          # Update packages
└── branding/         # Uploaded logos/favicons

/var/log/vistralio/
├── connect.log       # vistralio-connect (API)
├── media.log         # vistralio-media
├── worker.log        # vistralio-worker
├── mqtt.log
└── license.log

/dev/shm/vistralio/   # Bridge camera live frames (tmpfs, in-memory)
  cam_{id}.jpg        # Latest JPEG written by detection worker (atomic rename)

/root/sentinel/       # Installer/update tooling and source snapshots

Configuration model

Boot-time config — /etc/vistralio/vistralio.yaml:

• Database URL, JWT secret, Fernet encryption key, storage paths, MQTT settings
• Changing this file requires service restarts

Runtime config — settings table in MariaDB:

• Everything editable from the UI (branding, DNS, SMTP, schedules, etc.)
• DNS hostnames: dns.current (web UI), dns.api_hostname (connect. endpoint)

Backup priorities

mysqldump --single-transaction sentinel | gzip > /backup/sentinel-$(date +%F).sql.gz
tar czf /backup/sentinel-config-$(date +%F).tar.gz /etc/vistralio /var/lib/vistralio/branding

Health checks

curl -s http://127.0.0.1:8000/api/health
curl -s http://127.0.0.1:8200/health
curl -s http://127.0.0.1:8300/health
curl -s http://127.0.0.1:8100/license-server

Service checks

systemctl status vistralio-connect vistralio-media vistralio-core vistralio-worker vistralio-ls vistralio-mqtt

Log tailing

tail -F /var/log/vistralio/connect.log
tail -F /var/log/vistralio/media.log
tail -F /var/log/vistralio/worker.log

Deployment workflow

Preferred path:

cd /root/sentinel
sudo ./scripts/update.sh

Manual/low-level path:

cd /opt/vistralio
sudo /opt/vistralio/venv/bin/pip install -r backend/requirements.txt
( cd frontend && npm install && npm run build )
sudo systemctl restart vistralio-connect vistralio-media vistralio-core vistralio-worker

Common tasks

Restart the full stack

sudo systemctl restart vistralio-connect vistralio-media vistralio-core vistralio-worker

Restart only media/bridge handling

sudo systemctl restart vistralio-media

Reset the admin password

sudo -u vistralio VISTRALIO_CONFIG=/etc/vistralio/vistralio.yaml \
  /opt/vistralio/venv/bin/python -c "
import sys; sys.path.insert(0, '/opt/vistralio/backend')
from app.core.db import SessionLocal
from app.core.security import hash_password
from app.models import User
db = SessionLocal()
u = db.query(User).filter(User.username == 'admin').one()
u.hashed_password = hash_password('NEW_PASSWORD')
db.commit()
print('done')
"

Apply a web hostname change

After staging and promoting via the DNS settings page:

sudo /opt/vistralio/scripts/apply-dns.sh

Rewrites nginx server_name, re-issues Let's Encrypt if requested, reloads nginx.

Change the API hostname (connect.)

In the UI: Settings → DNS → API / Connect hostname → Set.

Updates dns.api_hostname in the settings table. Edge router enrollment commands in the admin UI update immediately. No nginx reload needed.

Check disk pressure

df -h
du -sh /var/lib/vistralio/recordings/*/ 2>/dev/null | sort -h

Inspect bridge camera live frames

ls -la /dev/shm/vistralio/
# cam_{id}.jpg files should have recent mtimes if detection is running

Camera worker internals

• Recording and detection each run as multiprocessing.Process children (isolated crash domains)
• camera_watcher() polls DB every 30 s and starts loops for newly-added cameras without restart
• Bridge cameras: worker writes latest JPEG to /dev/shm/vistralio/cam_{id}.jpg (atomic)
• Direct cameras: media service runs FFmpeg subprocess per camera (scales to max 1280 px, MJPEG output)
• RTSP connection limit: most bridge cameras allow max 2 simultaneous connections; the shared-file approach means live view clients never open a 3rd connection

Edge bridge connection URL

New edge routers connect to:

wss://connect.vistralio.co.uk/api/edge/ws

The install command shown in Admin → Setup → Edge Routers always reflects the current dns.api_hostname setting. Existing edge routers connected to wss://saas.vistralio.co.uk/api/edge/ws continue to work — both hostnames route to the same vistralio-media endpoint.

On-call notes

"Vistralio is down"

1. Check vistralio-connect, vistralio-media, vistralio-core, vistralio-worker, vistralio-ls
2. Check MariaDB (systemctl status mariadb)
3. Check disk pressure
4. Check nginx (nginx -t && systemctl status nginx)

"Bridge cameras stopped"

1. Check vistralio-media (journalctl -u vistralio-media -n 50)
2. Check Admin → Setup → Edge Routers or GET /api/edge-routers
3. Check vistralio-edge on the remote bridge device
4. Verify bridge is connected to wss://connect.vistralio.co.uk/api/edge/ws
5. Check /dev/shm/vistralio/ — frames must be updating if detection runs

"Alarms aren't firing"

1. Check vistralio-worker (tail -f /var/log/vistralio/worker.log)
2. Confirm the device has a detect path and detection enabled
3. Confirm the target object label is in the object list
4. Check zones/masks and schedule suppression
5. Verify events appear in DB before blaming MQTT/SMTP

"Playback / export fails"

1. Check recording segment availability via /api/recordings/segments
2. Check filesystem pressure
3. Check FFmpeg errors in worker.log

Security notes

• JWT secret in /etc/vistralio/vistralio.yaml — never commit to git
• Fernet key tag b":sentinel-secret-v1" must not change — changing it silently breaks decryption of all stored camera passwords, SMTP credentials, and license keys
• Edge bridge enrollment tokens are sensitive — rotate if a device is lost or compromised
• Bridge device fingerprints are pinned after first connect; token re-use on a different device is refused until an admin rotates the token
• Tenant isolation is enforced at the SQLAlchemy query layer; admin overrides are explicit

Known architectural gaps

• SQLAlchemy create_all instead of full Alembic migration workflow
• No Prometheus/Grafana exporter built in yet
• Worker sharding is a manual concern for very large installs (many cameras per server)