Autonomous operations · homelab

The homelab that runs itself.

One Proxmox box, a rack of self-hosted services, and a small fleet of agents that watch, triage, rehearse, update, and repair — surfacing a human only where judgment actually matters. This page is generated from the system's own records, and refreshes on its own.

All systems nominal ·12 agents on watch ·1 awaiting a human ·refreshed 16 Jul 2026
drift success failure-watcher success log-triage success dep-update success resource-watch success
1
Proxmox host
~14
Services
12
Autonomous agents
9 / 14d
Issues self-resolved
The event stream

What it handled without being asked.

The top of this list is generated live from the system's own issue log — caught, healed, or self-resolved in the last fortnight. Below it, a few signature moments worth keeping.

Latest · this fortnight, automatically

  1. failure-watcherSelf-resolved

    CI failing: CI

  2. resource-watchSelf-resolved

    Resource: sustained high host load

  3. failure-watcherSelf-resolved

    CI failing: CI

  4. failure-watcherSelf-resolved

    CI failing: triage-postmortem

  5. failure-watcherSelf-resolved

    CI failing: deploy media stacks

  6. failure-watcherSelf-resolved

    Healthcheck DOWN: homelab/backup/healthchecks-epg-ping-url

  7. resource-watchSelf-resolved

    Resource: sustained high host load

  8. resource-watchSelf-resolved

    Resource: high swap usage

Signature moments

  1. media-malware-sweepQuarantined

    Eleven pieces of malware were found hiding in the media library — disguised as TV episodes.

    Fake “releases” — Windows executables padded to gigabytes and named like episodes — had settled in from public trackers. A library scan identified all eleven by signature and quarantined them. None had ever been recognised as media, so nothing was playing them; they were simply sitting there, unnoticed, until the sweep looked.

  2. clone rehearsalRehearsed → Shipped

    A risky major upgrade was rehearsed on a disposable clone before it touched production.

    A media-server version jump was first replayed against a throwaway copy of the real box. The rehearsal caught five breaking surprises — a plugin that crashes the server on boot, a migration that wipes the library if a mount isn’t ready — every one fixed before a single byte reached production. Then it shipped clean, full library and watch-history intact and verified.

  3. gitleaks + rotationRemediated

    A live credential leaked into a machine-written commit — and was killed within the hour.

    An automated log-capture accidentally carried an API key into the repo. The secret scanner flagged it, the key was rotated and revoked, and redaction was added at two independent layers so a credential can never reach the logs or the repository again. Root cause fixed, not just the symptom.

  4. dep-updateProposed

    A kernel CVE landed upstream. It was flagged, risk-assessed, and queued — for a human.

    The update scout saw the advisory, weighed it against the running version and the project’s maturity policy, and proposed the patch. Then it stopped and waited: a reboot of the whole house is not a decision an agent gets to make.

The roster

Twelve agents, each with one job.

Small, single-purpose, and boring by design — the interesting behaviour is in how they compose.

log-triageclusters error logs into deduplicated, root-caused issues
failure-watcherturns silent failures into auto-closing issues
drift-checkpinned-vs-running container digests, every day
dep-updatewatches the version layer, proposes safe upgrades
resource-watchhost load & memory; right-sizes and alerts
renovatedependency & container-image bumps, digest-pinned
recording-watchdogre-arms a stalled DVR recording within 2 min
seerr-reconcilerevery request ends fulfilled — or flagged
autohealrestarts a container the moment it goes unhealthy
malware-sweepquarantines non-media payloads in the library
smoke-test gateblocks a bad deploy before it ever lands
healthcheck-triagewatches the watchers, and pages when one goes quiet
How it decides

Autonomous where it’s safe. Gated where it isn’t.

The system is deliberately not a black box that changes things on a whim. Four rules keep it trustworthy.

01

Git is the source of truth

Every change is a branch, a review, a merge. Nothing is configured by hand and forgotten. The repository can rebuild the box from scratch.

02

Rehearse before prod

Anything destructive is replayed on a throwaway clone of production first. The dangerous surprises get caught on a copy, not on the live system the household depends on.

03

Diagnose, don’t blind-fix

Agents find root causes and propose. A human approves anything that changes state. Confident-but-wrong auto-fixes were tried once — and deliberately removed.

04

Defense in depth

Secrets are scrubbed at the log shipper and again before the repository. A leaked credential is a bug to fix at the source, at multiple layers — never a fire drill.