Autonomous operations · homelab

The homelab that finds its own faults.

One Proxmox box running a rack of self-hosted services, with watchers that check its state on a schedule, diagnose what they find, and bring you a researched fix to approve. Below is every fix that has been made: the symptom, what actually went wrong, and how it was resolved. The page is generated from the system's own records and redeployed within a minute of each one.

230 fixes logged ·67 closed without a human ·refreshed 20 Aug 2026, 08:07 UTC
230
Fixes logged
67
Self-resolved
88
Root-caused
17
Ways it notices
What it can do

Five things, and it can prove each one.

Every number below is read from the system's own registries when this page is built — the action vocabulary from the code that enforces it, the services from the file that decides how loudly each one pages, the guests from a capture taken off the hypervisor six-hourly. Nothing here is typed by hand, because a feature list written by hand drifts away from the thing it describes and nobody notices.

17

ways it notices

Each watches something the others structurally cannot see — a dead indexer is a state, not an error line; a remote outage writes no local log at all.

backlog, config-export-monitor, dep-update, dependency-watch, failure-watcher, gap-audit, guest-resource, ha-watch, healthcheck-down, indexer-watch, media-malware-sweep, new-log-template, recording-watchdog, repo-audit, resource-watch, spec-check, stuck-acquisition

10

things it may do

A closed vocabulary. An action outside it cannot be selected, whatever the model proposes, and widening the list is a reviewed change to the code rather than a judgement made at three in the morning.

prune-build-cache, prune-images, restart-container, restart-guest, restart-service, set-cpulimit, set-memory, start-container, stop-container, vacuum-journal

28

rules it checks against itself

Each was paid for by a specific failure and fails on purpose before it is trusted. A check nobody has seen go red has not been tested — it has only been written.

0 health checks · 28 workflows

0

services it looks after

Graded by who notices when they stop, which decides how loudly a fault is reported and whether a change may be applied without asking.

0 reachable from outside · 0 household · 0 LAN-critical · 0 back-office

13

machines it can describe exactly

Full configuration, installed packages and mounts, captured from the hypervisor and compared against the committed copy. A change made by hand shows up as a failed check rather than as a surprise during the next restore.

plus 4 application settings that used to exist only as a note asking someone to remember them

What it runs

A rack's worth of services — on a single small box.

Media, home automation, networking, and the platform to run them — around a dozen self-hosted services, each right-sized to a tight CPU and memory budget and kept honest by a resource watchdog. Density by discipline, not by throwing hardware at it.

1 Proxmox host0 services every one CPU-limited + memory-floored right-sized weekly0 open inbound ports

Media

  • Streaming server + live TV & DVR
  • Self-service request manager
  • Movies · TV · indexers · subtitles · audiobooks
  • Quality profiles + failed-download cleanup

Home

  • Home automation — lighting, heating, presence
  • Camera AI + NVR (person / car detection)
  • Room-level routines & scenes
  • Phone notifications & live dashboards

Access & edge

  • Reverse proxy in front of every service
  • Zero-open-port public tunnel
  • User management + invites
  • Network-wide DNS + DHCP with ad-blocking

Data & safety

  • VPN-gated downloads with a kill-switch
  • Nightly encrypted offsite backups
  • Library malware sweep + quarantine
  • Quality / format enforcement

Observability

  • Central log aggregation + dashboards
  • Dead-man heartbeats on every scheduled job
  • Active reachability probes
  • Researched-fix push to your phone

How it ships

  • GitOps — the merge is the deploy
  • Secrets rendered at deploy, never in git
  • Smoke-tested behind an approval gate
  • One-command guided rollback
Every fix, in full

What it handled — and exactly how.

Each entry is a real fix, pulled from the system's own issue log. Click any one to read the symptom, what actually went wrong, and how it was resolved. Internal specifics are scrubbed; the reasoning is not.

Loading the feed…

The roster

One job each.

Each watcher checks one kind of state on a schedule and files what it finds as an issue. None of them applies a change.

log-triageclusters error logs into deduplicated, root-caused issues
failure-watcherturns silent failures into auto-closing issues
drift-checkpinned-vs-running container digests, every day
dep-updatewatches the version layer, proposes safe upgrades
resource-watchhost load & memory; right-sizes and alerts
renovatedependency & container-image bumps, digest-pinned
recording-watchdogre-arms a stalled DVR recording within 2 min
seerr-reconcilerevery request ends fulfilled — or flagged
autohealrestarts a container the moment it goes unhealthy
malware-sweepquarantines non-media payloads in the library
smoke-test gateblocks a bad deploy before it ever lands
dependency-watchevery service the box leans on, internal and remote, every 15 minutes
stuck-acquisitionan acquisition loop burning its own inventory while every step succeeds
indexer-watchindexer health, which is a state and so writes no error line
ha-watchHome Assistant config rot: orphaned entities, dead automations
triage agentreads the issue, diagnoses it against live state, proposes a declared remedy
How it decides

Autonomous where it’s safe. Gated where it isn’t.

Six rules decide what an agent may change on its own, and what has to wait for a person.

01

Git is the source of truth

Every change is a branch, a review, a merge. Nothing is configured by hand and forgotten. The repository can rebuild the box from scratch.

02

Rehearse before prod

Anything destructive is replayed on a throwaway clone of production first. The dangerous surprises get caught on a copy, not on the live system the household depends on.

03

Diagnose, don’t blind-fix

Agents find root causes and propose. A human approves anything that changes state. Confident-but-wrong auto-fixes were tried once — and deliberately removed.

04

Defense in depth

Secrets are scrubbed at the log shipper and again before the repository. A leaked credential is a bug to fix at the source, at multiple layers — never a fire drill.

05

Gated by blast radius

Every change is classed by how much it can break and who it touches. A back-office tweak lands instantly; anything that reaches the network, the household, or an outside user needs a live preflight and an explicit go-ahead — and fails closed if a check is unsure.

06

Right-sized on purpose

One small box, not a cluster. Every service runs on a deliberately tight CPU and memory budget, rebalanced on a schedule — capacity is a decision, not an afterthought. Over-building for scale it will never have is treated as its own failure mode.

How it reaches you

A researched fix reaches your phone, and one tap records the decision.

The difficulty is the last mile: a problem that needs a person has to reach one, with enough context to decide in seconds, and silence has to mean healthy rather than unheard. The tap records intent; the change still goes through review.

01

Detect

Agents cluster errors, watch resources, probe reachability, and track versions — turning noise into deduplicated, root-caused issues.

02

De-noise

A one-off blip auto-closes as a transient; a fixed issue reopens only if it comes back sustained. The queue stays honest — no flapping, no silent pile-up.

03

Research the fix

Before anything reaches you, per-tool experts ground the fix in the official docs and the community grounds it in real-world experience: root cause, live state, upstream behaviour. What arrives is a recommendation with its evidence attached.

04

Escalate

The researched fix lands on your phone with one-tap Approve / Reject. A daily digest arrives even when everything’s green — so a quiet day is proof the loop is alive, and a missing digest is itself the alarm.

05

Apply — gated

Approve records the decision and routes it into the reviewed, rehearsed path. It never auto-applies a change on its own — the human stays in the loop where it counts.

The actual shape of a decision push. The button carries a one-time signed token — never a credential.

Researched
A root-caused recommendation, not a raw alert to go investigate.
Silence = healthy
A daily heartbeat makes "no news" trustworthy instead of ambiguous.
Human-gated
One tap records intent; the apply still goes through review + rehearsal.
Hardened
Signed, token-free taps, on a channel that watches its own health.