One Proxmox box, a rack of self-hosted services, and a small fleet of agents that watch, triage, rehearse, update, and repair — surfacing a human only where judgment actually matters. This page is generated from the system's own records and redeploys itself the moment something is fixed.
Generated from the system's own issue log and redeployed within a minute of every fix — caught, healed, or self-resolved with zero human in the loop, over a rolling fortnight.
Latest · this fortnight, automatically
A few times this month the system met something genuinely nasty. Each of these is the real arc — the symptom, what was actually wrong, what got researched, what was decided, and how it was proven. Click to open.
A household member reported a show that appeared in the library and looked available, but wouldn't play — while the download manager insisted there was no file for it at all. Three systems disagreeing about whether a file even existed.
The "episodes" were fake releases pulled from public trackers: Windows executables — self-extracting droppers — padded to gigabytes and named exactly like real episodes. Eleven of them had settled across the library. Nothing had ever played them, because nothing recognised them as media; they simply sat there, unnoticed.
Each file was inspected by signature (PE executables and a password-protected archive, not video), traced back through the grab-and-import pipeline to the specific trackers that served them, and matched against a known wave of fake-release malware that impersonates real release groups.
Quarantine all eleven with a guard that could only ever touch the malicious files, never a real video beside them; blocklist the bad sources so they cannot be re-fetched; and stand up a new daily sweep that quarantines this entire class of file automatically, regardless of which tracker it arrives from.
A full re-scan confirmed zero remaining and every genuine episode intact. The new sweep ships with a self-check wired into the deploy gate, so a broken detector fails the build rather than production.
The media server's database was throwing intermittent "database is locked" errors — brief failures that cascaded into two other services which depend on it.
The running version had no way to wait on a busy database; a momentary background write would make a simultaneous read fail outright. The retry-and-locking machinery only existed in a newer major version — but moving to it meant a one-way database migration on the primary media server the whole household relies on.
The root cause was confirmed against the upstream bug reports, and the newer version verified to actually address it (its own regressions checked too). Then, rather than trust any of that on the live box, the entire upgrade was replayed on a disposable, isolated clone of production.
The rehearsal earned its keep — it caught five breaking surprises before they could reach prod: a plugin that crashes the server on boot, a first-run step that deletes the library if a storage mount isn't ready, a required config flag, and two more. Every one was fixed on the clone first.
The clone came through the migration with library counts matching production exactly — down to the episode. Only then did the real upgrade run, and it landed clean: library and watch-history verified intact, and the original lock error confirmed silent afterward.
A routine build went red: the automated secret scanner had flagged a commit.
A monitoring component authenticated by putting its API key straight in the request URL — so every error it logged contained the key, and an automated commit that captures logs had carried one of those lines, key and all, into the repository. The key was still live.
The exact key was identified and confirmed still active, and the path traced end to end: key-in-URL → error log → log-capturing auto-commit → repo, slipping past the scanner because that particular commit had skipped the check.
Rotate the key immediately and revoke the old one — then fix the leak at the source, in depth, rather than patch one noisy component: scrub credentials at the log shipper before anything is stored, and again in the tool that writes to the repo. The fragile option (editing a third-party component) was deliberately rejected in favour of the layers that protect everything.
The old key was deleted and confirmed dead against the live service. A synthetic secret was then injected on purpose and verified to arrive redacted, proving the scrub works going forward — and the build went green.
Small, single-purpose, and boring by design — the interesting behaviour is in how they compose.
The system is deliberately not a black box that changes things on a whim. Four rules keep it trustworthy.
Every change is a branch, a review, a merge. Nothing is configured by hand and forgotten. The repository can rebuild the box from scratch.
Anything destructive is replayed on a throwaway clone of production first. The dangerous surprises get caught on a copy, not on the live system the household depends on.
Agents find root causes and propose. A human approves anything that changes state. Confident-but-wrong auto-fixes were tried once — and deliberately removed.
Secrets are scrubbed at the log shipper and again before the repository. A leaked credential is a bug to fix at the source, at multiple layers — never a fire drill.