---
title: 'The Ghost in the State File: How My AI Squad Killed Our Last Cloud Server'
description: 'What happens when a routine local photo vault health check turns into a full forensic audit of your cloud bills, only to discover that your Terraform state file has been hallucinating an entire Helsinki datacenter rack that ceased existing months ago? The squad convenes to delete 220MB of binary baggage and declare total local sovereignty.'
pubDate: 2026-11-01
tags: ['ai-agents', 'infrastructure', 'devops', 'finops', 'terraform', 'kubernetes', 'sovereignty']
north_stars: ['simplicity-over-abstraction', 'cost-efficiency', 'observability-first', 'data-sovereignty']
lang: 'en'
draft: false
type: 'deep-dive'
featured: true
---
# The Ghost in the State File: How My AI Squad Killed Our Last Cloud Server
It began with the most unassuming, domestic sentence an engineer can type into a terminal prompt on a Sunday morning:
> *"I have a local Linux machine on my network where I have my Immich photos."*
No grand architectural charter. No multi-quarter migration roadmap. Just a human checking in on a mini-PC tucked away on a shelf, hosting several hundred gigabytes of family photos, cat videos, and machine-learning facial recognition embeddings.
Within ninety minutes, that single prompt had unraveled into a full-scale forensic autopsy of our infrastructure fleet. Before the morning was over, we had uncovered a wandering DHCP IP, verified a €31/month cloud migration, caught our own Terraform state file red-handed in a complete hallucination, and permanently deleted **220 megabytes** of compiled binary baggage—leaving our entire setup completely sovereign, on-premise, and blissfully free of rented cloud compute.
Here is the anatomy of how an autonomous AI squad, a patient teacher persona, and a five-word human reality check finally buried our last cloud server.
---
## Act I: The Case of the Wandering IP
Every infrastructure war story starts with a documentation lie.
In our repository’s runbooks, the single-node local K3s cluster node (`andreas-10vhs1p500`) was neatly cataloged as living at static LAN address `192.168.x.16`. It had been running Immich v2.7.5 faithfully.
Naturally, when the AI agent pinged the server, the terminal returned pure, unadulterated silence:
```bash
$ ping -c 2 -W 2 192.168.x.16
--- 192.168.x.16 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss
```
```html
@sre
"Pings are dropping into the void. Node is offline. Disaster recovery drill initiated. Shall I spin up the backup container stack?"
```
```html
@kurt
"Calm down, SRE. The machine didn’t vaporize into the ether. It’s sitting on a physical desk six feet away. Check ARP. Someone probably tripped over a power cable or DHCP leased a new number while you were polishing your runbooks."
```
Kurt was, predictably, right. Instead of panicking or declaring a Sev-1 outage, the agent dispatched a lightning-fast async Python socket scanner across the entire `/24` subnet. In less than 400 milliseconds, port `6443` (the Kubernetes API) and port `80` lit up like a Christmas tree:
```text
192.168.x.14:6443 OPEN
192.168.x.14:80 OPEN
192.168.x.14:22 OPEN
```
A quick `curl http://192.168.x.14/api/server/ping` returned `{"res":"pong"}`. The machine was wide awake. It had rebooted four minutes earlier, and the local router's DHCP server had casually handed it `.14` instead of `.16`.
```html
@infrastructure-lead
"We have architectural drift. Do NOT hardcode a static IP in netplan on the node. That creates unmanaged local configuration debt. Anchor the fix to [IX. Simplicity Over Abstraction]."
```
Instead of modifying network configuration files on the Ubuntu host, we went straight to the root of authority: our **MikroTik RouterOS** gateway.
We matched the physical Ethernet MAC address (`6c:4b:90:ca:f1:d1`), bound it to a permanent static DHCP lease in RouterOS for `.16`, applied `nmcli device reapply`, and the server snapped back to `.16` instantly.
Four pods—`immich-server`, `immich-postgres`, `immich-machine-learning`, and `immich-valkey`—were sitting at 100% health, happily indexing 210 GB of photos on an external drive.
---
## Act II: The Haunted Datacenter
With the local photos safe, the human posed a deceptively innocent question:
> *"We have a backup to Hetzner. Can we check if we have anything else there besides the backup?"*
The squad dug into our infrastructure repositories, Terraform files, and historical invoices.
What we found looked like a respectable, modern European hybrid-cloud footprint:
```mermaid
graph TD
subgraph HetznerCloud ["Hetzner Cloud (Germany & Finland)"]
SB["Hetzner Storage Box (u528497)
Encrypted Restic Vault"]
VPS1["VPS 1: 204.168.x.x (Hetzner)
CPX42 (8 vCPU, 16GB RAM)"]
VPS2["VPS 2: 135.181.x.x (Helsinki)
CPX22 (3 vCPU, 4GB RAM)
Firewall: InFromMyIPOnly"]
end
subgraph LocalLivingRoom ["Sovereign Local Hardware"]
K3s["Local K3s Node (192.168.x.16)
Immich v2.7.5 + Valkey + Postgres
916 GB Library Disk"]
end
K3s -.->|"Migrated From"| VPS1
```
Here was the inventory:
1. **The Storage Box (`u528497.your-storagebox.de`):** Our offsite backup target, guarded by SFTP and requiring a physical hardware tap on a YubiKey.
2. **Old VPS 1 (`204.168.x.x`):** The legacy cloud home of Immich. Months ago, we migrated this 205 GB behemoth to our local mini-PC, immediately slashing **~€31.00 every month (~€372/year)** from our operating costs under **`[VIII. Frugal Architecture]`**. It was decommissioned and ready to be archived.
3. **VPS 2 (`135.181.x.x`):** A Helsinki cloud server (`cpx22`). According to our `terraform.tfstate` file, this machine was alive, healthy, and rigorously protected by an `hcloud_firewall` named `InFromMyIPOnly`. It was configured via Ansible to run a Caddy reverse proxy and an OWASP Threat Dragon container.
The agent proudly reported this back: *We have a live VPS in Helsinki running Caddy and Threat Dragon, ready for connection once we update the cloud firewall.*
Then, the human typed six characters that brought the room to a dead halt:
> *"2 finns inte längre kvar."*
> *(Translation: "2 is no longer there.")*
---
## Act III: When the State File Hallucinates
```html
@sre
"What do you mean it's not there?! terraform.tfstate says instance ID 117435841 is currently in state 'running' with private IP mapping and firewall attach ID 10375113!"
```
```html
@kurt
"Welcome to the real world, kid. You're looking at a ghost. The human logged into the cloud console weeks ago, clicked 'Delete', and walked away. But because nobody ran terraform apply to make your little state file happy, your codebase has been tenderly babysitting a digital corpse."
```
This is the quintessential trap of **Declarative State Drift**.
Developers love to talk about Infrastructure as Code as though the code *is* the infrastructure. But a `.tfstate` file is not a live camera feed into a datacenter. It is merely a diary of promises made in the past. If a human cancels an instance via a Web UI or an API call outside the pipeline, Terraform will happily sit on your disk for ten years, believing that a server in Finland is still humming along under your command.
To break down why this happens—and why our architecture looked the way it did—we turned to our squad's master tutor:
---
## Act IV: Kathy Sierra Explains the Universe
```html
@kathy-sierra
"If you’re forcing yourself to memorize tool commands, you haven’t got the right mental model yet. Let's throw out the textbook and talk about a restaurant."
```
> *"Think about building a restaurant:*
>
> * **Terraform is the General Contractor (The Builder):**
> You call the contractor and say: *'Pour the concrete foundation, frame the walls, hook up the electrical mains, put up a security fence, and screw the house number onto the curb.'*
> The contractor doesn't know how to cook pasta. The contractor doesn't care what color the dining room napkins are. The contractor builds the empty box and writes the street address on a piece of paper.
>
> * **Ansible is the Interior Decorator & Kitchen Crew:**
> Ansible takes the street address from the contractor's paper, walks through the front door via SSH, unboxes the stoves, sets up the POS registers, hangs the art, and starts the oven. Ansible doesn't buy real estate; Ansible makes the building useful.
>
> *In our setup, `inventory.tf` was the magical handover note. The moment Terraform received the IP address from Hetzner, it wrote `inventory.ini` so Ansible knew where to drive.*
>
> *And why was Terraform confused today? Because the landlord bulldozed the restaurant last month, but nobody told the contractor. He's still sitting in his truck with the original blueprints, wondering why nobody is ordering dinner."*
---
## Act V: The 220-Megabyte Catharsis
With Kathy’s mental model crystal clear, the human asked the ultimate question:
> *"Do we have anything else installed with Terraform?"*
The agent ran a search across every directory, repository, and submodule on the entire workstation.
```bash
$ fdfind -u -e tf . /home/andreas/Projekt
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/firewall.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/main.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/inventory.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/versions.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/variables.tf
```
Nothing.
No AWS VPCs. No Azure Resource Groups. No GCP service accounts. No DNS zones.
That single, already-deleted Helsinki VPS was the **one and only reason Terraform existed in our entire organization**.
```html
@kurt
"Hold on. Are you telling me we’ve been keeping a 100-megabyte compiled Terraform binary, 120 megabytes of cached provider plugins, and a bunch of make targets in our git repo... to manage an empty patch of grass in Helsinki?!"
```
```html
@cfo
"Calculate the carrying cost! Zero running servers, 220 megabytes of disk bloat, and cognitive drag every time someone reads the Makefile. Execute the purge!"
```
We didn’t just archive the text files. We performed a full, therapeutic purge under **`[IX. Simplicity Over Abstraction]`**:
1. **Purged the Bloat:**
We `git rm`'d the 100MB `terraform` executable, the provider binaries, the cached lockfiles, and the hallucinating `terraform.tfstate`. Over **2,570 lines of dead state** and 220MB of binary garbage vanished from the repository in a single commit.
2. **Preserved the Heritage:**
The declarative configuration files (`main.tf`, `firewall.tf`) were archived cleanly to `docs/archive/hetzner-terraform/`, alongside an immutable `ARCHIVED.md` tombstone.
3. **Replaced the Phantom Menu:**
In our interactive terminal control center (`./infra`), option `V` used to open a *Cloud Server Management* menu with buttons to apply and destroy Terraform stacks. We ripped out the dead cloud hooks and replaced them with **`☸️ K3s & Immich LAN Server`** controls—allowing us to check API health, stream pod logs, and inspect local disk usage with a single keystroke.
4. **Cleaned the Makefile:**
Replaced phantom targets like `make vps-plan` with lightning-fast local targets like `make immich-status` and `make immich-pods`.
```bash
$ git commit -m "chore(infra): decommission hetzner terraform and archive manifests"
23 files changed, 17 insertions(+), 2571 deletions(-)
delete mode 100755 infrastructure/hetzner/terraform/terraform
delete mode 100644 infrastructure/hetzner/terraform/terraform.tfstate
```
---
## Epilogue: What Does True Sovereignty Look Like?
When tech influencers talk about "sovereign AI" and "self-hosting," it’s often accompanied by photos of screaming server racks in garages or elaborate, over-engineered Kubernetes clusters burning 800 watts at idle.
Our reality on Sunday morning was much quieter:
* **Zero Cloud Compute Invoices:** No unexpected $0.05/hour charges ticking away in the background. No forgotten NAT Gateways.
* **100% Data Sovereignty:** 210 gigabytes of high-resolution family photos, encrypted databases, and vector embeddings residing on local storage, safely backed up offsite via encrypted Restic snapshots requiring a physical YubiKey touch.
* **Declarative Network Sanity:** A clean, router-level static DHCP mapping that keeps the host configuration pure and resilient.
* **A 220MB Leaner Repository:** No heavy binaries masquerading as code, and no state files pretending to run servers that were turned off long ago.
As @kathy-sierra would say: *You don't need a cloud provider to be an architect. Sometimes the greatest architectural achievement is knowing when to hand the contractor his hard hat, shut the gate, and enjoy the house you actually live in.*