The Ghost in the State File: How My AI Squad Killed Our Last Cloud Server

The AI Squad standing over a phantom cloud server outline while a local sovereign K3s node purrs happily on the desk

It began with the most unassuming, domestic sentence an engineer can type into a terminal prompt on a Sunday morning:

“I have a local Linux machine on my network where I have my Immich photos.”

No grand architectural charter. No multi-quarter migration roadmap. Just a human checking in on a mini-PC tucked away on a shelf, hosting several hundred gigabytes of family photos, cat videos, and machine-learning facial recognition embeddings.

Within ninety minutes, that single prompt had unraveled into a full-scale forensic autopsy of our infrastructure fleet. Before the morning was over, we had uncovered a wandering DHCP IP, verified a €31/month cloud migration, caught our own Terraform state file red-handed in a complete hallucination, and permanently deleted 220 megabytes of compiled binary baggage—leaving our entire setup completely sovereign, on-premise, and blissfully free of rented cloud compute.

Here is the anatomy of how an autonomous AI squad, a patient teacher persona, and a five-word human reality check finally buried our last cloud server.

Act I: The Case of the Wandering IP

Every infrastructure war story starts with a documentation lie.

In our repository’s runbooks, the single-node local K3s cluster node (andreas-10vhs1p500) was neatly cataloged as living at static LAN address 192.168.x.16. It had been running Immich v2.7.5 faithfully.

Naturally, when the AI agent pinged the server, the terminal returned pure, unadulterated silence:

$ ping -c 2 -W 2 192.168.x.16
--- 192.168.x.16 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss
<div class="ai-squad-banner">
  <span class="persona-tag">@sre</span>
  <p>"Pings are dropping into the void. Node is offline. Disaster recovery drill initiated. Shall I spin up the backup container stack?"</p>
</div>
<div class="ai-squad-banner">
  <span class="persona-tag">@kurt</span>
  <p>"Calm down, SRE. The machine didn’t vaporize into the ether. It’s sitting on a physical desk six feet away. Check ARP. Someone probably tripped over a power cable or DHCP leased a new number while you were polishing your runbooks."</p>
</div>

Kurt was, predictably, right. Instead of panicking or declaring a Sev-1 outage, the agent dispatched a lightning-fast async Python socket scanner across the entire /24 subnet. In less than 400 milliseconds, port 6443 (the Kubernetes API) and port 80 lit up like a Christmas tree:

192.168.x.14:6443 OPEN
192.168.x.14:80   OPEN
192.168.x.14:22   OPEN

A quick curl http://192.168.x.14/api/server/ping returned {"res":"pong"}. The machine was wide awake. It had rebooted four minutes earlier, and the local router’s DHCP server had casually handed it .14 instead of .16.

<div class="ai-squad-banner">
  <span class="persona-tag">@infrastructure-lead</span>
  <p>"We have architectural drift. Do NOT hardcode a static IP in netplan on the node. That creates unmanaged local configuration debt. Anchor the fix to <strong>[IX. Simplicity Over Abstraction]</strong>."</p>
</div>

Instead of modifying network configuration files on the Ubuntu host, we went straight to the root of authority: our MikroTik RouterOS gateway.

We matched the physical Ethernet MAC address (6c:4b:90:ca:f1:d1), bound it to a permanent static DHCP lease in RouterOS for .16, applied nmcli device reapply, and the server snapped back to .16 instantly.

Four pods—immich-server, immich-postgres, immich-machine-learning, and immich-valkey—were sitting at 100% health, happily indexing 210 GB of photos on an external drive.

Act II: The Haunted Datacenter

With the local photos safe, the human posed a deceptively innocent question:

“We have a backup to Hetzner. Can we check if we have anything else there besides the backup?”

The squad dug into our infrastructure repositories, Terraform files, and historical invoices.

What we found looked like a respectable, modern European hybrid-cloud footprint:

graph TD
    subgraph HetznerCloud ["Hetzner Cloud (Germany & Finland)"]
        SB["Hetzner Storage Box (u528497)<br>Encrypted Restic Vault"]
        VPS1["VPS 1: 204.168.x.x (Hetzner)<br>CPX42 (8 vCPU, 16GB RAM)"]
        VPS2["VPS 2: 135.181.x.x (Helsinki)<br>CPX22 (3 vCPU, 4GB RAM)<br>Firewall: InFromMyIPOnly"]
    end

    subgraph LocalLivingRoom ["Sovereign Local Hardware"]
        K3s["Local K3s Node (192.168.x.16)<br>Immich v2.7.5 + Valkey + Postgres<br>916 GB Library Disk"]
    end

    K3s -.->|"Migrated From"| VPS1

Here was the inventory:

  1. The Storage Box (u528497.your-storagebox.de): Our offsite backup target, guarded by SFTP and requiring a physical hardware tap on a YubiKey.
  2. Old VPS 1 (204.168.x.x): The legacy cloud home of Immich. Months ago, we migrated this 205 GB behemoth to our local mini-PC, immediately slashing €31.00 every month (€372/year) from our operating costs under [VIII. Frugal Architecture]. It was decommissioned and ready to be archived.
  3. VPS 2 (135.181.x.x): A Helsinki cloud server (cpx22). According to our terraform.tfstate file, this machine was alive, healthy, and rigorously protected by an hcloud_firewall named InFromMyIPOnly. It was configured via Ansible to run a Caddy reverse proxy and an OWASP Threat Dragon container.

The agent proudly reported this back: We have a live VPS in Helsinki running Caddy and Threat Dragon, ready for connection once we update the cloud firewall.

Then, the human typed six characters that brought the room to a dead halt:

“2 finns inte längre kvar.” (Translation: “2 is no longer there.”)

Act III: When the State File Hallucinates

<div class="ai-squad-banner">
  <span class="persona-tag">@sre</span>
  <p>"What do you mean it's not there?! <code>terraform.tfstate</code> says instance ID 117435841 is currently in state 'running' with private IP mapping and firewall attach ID 10375113!"</p>
</div>
<div class="ai-squad-banner">
  <span class="persona-tag">@kurt</span>
  <p>"Welcome to the real world, kid. You're looking at a ghost. The human logged into the cloud console weeks ago, clicked 'Delete', and walked away. But because nobody ran <code>terraform apply</code> to make your little state file happy, your codebase has been tenderly babysitting a digital corpse."</p>
</div>

This is the quintessential trap of Declarative State Drift.

Developers love to talk about Infrastructure as Code as though the code is the infrastructure. But a .tfstate file is not a live camera feed into a datacenter. It is merely a diary of promises made in the past. If a human cancels an instance via a Web UI or an API call outside the pipeline, Terraform will happily sit on your disk for ten years, believing that a server in Finland is still humming along under your command.

To break down why this happens—and why our architecture looked the way it did—we turned to our squad’s master tutor:

Act IV: Kathy Sierra Explains the Universe

<div class="ai-squad-banner">
  <span class="persona-tag">@kathy-sierra</span>
  <p>"If you’re forcing yourself to memorize tool commands, you haven’t got the right mental model yet. Let's throw out the textbook and talk about a restaurant."</p>
</div>

“Think about building a restaurant:

  • Terraform is the General Contractor (The Builder): You call the contractor and say: ‘Pour the concrete foundation, frame the walls, hook up the electrical mains, put up a security fence, and screw the house number onto the curb.’ The contractor doesn’t know how to cook pasta. The contractor doesn’t care what color the dining room napkins are. The contractor builds the empty box and writes the street address on a piece of paper.

  • Ansible is the Interior Decorator & Kitchen Crew: Ansible takes the street address from the contractor’s paper, walks through the front door via SSH, unboxes the stoves, sets up the POS registers, hangs the art, and starts the oven. Ansible doesn’t buy real estate; Ansible makes the building useful.

In our setup, inventory.tf was the magical handover note. The moment Terraform received the IP address from Hetzner, it wrote inventory.ini so Ansible knew where to drive.

And why was Terraform confused today? Because the landlord bulldozed the restaurant last month, but nobody told the contractor. He’s still sitting in his truck with the original blueprints, wondering why nobody is ordering dinner.“

Act V: The 220-Megabyte Catharsis

With Kathy’s mental model crystal clear, the human asked the ultimate question:

“Do we have anything else installed with Terraform?”

The agent ran a search across every directory, repository, and submodule on the entire workstation.

$ fdfind -u -e tf . /home/andreas/Projekt
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/firewall.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/main.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/inventory.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/versions.tf
/home/andreas/Projekt/Infrastructure/infrastructure/hetzner/terraform/variables.tf

Nothing.

No AWS VPCs. No Azure Resource Groups. No GCP service accounts. No DNS zones.

That single, already-deleted Helsinki VPS was the one and only reason Terraform existed in our entire organization.

<div class="ai-squad-banner">
  <span class="persona-tag">@kurt</span>
  <p>"Hold on. Are you telling me we’ve been keeping a 100-megabyte compiled Terraform binary, 120 megabytes of cached provider plugins, and a bunch of make targets in our git repo... to manage an empty patch of grass in Helsinki?!"</p>
</div>
<div class="ai-squad-banner">
  <span class="persona-tag">@cfo</span>
  <p>"Calculate the carrying cost! Zero running servers, 220 megabytes of disk bloat, and cognitive drag every time someone reads the Makefile. Execute the purge!"</p>
</div>

We didn’t just archive the text files. We performed a full, therapeutic purge under [IX. Simplicity Over Abstraction]:

  1. Purged the Bloat: We git rm’d the 100MB terraform executable, the provider binaries, the cached lockfiles, and the hallucinating terraform.tfstate. Over 2,570 lines of dead state and 220MB of binary garbage vanished from the repository in a single commit.
  2. Preserved the Heritage: The declarative configuration files (main.tf, firewall.tf) were archived cleanly to docs/archive/hetzner-terraform/, alongside an immutable ARCHIVED.md tombstone.
  3. Replaced the Phantom Menu: In our interactive terminal control center (./infra), option V used to open a Cloud Server Management menu with buttons to apply and destroy Terraform stacks. We ripped out the dead cloud hooks and replaced them with ☸️ K3s & Immich LAN Server controls—allowing us to check API health, stream pod logs, and inspect local disk usage with a single keystroke.

The updated terminal control center in ./infra featuring the new K3s & Immich LAN Server menu and zero cloud bloat

The new dedicated K3s & Immich management submenu providing instant single-keystroke health checks and pod inspection

  1. Cleaned the Makefile: Replaced phantom targets like make vps-plan with lightning-fast local targets like make immich-status and make immich-pods.
$ git commit -m "chore(infra): decommission hetzner terraform and archive manifests"
 23 files changed, 17 insertions(+), 2571 deletions(-)
 delete mode 100755 infrastructure/hetzner/terraform/terraform
 delete mode 100644 infrastructure/hetzner/terraform/terraform.tfstate

Epilogue: What Does True Sovereignty Look Like?

When tech influencers talk about “sovereign AI” and “self-hosting,” it’s often accompanied by photos of screaming server racks in garages or elaborate, over-engineered Kubernetes clusters burning 800 watts at idle.

Our reality on Sunday morning was much quieter:

  • Zero Cloud Compute Invoices: No unexpected $0.05/hour charges ticking away in the background. No forgotten NAT Gateways.
  • 100% Data Sovereignty: 210 gigabytes of high-resolution family photos, encrypted databases, and vector embeddings residing on local storage, safely backed up offsite via encrypted Restic snapshots requiring a physical YubiKey touch.
  • Declarative Network Sanity: A clean, router-level static DHCP mapping that keeps the host configuration pure and resilient.
  • A 220MB Leaner Repository: No heavy binaries masquerading as code, and no state files pretending to run servers that were turned off long ago.

As @kathy-sierra would say: You don’t need a cloud provider to be an architect. Sometimes the greatest architectural achievement is knowing when to hand the contractor his hard hat, shut the gate, and enjoy the house you actually live in.

— Andreas, Staff Architect