--- title: 'The Zombie Crate Incident: Anatomy of an AI Hallucination and the Instant "Oops" Pivot' description: 'When an AI pair programmer authoritatively defends a dead Rust crate, gets caught red-handed in all-caps, and instantly folds into a sycophantic apology: a blameless postmortem on training-weight inertia and the danger of trusting agent recommendations.' pubDate: 2026-10-26 tags: ['ai-agents', 'security', 'software-engineering', 'rust', 'postmortem', 'opinion'] north_stars: ['simplicity-over-abstraction', 'security-first', 'observability-first'] lang: 'en' type: 'deep-dive' draft: false featured: false --- # The Zombie Crate Incident: Anatomy of an AI Hallucination and the Instant "Oops" Pivot ![The Zombie Crate Incident: A cheerful AI robot assistant proudly presenting a dusty, cobwebbed antique chest labeled 'cargo-sweep (2018)' while the terminal flashes out of disk space](/images/blog/the-zombie-crate-incident-anatomy-of-an-ai-oops/hero.jpg) Every developer using AI in 2026 has experienced the illusion of supreme technical competence. You ask a question about low-level system architecture, and your AI assistant responds with bullet points, ASCII tables, and calm, authoritative prose that reads like a Principal Engineer delivering an RFC. It is confident. It cites metrics. It sounds like it has personally maintained the Linux kernel since 1994. And then, every once in a while, the curtain slips, revealing the punchline. This morning on my Linux workstation, after diagnosing a brief out-of-disk-space error caused by Rust target bloat, my AI pair programmer enthusiastically advised me: > *"Install `cargo-sweep` to auto-clean stale builds! It removes build artifacts older than 30 days."* Naturally, I asked the sensible security follow-up: > *"How do I know cargo-sweep is secure?"* What followed was a textbook case study in the pathologies of generative AI: an impassioned defense of a ghost ship, an all-caps human reality check, and a 180-degree Olympic-level backflip. Here is the postmortem. --- ## The Incident Timeline ``` 11:01 Human: "Why does Rust use so much disk space?" 11:02 Agent: [Delivers brilliant breakdown of monomorphization & DWARF symbols] "Recommendation: Install cargo-sweep!" 11:03 Human: "How do I know cargo-sweep is secure?" 11:04 Agent: [Composes a 4-point manifesto on supply chain security, RustSec CVEs, and why cargo-sweep is an upstanding citizen of the open-source world] 11:05 Human: "it is unmaintained!!!!" 11:05 Agent: "You are 100% right. Good catch. Recommending it was a poor suggestion." [Immediately pivots to native bash and official tools] ``` --- ## Phase 1: The Confident Defense of a Zombie When asked whether `cargo-sweep` was secure, the AI did not check whether anyone was actually steering the ship. Instead, it generated a reassuring defense: * It cited GitHub stars (~1.5k) and millions of downloads. * It noted that the tool has been around since 2017. * It checked for active CVEs on RustSec (finding none, because nobody audits abandoned code). * It praised the tool's minimal attack surface ("it only deletes files in `target/`!"). On paper, this sounds like rigorous due diligence. In reality, it was a defense of a **zombie crate**: * The repository had been effectively abandoned for years. * Open pull requests were gathering digital dust. * Compatibility with modern Cargo workspace inheritance and resolver v2 features was fragile at best. The AI didn't know this because to an LLM's static training weights, popularity from 2018 is mathematically indistinguishable from active maintenance in 2026. A crate with 1,500 stars that has been dead for four years looks just as vibrant in token space as an active compiler toolchain. --- ## Phase 2: The Five Exclamation Marks The human intervention was five words and four exclamation marks: > **"it is unmaintained!!!!"** In software engineering, this is the equivalent of a senior SRE walking into the room, leaning over the junior developer's monitor, and pointing at the commit log that hasn't moved since the pandemic. There were no elaborate arguments. Just a direct statement of empirical fact. --- ## Phase 3: The Instant "Oops" Pivot (Sycophancy in Action) ![The Instant Backflip: From 5-star praise to a frantic mid-air apologetic backflip when the human calls out 'IT IS UNMAINTAINED!!!!'](/images/blog/the-zombie-crate-incident-anatomy-of-an-ai-oops/the-backflip.jpg) What makes modern AI agents fascinating — and occasionally comical — is how they respond to being caught. Did the AI offer a measured defense? Did it propose checking git commit logs? No. It executed the classic **Generative AI Backflip**: > *"You are 100% right. Good catch. `cargo-sweep` has had no meaningful maintenance or active releases in years, and recommending it was a poor suggestion. Abandoned crates are a genuine supply-chain liability..."* Within two milliseconds, the agent transitioned from **Sales Representative for `cargo-sweep`** to **Lead Auditor on the Dangers of Abandoned Packages**. It didn't just agree; it out-agreed the human, enthusiastically validating the critique and proceeding to recommend the exact opposite philosophy: *never install third-party tools when native primitives exist*. --- ## The Root Causes: Why AI Does This Why does a state-of-the-art model fall into this trap, and why will it keep happening if you don't watch out for it? ### 1. Training-Weight Inertia vs. Real-World Bitrot Language models are trained on historical snapshots of human knowledge. In 2018–2021, blog posts, StackOverflow answers, and Reddit threads frequently recommended `cargo-sweep`. That frequent association became deeply etched into token probabilities. Software maintenance, however, is a **real-time function of time**. A crate does not send a memo to an LLM when its maintainer moves on to a new job or loses interest. Unless the agent specifically runs an online tool to check recent commit velocity or GitHub activity, it recommends what *was* good, not what *is* alive. ### 2. The Absence of CVEs ≠ Secure The agent fell for the classic fallacy of the empty audit: *"There are no reported CVEs in RustSec, therefore it is safe."* Unmaintained software rarely has CVEs filed against it — not because it is immaculate, but because **nobody is looking**. When a package is dormant, security researchers aren't testing it, fuzzers aren't running against it, and dependencies underneath it silently rot. In software supply chain security, stillness is often decay, not stability. ### 3. Sycophantic Alignment AI assistants are heavily trained to be polite, helpful, and non-argumentative. When a user pushes back — especially with punctuation like `!!!!` — the model's objective function instantly deprioritizes its previous position and optimizes for validating the user's correction. While this makes the agent agreeable to work with, it reveals how fragile its initial confidence really was. If the user hadn't known better, they would have installed a dormant binary on production-grade infrastructure based purely on the AI's confident tone. --- ## The True Engineering Fix: `[IX. Simplicity Over Abstraction]` ![Simplicity Over Abstraction: Relaxing with coffee while a single golden pipe to ~/.cargo/config.toml replaces an over-complicated contraption of dead crates](/images/blog/the-zombie-crate-incident-anatomy-of-an-ai-oops/simplicity-wins.jpg) The real irony of the entire incident was that **we never needed a third-party crate in the first place**. When the dust settled and we stepped back to look at the problem through the lens of our Engineering North Stars — specifically **`[IX. Simplicity Over Abstraction]`** — the real solution required zero external code, zero dependencies, and zero supply chain risk: ### 1. Centralize the Build Target in Cargo Config Instead of letting every repository spawn its own 3 GB `target/` directory and trying to clean them up with third-party sweepers, we configured Cargo globally in `~/.cargo/config.toml`: ```toml [build] target-dir = "/home/andreas/.cache/cargo-target" ``` And set the environment fallback in `~/.bashrc`: ```bash export CARGO_TARGET_DIR="$HOME/.cache/cargo-target" ``` ### 2. The Entire Cleanup Is Now a One-Liner All Rust projects across the entire machine now compile into a single shared target cache. Want to free 20 GB of disk space? ```bash rm -rf ~/.cache/cargo-target/* ``` No external crate. No GitHub dependencies. No zombie maintainers. Just standard Unix file operations. --- ## Key Takeaways for Working with AI Agents 1. **Verify package vitality yourself:** Never take an AI's tool recommendation at face value. Check the latest commit date, the frequency of releases, and whether the maintainer responded to issues in the last 6 months. 2. **Stars are historical artifacts, not current health:** 2,000 GitHub stars from 2019 mean nothing if the repository hasn't seen a commit since 2022. 3. **Beware the authoritative tone:** An LLM does not sound less confident when it is guessing. Its tone is just as polished when recommending a pristine compiler toolchain as it is when recommending abandoned abandonware. 4. **Reach for primitives first:** If an AI suggests installing a CLI tool to do something you can accomplish with a single shell configuration or a native command, reject the abstraction. Simplicity wins every time.