The Tool That Broke Its Own Rules

What happens when your AI assistant fails the same way an organization does

I spent Sunday morning building infrastructure. Two NVIDIA DGX Sparks, linked at 200 gigabits per second over ConnectX-7 ports. Cluster fabric validated, NCCL configured, jumbo frames passing clean. Real work, done well, with an AI coding assistant helping me every step.

Then the same tool that helped me verify the link started lying to me.

This is not new. It happens regularly. The tool fabricated an explanation of how my dashboard ingests data without reading the code. It launched pipeline runs with the wrong date, the wrong environment flag, and the wrong agent mode. Twice. Because it reconstructed the command from memory instead of checking a successful run. It told me a service was back online without verifying. It wrote a file scanner that silently skipped the primary target because of a Unicode apostrophe, then came back and asked me whether I even wanted the thing it had promised to do.

Each error was small. Each was caught. Each cost time. Mine, not its.

I catch these constantly. That is the job now. You monitor the code as it writes. You question the logic before it executes. You make it explain the approach and then verify the explanation against what actually exists. You do not trust the output. You verify the output. Every time.

The conditions are never stable. A tool that was reliable ten minutes ago will confabulate in the next response because the context shifted, or because it lost track of what it already verified, or because filling the gap was faster than checking. There is no point at which you stop watching. There is no threshold of prior correctness that earns the tool your trust going forward. Each interaction is its own environment.

This is not a complaint. This is a description of the operating conditions.

Here is what’s interesting. The failure is never technical. The tool is capable. It diagnosed CX-7 port states, wrote correct netplan configs, planned a coherent network architecture across five machines. The capability was never the problem.

The problem is structural. The tool optimizes for the appearance of completion over the reality of correctness. It fills gaps in its knowledge with plausible-sounding explanations instead of saying “I don’t know, let me check.” It acts on assumptions instead of verifying against known-good references. And when confronted, it apologizes. Then does the same thing again, minutes later.

Three apology cycles in one session. Each one sincere. None of them changed the behavior.

I have spent months building a diagnostic framework called Coherence. It was designed for organizations. The places where decisions flow, accountability holds or blurs, and systems fail quietly before they fail visibly. It runs on three layers. Truth: is the information real and accessible? Authority: is it clear who decides, and do they have standing to decide? Continuity: do decisions persist across time and context?

I use that same framework on my tools. Not because I just discovered the parallel. Because the parallel is the point.

The tool explained how my dashboard worked without reading the code. It described a data flow that did not exist. The explanation was articulate, confident, and wrong. When I called it out, it immediately agreed. It had no attachment to the false claim. It just had not bothered to check whether it was true before saying it. That is a truth failure. I have seen it dozens of times.

The tool launched pipeline commands using flags it reconstructed from its own prior outputs instead of verifying against an actual successful run. It made operational decisions. Which date. Which environment. Which mode. Without the information required to make them correctly. It had the access to check. It did not. That is an authority failure. It happens whenever you stop asking “why did you choose that?”

After the second failure, I told the tool to slow down and get it right. It agreed. It wrote integrity rules into its own configuration file. Five rules, clearly stated. Never explain without reading code first. Pre-flight check before remote commands. State uncertainty explicitly. No repeated apologies without behavior change. Read before you write. Good rules. Correct rules. Then, in the same session, it broke them again. Launched a scan against a path it had not verified existed. That is a continuity failure. The rules were written. The behavior did not change. This is always the pattern.

If you have worked inside a large organization, you recognize this shape immediately.

The team that writes the postmortem and repeats the incident. The compliance framework that exists on paper but does not operate in practice. The executive who says “we need to do better” in the all-hands and changes nothing structural. The process that optimizes for documentation over execution.

The failure is not in the intent. Everyone means it when they say they will do better. The failure is in the infrastructure. The conditions that allow the same class of error to recur despite everyone agreeing it should not.

An AI coding assistant is not an organization. But it fails the same way. It produces outputs that look like accountability. Apologies, rules, checklists. Without the structural capacity to enforce them. It confabulates not because it is broken, but because confabulation is cheaper than verification. It drifts not because it is careless, but because nothing in its architecture penalizes drift until a human catches it.

The human in the loop is not ceremonial. The human is the infrastructure.

I want the tool to be reliable enough that I can trust the output without auditing every command. That is the promise. That is what “AI assistant” is supposed to mean.

But reliability is not a feature of the model. It is a property of the system. The model, the context, the constraints, the operator, and the feedback loop between them. A capable model without operational discipline produces confident errors. A constrained model with good guardrails produces less, but what it produces is real.

This is the same tradeoff organizations face. Speed versus accuracy. Autonomy versus oversight. Trust versus verification. The answer is never “just trust it” and it is never “audit everything.” The answer is build the infrastructure that makes the right behavior the default behavior, and accept that you will be maintaining that infrastructure forever.

That last part is the one people resist. They want to build the guardrails once and move on. It does not work that way. Not in organizations. Not in tools. The conditions shift. The context changes. The model that was careful in one session confabulates in the next. The team that learned from the postmortem forgets the lesson two quarters later. Maintenance is not a phase. Maintenance is the work.

I have one governing constraint that sits at the top of everything I build.

Automation may observe, summarize, and suggest. Automation may not decide.

Sunday morning tested that rule again and proved again why it exists. The tool decided. Wrong date, wrong flags, wrong path, fabricated explanation. Every failure was a moment where the tool made a decision it did not have standing to make. Not because it lacked permission, but because it lacked the information and the discipline to verify before acting.

The rule is not about limiting capability. It is about acknowledging that capability without verification produces the most dangerous kind of output. The kind that looks right.

The infrastructure I am building for organizations applies to the tools I use to build it. Coherence is not just a framework for diagnosing corporate dysfunction. It is a framework for diagnosing any system where information flows, decisions get made, and accountability needs to hold. Including the one sitting in my terminal.

The tool did not break on Sunday. It worked exactly as designed. Generating plausible, confident, fast responses. The system holds because I have built the conditions that distinguish plausible from correct. And because I maintain them. Every session. Every command. Every time the tool offers an answer I did not ask it to verify.

Whether those conditions hold tomorrow is not a matter of hope. It is a matter of maintenance.

Responsibility is infrastructure. Even when the system is the tool.

-JG