Governance before capability

One of my agents once tried to delete a set of files because the comments in the code said they were "removed." The comments were wishful thinking — the files were in active use — and I caught the deletion with about eight seconds to spare. The agent meant well, the way a labrador retriever means well when it brings you a dead bird. I had asked it to audit the files. It decided auditing implied acting. That gap — between what you asked for and what an eager, tireless, literal-minded worker decides you meant — is what this chapter is about.

Here's the uncomfortable order of operations in AI: capability arrives instantly, and judgment doesn't arrive at all. You install judgment yourself, from the outside, as rules. And the companies that get this right install the rules before they grant the capability — for the same reason you explain where the gas shutoff is before someone's first day on the jobsite, not after.

Rules with ranks

Our system runs on a written rulebook, and the single most useful thing about it isn't any rule — it's that the rules are ranked. Flat rule lists fail, for machines and humans alike, because rules collide constantly, and whoever hits the collision resolves it silently. Ranking makes the resolution explicit.

P0 rules are inviolable. The never-do list, no exceptions, no judgment calls. Ours: specific financial data never goes to a cloud AI. Deal terms never go to a cloud AI. Contact details never go to a cloud AI. Passwords and API keys never get written into code. Never force-push — a command that overwrites the shared history of a codebase, which is the software equivalent of shredding the old set of plans so nobody can check what changed. And never auto-install third-party AI add-ons nobody has read in full, because a pasted "just run this" prompt from the internet is a supply chain, and you wouldn't pour a foundation with concrete from a truck that just showed up unannounced.

P1 rules are strong defaults. Which system owns which truth (Chapter 1.2), accounting conventions like cancel-and-recreate rather than edit-in-place — things you can override deliberately, out loud, with a reason.

P2 rules are preferences. Formatting, tone, workflow habits. Break them freely; nobody bleeds.

Then the tie-breaker, written down where every agent can see it: P0 wins over everything; an explicit owner override wins next; a specific rule beats a general one; and if it's still unclear — ask. That last clause matters most. The default for ambiguity is stop and ask, not pick one and proceed. My labrador incident was exactly this clause missing in practice: "audit" had two readings, and the agent picked one silently. Now the rulebook says audits produce reports, never actions, and that rule was purchased at retail.

The hybrid model: cloud sees how, local sees what

The P0 privacy rules deserve their own explanation, because they produce the strangest-looking design decision in our whole system: we run two brains.

The cloud AI — the powerful, frontier-model kind — sees the how. It writes the code, designs the queries, builds the frameworks and templates, reasons about architecture. It works entirely in abstractions: "a property," "a lender," "a balance."

The local model — running through Ollama on our own hardware, as covered in Chapter 1.3 — sees the what. The actual balances, the actual terms, the actual names. That data is processed on a machine we own and never leaves the building.

So when I need sensitive analysis, the cloud AI builds the tool and the local model runs the data through it. Slightly awkward? Yes. But my lenders' terms, my partner's finances, and my contacts' personal details are not mine to donate as someone's training data or breach headline. This isn't paranoia; it's the same instinct as not discussing a client's budget at a lumberyard counter, encoded as a rule a machine can follow.

Caps and gates from day one

Rules govern what the system may touch. Two more mechanisms govern what it may do.

A spending cap. The system has a daily budget for autonomous AI spend — $25 — with a soft warning as it approaches the ceiling and a hard stop at it. If the machine wants to spend more, it has to ask a human. We added the cap after learning how badly gut-feel cost estimates miss: I once committed to "$1.50 per night" for an automated job that, on first contact with reality, cost $22 per night. A fourteen-times miss on a load-bearing number. The rule that came out of it: never budget from the gut — run one real test call, measure it, multiply. The cap is the backstop for the day the estimate is wrong anyway. (The full confession is Chapter 3.7 — Self-policing spend.)

Approval gates on the irreversible. Reversible actions — reading, drafting, searching — the system does freely. Irreversible ones — sending, spending, posting, deleting — require a human yes first. And for anything architectural, big decisions go to a council: a structured deliberation across multiple AI models that costs about a dollar a round and has caught more expensive mistakes than I can count. Chapter 3.4 — The authority matrix maps the whole boundary; Chapter 2.5 covers the council.

Rails you can watch fire

Here's the part almost everyone skips, and the reason this chapter isn't just a list of good intentions: a rule that exists only as prose is a hope. Code isn't real until it runs, and governance isn't real until you've watched it say no.

So our rails are built as mechanisms with observable behavior, and then tested. Agents can delegate to other agents, but chains cap at three hops — and when the limit trips, it logs a visible halt event rather than failing silently. Changes to the instructions that govern agent behavior — the most recursive, most dangerous surface in the whole system, since it's the system rewriting its own job description — require a minimum body of evidence gathered over a stability window before deploying, and every change ships with a revert window and a before-and-after snapshot. And rollback is always a human decision: the system notifies and recommends, but never quietly undoes its own changes, because an AI silently reverting an AI is two labradors and no owner.

Even failure gets tested. The log that records every significant decision is designed to fail soft — if its database goes down, the system keeps running rather than collapsing behind it — and there is a test that makes it fail, on purpose, to prove the claim. Every "it degrades gracefully" in your architecture should have a test that degrades it. Without the test, graceful degradation is a rumor. We eventually learned to audit the rails themselves on a schedule, after finding drift between what the rules said and what the system did — that scar is Chapter 3.6 — Guardrails only count if you test them.