Giving an AI production access without giving it the keys
A conversational agent that can inspect live infrastructure on request and cannot change any of it without a one-time code from a human — running on self-hosted models, so the questions never leave the building.
The challenge
The useful version of an operations assistant is one that can actually see the estate — is the cluster healthy, is replication behind, would failover work right now, which box is nearly out of disk. The moment it can see that, the obvious next step is letting it act, and that is where most people either stop or do something they regret. An agent with credentials is an agent that can be talked into using them, and the honest reason to hesitate is not that the model is stupid. It is that there is no way to be sure what a language model will do with an instruction it has never seen before, and production is the wrong place to find out.
Read freely. Change nothing without a human.
The design answer is an asymmetry, and it is the whole idea: reads are unrestricted, writes are gated by a person. Ask about cluster quorum, host CPU and memory, replication job status, failover readiness, storage headroom, interface and bond state — all of it answers immediately, because none of it can hurt anything. Ask it to restart a machine and it requires a time-based one-time code from the operator's phone before it will proceed. The model proposes; a human authorizes; the authorization is something the model structurally cannot produce for itself. That last clause is what makes it a control rather than a policy — it is not trusting the agent to refuse, it is making the destructive path require a factor the agent does not have. Access is scoped to a single principal on top of that, so there is no ambiguity about who asked.
Why the inference runs locally
The models run on hardware in the rack, not on somebody's API, and the reason is specific rather than ideological. Every question this thing answers is infrastructure state: topology, hostnames, capacity, what is currently broken. That is reconnaissance material. Sending it to a third party is a disclosure decision, and it is one most teams make by accident because the convenient tool happens to be a hosted one. Running the inference locally means the operational picture never leaves the network and there is no vendor to have a data-retention conversation about. The trade is real — a small local model is not a frontier model — and it is the right trade here, because the job is summarizing structured facts the system already knows, not reasoning from scratch.
The outage that corrected the design
The sharpest lesson came from a failure, not a feature. During a full estate outage the agent went down too, and stayed down after the hosts came back — its scoped credential from the secrets manager had expired, and nothing renewed it. The tool most useful during an incident had a dependency on the very system the incident had taken out. That is an embarrassing thing to find and a genuinely important one: an incident-response tool that shares a failure domain with the incident is decoration. The general form is worth stating plainly, because it generalizes past this system — anything whose job is to help during a failure has to be checked against the question 'what if the thing it depends on is what failed?' The current fix is credential renewal that survives a restart; the durable fix is treating the assistant's own liveness as a monitored dependency rather than assuming it.
What transfers
Almost none of the value here is the chat interface, which is the part everyone looks at. The value is the shape: a hard split between reading and changing, a second factor that the model cannot supply, a single authorized principal, inference placed according to how sensitive the questions are rather than how convenient the vendor is, and a written answer to what happens when the assistant itself is part of the outage. That shape is the answer to the question every cautious team asks about putting AI near production — not a reassurance that the model is careful, but an architecture where carefulness is not required. It is the same pattern I would build into an AI integration for anyone whose systems matter, and it is why I will say no to an agent design that asks to be trusted.