Frank Buytendijk of Gartner asked a good question on LinkedIn this week. Gartner is updating its AI ethics research, and he sees the long-term future as clear: “guardian agents and constitutional AI”. What he wants to know is how we move there from the mostly manual use case reviews that are common practice today.
I think the question is right and the destination is wrong. This is my answer, and it ends somewhere else: ethical AI is growing into decision intelligence.
Who guards the guardian
A guardian agent is an agent that watches other agents. We build it because we do not trust the agents. But the guardian is an agent too, so we have the same reason not to trust it. Quis custodiet ipsos custodes: who guards the guardians? Putting a second agent above the first only moves the question up one level.
It is worse than a regress. A guardian has to see everything and be able to stop anything, which makes it the most powerful thing in the system. If it also reasons and has goals of its own, it is not the safest component. It is the most dangerous one. The prisoner is dangerous in proportion to the leaks in the cell. The guardian is dangerous in proportion to its powers.
A constitution cannot know intention
Constitutional AI is naive in a different way. It presupposes right and wrong. It writes them down and asks the model to judge by them. But right and wrong depend on intention, and the model does not have the context of intention. It sees a request. It cannot see why the request was made, and it has no reliable way to find out.
A model that judges right and wrong without knowing intention is guessing. I cannot inspect the guess and I cannot repeat it.
Ethical means to do what you say
My definition is smaller. Ethical means to do what you say. I wrote last year that ethos is the ability of an organization to follow its guidelines, and I still think that is the whole of it.
An AI weapon can be ethical if its stated purpose is to kill and that is what it does. An AI agent that gives dishonest advice is unethical if its purpose was to give correct advice. The first example is uncomfortable, and it is meant to be. It shows that there are two questions here and we keep mixing them up.
Is the purpose acceptable? That is a question for law and for democratic debate, for people who can be held to account. It is not a question for a model.
Does the system do what it says? That is a question we can answer. It is the only one of the two that can be built into a system and checked.
This is why the directives that govern AI should be measurable by deterministic means. And that leads to the thing I would put in the guardian’s place.
The governor
In 1788 James Watt put a governor on his steam engine. Two weights spin with the shaft. When the engine runs faster the weights swing out and close the steam inlet. The governor knows nothing about steam, about the load, or about what the engine is for. It measures one thing and it closes one thing. It cannot be persuaded, and it wants nothing.
That is what I want watching an AI agent.
An agent with a goal behaves like pressure. It pushes on every surface of the space it is given, and it does not distinguish the walls we meant from the ones that just happened to be there. We do not handle pressure by building a second boiler to keep an eye on the first. We handle it with instruments: a gauge that reads the pressure, a relief valve that vents when it crosses the line, and a governor that closes the inlet before it gets there.
The obvious objection is that an agent can do a million things and a governor can do one. How can the small thing bound the large one? The steam engine already answers this. The governor does not regulate the engine’s behaviour. It regulates one consequence of that behaviour, shaft speed, and the engine may do whatever it likes as long as that one number stays in range.
So the design work is to choose the variable. Find the one point that every path has to cross, and make that point mechanical. For an agent, that point is where a proposal becomes a fact: a write to the record, a payment, a commit, a message that leaves the building. The model may draft whatever it likes. What becomes real has to pass a schema, a rule, a signature.
The principle underneath is that a bound must be of a different kind from the thing it bounds. You cannot bound a goal-seeker with another goal-seeker, because the second one brings its own pressure and its own reach. You bound it with a mechanism. Whoever watches must not want.
If we want something that reasons to look over the work, and often we do, then split the watcher in two. The critic gets a voice and no valve: it can object, and the objection is kept, but it cannot act. The governor gets the valve and no opinion. A mind without hands, and hands without a mind.
There is one more thing in the history. Boilers kept exploding for most of a century, until inspection codes made the valves mandatory. The engineering answer and the regulatory answer arrived together, and neither held alone.
It has to be red before we accept green
A governor I have never seen close is a decoration.
This is the part I have learned from building software together with AI, inside a harness. The rule there is simple: it has to be red before we accept green. The acceptance test is committed first, alone, and it has to fail. Then the change is made. Then a fresh context, one that did not write the change, confirms three things: the test was red before, it is green now, and nobody weakened the test in between. The context that wrote the change never gets to say that it works.
Why so strict? Because a model is very good at producing green. Green is what plausible looks like. A check that has never failed tells me nothing about whether it can fail. Verification means nothing if we cannot falsify.
Now hold the two futures up against that rule.
A guardian agent has approved ten thousand actions. Can it say no? When did it? Show me the case it got wrong, and how we found out.
A constitution says the model will act rightly. Which observation would show that it did not? If nobody can name one, what we have is not verification. It is reassurance.
The same rule applies to the governor, and that is the point. Every directive should come with its red case: an input we know violates it, shown to be refused, before we trust a single pass. The boiler inspector did not read the valve’s documentation. He lifted it by hand.
We met this at the Danish Business Authority before we had the words for it. Caseworkers looked at the companies the model flagged for fraud. But we also had to look at some of the ones it passed. A model drifting towards more false negatives would have stayed green forever. So we sampled deliberately and blinded the scores, because you have to look where the system says nothing is wrong.
This is decision intelligence
In 2017 the industry was asking “is this fair?”. We found that the real question was another one: can we quantify what we are trying to achieve, and measure whether we are achieving it? We set out to build ethical AI and ended up building the capability to know what our systems were doing and why. I called it Decision Operations. I wrote at the time that this was not ethics theater, it was decision intelligence.
Back then the model scored a case and a person decided. Today the model drafts, proposes and acts. The answer is the same, moved to where the decision now starts:
- Say what the system is for, precisely enough that it can be tested.
- Let the model propose freely.
- Put a governor on the point where a proposal becomes a fact.
- See it red before you accept green.
- Let a person accept what remains. That is the only check on sense, so keep it cheap and informed.
- Keep the decision: what was proposed, what was refused, who accepted it, what the alternatives were.
- Return to it, and ask what would have happened if we had decided otherwise.
Read that list with ethics in mind and it is an ethics programme. Read it with decision intelligence in mind and it is the same list. If ethical means to do what you say, then an ethical organization is one that can show, for any decision that mattered, what it said it would do, what it did, and what would have proved it wrong. Ethics stops being a review that sits beside the system. It becomes a property of how decisions are made and kept.
So, how do we move on from manual reviews?
A manual use case review does two jobs at once. It judges whether the purpose is acceptable, and it tries to judge whether the system will behave accordingly.
Keep the first job manual. It is the one decision that needs a person, and it should end in a declared purpose that can be tested, with its red cases, not in an approval memo.
Automate the second, by governor and not by guardian. Measure the system against what it said it would do, continuously, with a record that keeps itself.
The review does not disappear into an agent. It shrinks to the one question only people can answer. Everything after that is measured, and everything measured has been seen to fail.
Do what you say, and be able to show it. The rest is hope.
Earlier article: How to Build Ethical AI - We Did It by Design (And Discovered Something Bigger). Frank Buytendijk’s post is paraphrased from his LinkedIn feed, linkedin.com/in/frankbuytendijk; the one phrase in quotation marks is his.