You Can't Control What You Don't Understand

You Can't Control What You Don't Understand
Photo by Chris Leipelt / Unsplash

I've spent the last few months wiring AI agents into everything. OpenClaw runs on a Fedora box in my house with local Ollama server, and another LLM runs an events aggregator website for me. At work my colleague and I are building a Real World Assets protocol doing the work of ~6 FTEs with agents. I've built dozens of apps and co-work setups that help with automation for clients. Every time, the same question shows up under the surface: how much am I actually controlling this, and how much am I just hoping the output behaves?

That question sent me back to the word itself. "Control" traces back to Latin contrarotulus, a counter-roll. It was a duplicate ledger, kept specifically to check an original account against fraud or error. It's not about restraining or commanding.

Control is a feedback loop. Measure, compare, adjust, repeat.

It's about checks-and-balances. A control group in an experiment checks a hypothesis. Climate control checks a room's temperature against a setpoint. Self-control checks an impulse against a standard before it becomes an action. Even "controlling a company" comes down to the same loop: you measure what's happening, compare it to what should be happening, and adjust when the two drift apart.

The loop needs two things

Once you see control as a loop instead of a grip, it becomes obvious what it needs: a setpoint. You need a desired state to compare against, and you need the ability to check whether the actual state matches it. Lose either one and the loop stops working.

If you don't know what you're aiming for, you can't steer toward it. Nobody hands a thermostat a temperature and expects it to figure out comfort on its own.

Even with a clear target, you still have to be able to tell whether you've hit it. Things get interesting with LLMs and AI agents, where the real question becomes whether you understand the output. There's a real distinction between understanding how something works and understanding a result well enough to judge it.

You don't need to know the mechanism, but you need to be able to check the output

I don't need to understand how a calculator computes a square root to control its use. I just need to know roughly what answer to expect, and I can check the output against that. In a lot of domains, verifying a result is much easier than generating it (hello hash functions). You can recognize a correct proof without being able to find one yourself.

So partial understanding, aimed at the output rather than the internal mechanism, can still support real control. This is the whole basis of code review, financial audits, and every QA process ever built. You don't re-derive the work. You check it.

Where the shortcut stops working

But that route only works if you can actually verify the output. If the result is genuinely opaque to you, if you can't judge whether it's right, or safe, or even close to what you wanted, the shortcut closes. At that point you've stopped exercising control. You've started extending trust.

Those are not the same relationship, and conflating them is how people lose money, ship broken systems (or systems they don't understand), and sign contracts they shouldn't. Control is a feedback loop you're inside of. Trust is a bet you're making from outside it, on the hope that someone or something else is running the loop correctly on your behalf.

An AI-generated result you don't understand well enough to judge isn't something you're controlling. It's something you're deferring to. It doesn't matter how confident the output sounds, how fast it arrived, or how good the tool's track record has been up to that point. If you can't check it against a standard you actually hold, you are not in the loop. You're a passenger. A passenger in a side-car at that.

In the past few months I've been working on Real World Assets (RWA) software. LLMs have played a key role in producing code for me and my colleague. The agent produced something plausible from the specs. Plausible is not the same as correct. Since RWA smart contracts control real assets, understanding exactly what the code does is non-negotiable.

Getting the loop back

If you find yourself deferring instead of controlling, there are really only two ways back into the loop.

One is to build your own understanding of the result, at least well enough to judge it. This is slower and it doesn't scale to every decision you'll ever delegate, but for anything with real consequences, it's worth the investment. This was the approach for my team: we built details specifications and we had one agent build the software and another agent, running a different model, check the results — then manual review on top for anything mission-critical. Then manual review for mission critical software. The key: if you don't understand a result well enough to judge it, don't build on top of it.

The other is to substitute independent verification for personal understanding. Testing, track record, other verifiers, adversarial review, someone whose judgment you trust more than you trust the black box. These can do the "compare to desired state" work even when you personally can't. This is exactly why audits, red teams, and second opinions exist. They're not bureaucratic overhead. They're how you keep a feedback loop closed when your own understanding runs out.

What you can't do is skip both and call it control anyway. That's the mistake I keep watching people make with AI agents right now, myself included on my more optimistic days. The tools are capable enough that the output looks trustworthy before you've actually built a way to check it. The counter-roll gets thrown away, and everyone assumes the original ledger is still accurate.

Bottom line: if you're not in control, you're hoping.