There is a quiet assumption running through most AI products today: that intelligence is a function. You give it an input, it returns an output, and you are done. Prompt in, completion out. Image in, label out. This framing is convenient, it maps neatly onto how we deploy models behind an API, and it is almost always wrong about the problems worth solving.
The systems I care about are not functions. They are loops. They observe the world, build a representation of it, decide what to do, act, and then observe the consequences of that action — continuously, under uncertainty, with no clean boundary between "input" and "output." A neural interface that adjusts stimulation based on the brain's response is a loop. A voice agent that has to recover when a person interrupts it is a loop. A monitoring system that has to keep estimating a patient's state as their physiology drifts is a loop. The intelligence is not in any single step. It is in how the steps close on each other.
A model is a component. A closed loop is a system. The gap between the two is where almost all of the real difficulty — and almost all of the value — actually lives.
Open-loop is easy to demo and hard to trust
Open-loop systems are seductive because they demo beautifully. You can show a single impressive output and let the audience extrapolate. But open-loop systems share a structural weakness: they have no mechanism to know when they are wrong, and no mechanism to recover. They emit a result and move on. If the result is bad, the consequences land on whoever is downstream.
A closed-loop system is held to a harder standard. It has to keep working as conditions change. It has to detect that its own estimate is degrading. It has to act, see that the action didn't produce the expected effect, and correct. This is the difference between a model that classifies a signal once and a system that has to track a state that is actively moving away from it.
The four verbs I keep coming back to are sense, represent, reason, act — and then the loop closes and you sense again. Each is a hard problem on its own. The interesting failures happen in how they compose.
- Sense — acquire signal from a noisy, partially observable world. The signal is almost never the thing you actually care about; it's an indirect, confounded shadow of it.
- Represent — turn that signal into a state you can reason over. This is where most of the modeling difficulty hides, because the right representation is rarely the raw measurement.
- Reason — decide what to do given an estimate that is uncertain and possibly stale.
- Act — intervene in the world, knowing the action itself perturbs the thing you were measuring.
The world is the part you don't control
The reason loops are hard is that the environment has a vote. In a function, the input is given. In a loop, your own actions change the distribution of inputs you'll see next. A stimulation pulse changes the neural activity you're about to record. A spoken response changes what the person says next. A treatment changes the physiology you're monitoring.
This coupling breaks a lot of comfortable assumptions. The data is no longer independent and identically distributed, because you generated part of it. Errors compound across time instead of staying local to a single prediction. And the cost of a mistake is not "one wrong answer" — it's a trajectory that drifts somewhere you didn't intend.
Designing for this means designing for uncertainty as a first-class citizen. The system needs to represent not just its best estimate but its confidence in that estimate, and it needs to behave differently when confidence is low — gathering more information, acting conservatively, or handing control to a human. A loop that acts decisively on a bad estimate is more dangerous than one that doesn't act at all.
Latency is part of correctness
There is a property of loops that function-thinking misses entirely: time. In a function, you can take as long as you like; the answer is judged only on whether it's right. In a loop, when you act is part of whether the action is correct, because the world has moved on while you were deciding. An estimate that was accurate two seconds ago can be actively misleading now if the state is drifting.
This shows up everywhere once you start looking. In a voice conversation, a perfectly-worded reply that arrives half a second late lands as hesitation or steps on the other person mid-sentence. In a monitoring system, an alert that fires correctly but too late is, operationally, a wrong alert. In a control loop, latency between sensing and acting determines whether the system is stable at all — too much delay and your corrections arrive out of phase with the thing you're trying to correct, and you amplify the very oscillation you meant to damp. So part of designing a loop is designing its tempo: how fresh the estimate has to be, how quickly the system must commit, and what it does when the honest answer isn't ready in time. None of that is visible if you think of intelligence as a function.
Why this is a systems problem, not a model problem
It is tempting to believe that a sufficiently good model dissolves all of this. It doesn't. A better model gives you a better single step, but the loop's behavior is a property of the whole architecture: the latency between sensing and acting, the way state is carried forward, the fallback behavior when a component fails, the human-in-the-loop controls, the way feedback is logged and fed back into learning.
I've watched the same lesson repeat across very different domains — transcranial ultrasound, non-invasive brain monitoring, and now agentic AI platforms. In every case, the component everyone obsesses over (the transducer, the decoder, the language model) was necessary but not where the system lived or died. The system lived or died on co-design: whether the sensing, the representation, the control logic, and the failure handling were designed together, against the constraints of the real environment, rather than optimized in isolation and bolted together at the end.
This is also why "just swap in the better model" so often disappoints. You improved one step in a loop whose behavior was determined by the other steps and by how they were coupled. The same dynamic explains a frustration anyone who has shipped these systems will recognize: a change that makes the headline metric better can make the system worse, because you optimized a step in isolation and quietly broke an assumption the rest of the loop depended on. Loops have to be reasoned about as wholes.
Feedback is what makes a loop a loop
It is worth dwelling on the last verb, because it's the one most often skipped. A system that senses, represents, reasons, and acts — but never feeds the consequences of its actions back into how it behaves next time — isn't really a closed loop. It's an open-loop system running in a fast cycle. The loop only truly closes when acting changes future sensing and the system uses that to get better, either within a single interaction or across many of them.
This is subtle in practice because there are two different timescales of feedback, and you need both. There's the fast loop — within an interaction, the system observes that its action didn't land as expected and corrects immediately. And there's the slow loop — across interactions, the system accumulates evidence about where it tends to be wrong and updates. Most deployed "AI" closes neither: it's frozen at training time and blind to the downstream effects of its own outputs. The systems that compound in value over time are the ones where real-world feedback actually flows back into behavior, instead of being discarded the moment the output is produced.
A worked example: the neural interface
Make this concrete with the loop I find clearest. Imagine a system that records neural activity and delivers stimulation in response — the canonical closed-loop neurotechnology. The open-loop version is easy to describe and easy to oversell: detect a pattern, fire a pulse. The closed-loop reality is where every hard problem hides.
It senses neural activity, but the signal is faint, drifting, and partly buried in the artifact created by the stimulation itself. It represents that signal as an estimate of brain state, knowing the right representation is not the raw voltage but something derived and uncertain. It reasons about whether and how to intervene, under real ambiguity about whether its estimate is even correct right now. And it acts — delivers a pulse that immediately changes the very activity it was just measuring, so the next thing it senses is partly its own footprint.
Every coupling I described above is present at once. The action perturbs the measurement. Errors propagate forward in time rather than staying local. Latency between sensing and acting determines whether the loop is stable or whether it oscillates. And the cost of being wrong is not an abstraction — it's a stimulation delivered to a person on the basis of a bad estimate. You cannot build this system by perfecting the decoder in isolation and bolting on a stimulator at the end. The decoder, the artifact handling, the control policy, the safety limits, and the latency budget are one design problem, and they have to be solved together. That, in a single device, is the entire argument of this essay.
What I look for
When I evaluate a system now, I mostly ignore the headline capability and ask loop-shaped questions:
- What does it sense, and how indirect is that signal from what it actually needs to know?
- Does it carry state forward, or does it start from scratch each step?
- How does it behave when its own estimate is uncertain or wrong?
- What happens when a component fails — does the loop degrade gracefully or catastrophically?
- Where does the human sit in the loop, and how is control handed back and forth?
- How does feedback from acting actually change future behavior — or is the loop never really closed?
These questions cut through demos quickly. They are also, not coincidentally, the questions that separate research prototypes from systems you can deploy in the real world. A prototype only has to produce one good output in front of an audience. A deployed loop has to keep producing good behavior, unattended, as the world shifts underneath it — and it has to fail safely on the days it can't.
My long-horizon bet is that the most important systems of the next decade — in neurotechnology, in healthcare, in human–AI interaction — will be defined not by how smart any single model is, but by how well they close the loop with humans and with the physical world. The model is the easy part to admire. The loop is the hard part to build, and it's the part that matters.