The most remarkable thing about large language models is also the source of their deepest limitation: they learned the world entirely through text. Everything they know, they know because a human wrote it down. This turns out to be enough to be astonishing — language carries a staggering amount of compressed human knowledge — and it also turns out to be a ceiling, one I think the field is only beginning to take seriously.
The fallacy I want to name is the belief that text is enough — that if you scale language models far enough, you converge on something like general intelligence, because language is the medium of thought and therefore contains everything. I understand the appeal; the progress has been so steep that extrapolating it feels safe. But I think it mistakes a compression of reality for reality itself, and the gap between the two is exactly the part of the world that matters most for the systems I care about. Language-only intelligence plateaus, and where it plateaus is the threshold of the physical, sensed, time-varying world.
Language is downstream of experience
Start with what language actually is. Text is not the world; it's a record humans made about the world, after the fact, filtered through what we found worth saying. It is a lossy, biased, radically compressed projection of reality onto a symbolic channel. A model trained only on text is learning the shadow, not the object — and inheriting, baked in, every limit of the shadow.
Three of those limits matter most. First, text captures only what humans bothered to write down, which is a vanishingly small and highly selective slice of what's true. The overwhelming majority of what happens in the world — every physiological process, every physical dynamic, every signal no one transcribed — was never written and is simply absent from the corpus. Second, text inherits all of our errors and biases; it's a record of what we believe and say, not of what is, and a model that learns from it learns our confusions as faithfully as our knowledge. Third, and most fundamentally, language is downstream of experience. We had bodies, senses, and a physical world long before we had words, and the words are a compression of that prior, richer thing. Learning the compression is not the same as having the experience that produced it.
This is why a text-only system can describe the taste of salt, the feeling of a fever breaking, the way a gait changes before a fall, or the rhythm of a heartbeat sliding into arrhythmia — fluently, even beautifully — without any of the grounding that the words are pointing at. It has the label without the referent. For a lot of tasks, the label is enough. For the tasks that involve actually sensing and acting in the physical world, it is precisely not enough, because the label was never the thing you needed.
The world is the missing dataset
Here's the reframe I find most useful. We talk about running out of training data, as if the supply of text were the limit. But text was never the dataset that mattered for understanding the physical, biological, time-varying world. That dataset hasn't been exhausted — it's barely been collected. The world itself, sensed directly, is the missing dataset, and it dwarfs everything we've ever written by orders of magnitude.
Think about what's in that larger dataset and absent from the text corpus. The continuous physiological signals of a living body — the slow drift of a patient's state, the subtle precursors of deterioration that show up in dynamics long before they show up in symptoms anyone would write down. The behavior of physical systems over time, where the information is in the trajectory, not any single snapshot. The full bandwidth of the senses — what things actually look, sound, and feel like, at a fidelity language can only gesture at. And the consequences of action: what happens when you do something and the world responds, which is the only way to learn cause and effect rather than the correlations text records.
None of this is written down, because most of it is unwritable — it lives in signals, not sentences. A text model can't learn it not because the model is too small but because the information was never in the channel it learned from. You cannot scale your way to knowledge that isn't in your data. The path past the plateau isn't a bigger corpus of the same kind of data. It's a different kind of data entirely: the world, sensed directly.
Sensing, signals, time, embodiment
If text is the thin channel, what's the rich one? Four things I keep coming back to, each a dimension the text-only view collapses.
Sensing. Direct measurement of the physical and biological world — the modalities I've spent my career on, where you're recovering a hidden state from indirect, confounded signals. This is information that exists only if you go and acquire it, and it's the substrate of everything that touches the real world: health, the body, physical systems, the environment.
Signals over time. A snapshot is a label; a trajectory is understanding. So much of what matters — in physiology, in dynamics, in any system that evolves — is in how things change, in the temporal structure that a single static input throws away. Text mostly encodes states and conclusions; the world is made of dynamics, and you can only learn dynamics from data that has time in it.
Time-series and the precursor problem. The most valuable thing you can often know is what comes next — the early, subtle signature of a change before it becomes obvious. That signal lives in continuous measurement over time, and it's exactly the signal that never makes it into text, because by the time something is worth writing down, the precursor has already passed.
Embodiment. Intelligence that can act in the world and observe the consequences learns things that no amount of passive reading can teach — cause and effect, the actual response of the environment to intervention, the closing of the loop between acting and sensing. This is the deepest gap of all, because text is fundamentally a record of observation, and a great deal of real understanding only comes from intervention.
What "multimodal" isn't
A fair objection at this point is that the field already has an answer: multimodal models. They take images, audio, video. Isn't the sensed world already being folded in? Partly — and the distinction here matters, because it's easy to mistake the current version of multimodal for the thing I'm actually arguing for.
Most of today's multimodality is, in effect, translation into the text channel. An image is encoded into a representation aligned with language; audio is transcribed or captioned; the other modality is treated as something to convert into the space the language model already understands. This is genuinely useful, and it's progress. But notice what it does: it brings other modalities to text, rather than treating them as first-class sources of information in their own right. The center of gravity is still language, and the other signals are tributaries flowing into it. The model still fundamentally reasons in the text-shaped space; it has just gained better on-ramps.
The deeper version is different, and it's the one I think the sensed-world frontier actually requires. It treats signals on their own terms — physiological time-series, confounded sensor measurements, continuous dynamics — without forcing them through a linguistic bottleneck that was never designed to carry them. A heartbeat sliding toward arrhythmia is not well described as a caption; the information is in the waveform, the timing, the trajectory, and flattening it into words throws away precisely the structure that matters. Learning from the sensed world means building systems whose native representations are suited to that world — not pipelines that translate everything into the channel text models already speak.
There's also a question of where the data even comes from. Image and audio multimodality leaned on the same trick text did: huge quantities of already-collected, internet-scale data. The hard sensed modalities — the biosignals, the hard-to-measure physical quantities, the continuous physiological streams — have no such corpus waiting to be scraped. They have to be acquired, deliberately, by building the sensors and deploying them and solving the inverse problems. So "we already have multimodal models" understates the gap. We have models that can ingest the easy, abundant modalities. The frontier is the hard, scarce ones — the signals that matter most and exist least in any dataset.
Why this is the harder and more valuable problem
There's a reason the field went to text first, and it isn't that text was the most important data. It's that text was the available data — already digitized, already labeled by virtue of being language, sitting in enormous quantities on the internet, waiting to be scraped. The sensed world has no such corpus. To learn from it, you have to go build the sensors, deploy them, acquire the signals, and solve the genuinely hard inverse problems of turning confounded measurements into meaningful state. There's no shortcut, no pre-existing pile to scrape. That difficulty is exactly why the text-only path was taken first — and exactly why the next frontier is so much harder and, I'd argue, so much more valuable.
It's more valuable because the highest-stakes problems live in the sensed world, not the written one. Health is the clearest example: a person's physiological state is a continuous, high-dimensional, time-varying signal that is almost entirely absent from any text corpus, and the value of being able to sense it — to catch deterioration early, to monitor continuously and non-invasively, to close a loop between measurement and treatment — is enormous and almost entirely untapped. The same is true across the physical world. The places where intelligence could matter most are precisely the places text can't reach.
And it's harder in a way that's actually good news for builders, because difficulty that doesn't commoditize is the foundation of something durable. Anyone can call a text model. Almost no one can sense a hard-to-measure signal from the physical world and turn it into something reliable enough to act on. That gap — between the abundant, commoditized text channel and the scarce, hard-won sensed one — is where I think the most important and most defensible work of the next decade gets done.
The plateau and the path past it
I want to be careful not to be misread as dismissing language models. They are a genuine and foundational achievement, and language is a real and important channel — the interface through which much of human–AI interaction will flow. The fallacy isn't using text. The fallacy is believing text is sufficient — that scaling the written channel converges on understanding the world.
It doesn't, because the written channel is a thin compression of a vastly richer reality, and the parts it leaves out — the sensed, the temporal, the embodied, the physical — are exactly the parts that matter for systems that have to operate in the actual world rather than describe it. Language-only intelligence will keep getting more impressive within its channel and will keep plateauing at the channel's edge.
The path past the plateau is not more text. It's the world — sensed directly, measured over time, learned through action. That's the missing dataset, and it's the one I've spent my career trying to collect, one hard signal at a time. The teams that go and gather it, rather than waiting for it to appear in a corpus that will never contain it, are the ones who will build the intelligence that finally touches the physical world instead of just talking about it.