The multibillion-dollar bet on language models might be solving the wrong problem.
The Summary
- Andrew Dai, ex-Google DeepMind researcher, launched Elorian AI to build models that reason in images, not just words converted from images
- Current multimodal models like Gemini don't actually "see"—they describe images with words, then reason about the description
- Models that can't count cups on a table or judge spatial relationships aren't ready to build physical-world agents, no matter how well they write code
The Signal
The AI industry has a vision problem. Not the kind where everyone wears Warby Parkers. The kind where frontier language models still can't reason effectively about the physical world. Andrew Dai spent years at Google DeepMind watching this play out. When a model looks at a motorcycle engine schematic and has to first translate it into words to understand it, something fundamental is lost.
Here's what most people miss about "multimodal" models. When Gemini processes an image, it creates a text description, maps those words by meaning, then reasons about the words. Not the image. The words. It's like trying to navigate your house with your eyes closed while someone describes what's in front of you. You might not walk into the coffee table, but you're not going to be rearranging furniture.
"Models that can't count cups on a table fall short of general intelligence, no matter how well they write or code."
This matters because Web4 isn't just chatbots and code generation. It's agents that interact with physical space:
- Robots in warehouses making real-time navigation decisions
- Autonomous systems interpreting sensor data without translation layers
- Design tools that understand spatial relationships, not just describe them
Elorian AI, cofounded by Dai and former Apple ML researcher Yinfei Yang, is building models where visual data gets equal weight with language tokens in the architecture itself. Not as an add-on. Not as a preprocessing step. Native visual reasoning. The bet is that spatial problems, navigation, and physical-world tasks require thinking in the same format the world actually exists in: geometry, depth, relationships between objects in space.
The technical architecture difference:
- Traditional multimodal: Image → Text description → Token map → Reasoning
- Elorian approach: Image → Visual representations → Direct visual reasoning
The timing here isn't coincidence. Language models are hitting capability plateaus. Dai describes them as "incredibly unstable." Scaling laws that worked from GPT-3 to GPT-4 are delivering diminishing returns. Meanwhile, the agent economy needs models that can do more than write emails and summarize documents. They need to move packages, assemble products, navigate spaces, manipulate objects.
You can't build a physical-world agent on a foundation that converts everything to words first. That's not intelligence. That's a translator working overtime.
The Implication
If Dai is right, the next frontier isn't bigger language models. It's models that think the way physical reality works. Watch which companies start hiring computer vision researchers instead of just NLP PhDs. Watch which agent platforms start emphasizing spatial reasoning benchmarks. And if you're building agents that need to interact with anything physical, ask what representation your model is actually reasoning in. If the answer is "words about images," you're using the wrong tool.
The agent economy will separate into two camps: text-native tasks and world-native tasks. Only one of those can be solved by scaling up transformers.