The data layer is catching up to AI's speed, and the companies solving storage, routing, and agent-ready databases just became infrastructure plays worth hundreds of billions.
The Summary
- Databricks raised $5B at a $190B valuation in August, building Lakebase—a database architecture that spins environments up and down at agent speeds, not human speeds.
- AI agent creation of test environments on Databricks grew from 0.1% to 97% in two years, exposing how legacy databases can't keep up with autonomous workflows.
- DataPelago (acquired by NetApp) routes compute tasks to the optimal processor—CPU, GPU, NPU, or FPGA—delivering 10x performance at 80% lower cost without code changes.
- The bottleneck isn't models anymore. It's the pipes feeding them data and the storage keeping pace with supercomputer-scale training runs.
The Signal
We talk about agents like they're coming. They're here. Databricks saw AI agent environment creation jump from 0.1% to 97% in 24 months. That's not adoption. That's replacement. Agents are building, testing, iterating, and tearing down environments faster than any human dev team could coordinate a standup. Traditional databases weren't built for this cadence. They're built for applications that humans write, deploy, and maintain over quarters and years, not workflows that spin up 50 parallel test environments in an afternoon.
Databricks responded with Lakebase, a full rearchitecture that treats database environments as disposable, instant infrastructure. The technical leap is handling the "multitude of small, fast workloads" that agents generate—not the monolithic batch jobs of the past. The market responded: $5 billion raised, $190 billion valuation, 80% year-over-year revenue growth. This is infrastructure money, not feature money.
"AI agent creation of environments jumped from 0.1% to 97% in two years—legacy databases couldn't keep up."
Meanwhile, the processor layer is fragmenting. Training and inference don't just run on GPUs anymore. You've got CPUs for certain logic tasks, NPUs for low-power inference, FPGAs for custom acceleration. The problem: routing workloads to the right chip without rewriting code every time. DataPelago's Nucleus solves this with a transparent layer that directs compute requests to the optimal processor type. The result: 10x performance gains, 80% cost reduction on Apache Spark workloads. NetApp bought them in August and made Nucleus the core of its AI stack. That's not an acqui-hire. That's buying the glue that holds heterogeneous compute together.
Then there's storage. DDN's EXAScaler is a parallel file system built for supercomputer-scale training runs. The constraint isn't model size anymore—it's data throughput. Training larger models means feeding them faster. Slow storage equals underutilized GPUs, which means wasted money at cloud scale. EXAScaler handles the fire hose.
Key pattern emerging:
- Databases are shifting from static repositories to dynamic infrastructure
- Compute routing is becoming invisible—agents don't care what chip runs the job
- Storage is the new bottleneck as models scale past GPU capacity
What ties this together: the data layer is being rebuilt for autonomous systems, not human-in-the-loop software development. The assumption that a person writes code, deploys it, and monitors it is breaking. Agents iterate in minutes. They need databases that appear and vanish on demand, processors that self-optimize without configuration, and storage that matches the speed of the most expensive hardware in the stack.
The Implication
If you're building agents, your infrastructure costs are about to drop or explode depending on whether you're using tools built for this. Legacy databases will kill your iteration speed. Cloud bills will balloon if you're not routing compute efficiently. Storage will bottleneck the training runs that matter.
Watch the companies building the pipes, not just the models. Databricks, NetApp, DDN—they're infrastructure plays in an agent economy. The $190 billion valuation tells you the market sees it too.