The edge AI revolution just skipped the iPhone and went straight to your toaster.
The Summary
- Cactus Compute released Needle 2, a 14MB agentic LLM that runs full sessions in 28MB of RAM — targeting the 21 billion IoT devices that aren't Macs or high-end phones
- At 45 million parameters with 2-bit compression, it hits 500 tokens/sec on a Raspberry Pi 5 and trades wins with models 5-70x larger on tool-calling benchmarks
- By reframing intelligence as function mapping rather than world knowledge generation, they built something that runs on wearables, smart home devices, and sub-$200 phones with no NPU
The Signal
The edge AI conversation has been living in a bubble. When people say "on-device AI," they mean MacBook Pros and flagship phones. They mean the 1.5 billion premium devices with dedicated neural processors and high-end GPUs. Meanwhile, 21 billion connected IoT devices sit in the real world, most running on tight power budgets with cheap or nonexistent acceleration hardware.
Needle 2 targets that 21 billion. The full model is a single 14MB binary. A complete inference session runs in 28MB of RAM. On a Samsung A-Series phone, which costs under $200 and ships without an NPU, it processes 300-700 tokens per second. On a Meta Quest 3S, it hits 400-1,500 tokens/sec. On a Raspberry Pi 5, it runs at 500 tokens/sec.
"Edge AI has lately meant Macs and PCs, but that is just 1.5 billion of over 21 billion connected IoT devices in the world today."
The efficiency comes from architectural choices. Needle uses Simple Attention Networks, which spend 70 MFLOPs per token. A conventional transformer of the same width and depth would spend 164. Even a transformer squeezed down to Needle's parameter count would burn 87 MFLOPs per token. On an always-on device, every MFLOP is milliwatt-hours. Needle spends 7-85x fewer of them than the smallest performant LLMs.
Key performance stats:
- 45 million parameters at 2-bit compression
- Trades wins with LFM2.5 (230M parameters) and Apple Foundation Model on tool-calling benchmarks
- Those competitors run at f16 precision while Needle runs at 2-bit — making Needle 5-70x smaller
The core insight: when you frame intelligence for consumer devices as functions with typed parameters, the hard part becomes mapping messy human sentences onto those functions. Which function. Which values. That problem requires no world knowledge and no open-ended prose generation. Which is why 45 million parameters suffice.
This matters because the distribution of compute in the real world looks nothing like the distribution of compute in AI research labs. In emerging markets, most phones ship under $200. No NPU. Cheap GPU. Your smart home devices, wearables, and small robots like Reachy Mini are running even tighter budgets. If agents are going to live on these devices, they need to operate inside these constraints.
The Implication
The agent economy doesn't scale if it requires cloud API calls for every interaction. Latency kills conversational interfaces. Privacy concerns kill adoption in sensitive contexts. Network costs kill viability in bandwidth-constrained regions. The devices that actually need agentic interfaces — wearables that need to respond instantly, home automation that shouldn't ping AWS to turn on a light, robots that need to operate offline — can't wait for the industry to trickle flagship phone capabilities down to them.
Needle's approach suggests a different path: purpose-built architectures optimized for tool calling and device control, not general knowledge work. If that tradeoff holds, the bulk of the IoT world just became viable territory for local AI agents. Watch what happens when 21 billion devices can run intelligence locally instead of 1.5 billion.