> ## Content Index
> Fetch the complete content index at: https://wire.fourthweb.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# Smartphone-Sized AI Model Hits 1200 Tokens Per Second
- URL: https://wire.fourthweb.ai/smartphone-sized-ai-model-hits-1200-tokens-per-second/
- Published: 2026-07-19T19:02:27.000Z
- Updated: 2026-07-19T19:02:28.000Z
- Description: Someone just trained a function-calling model smaller than a smartphone app that runs at 1200 tokens per second. Cactus Compute released Needle, a 26 million parameter model distilled from Gemini 3.1 that runs function calls on consumer devices at 6000 tok/s prefill and 1200 tok/s decode
- Author: Travis Wright
- Tags: AI Agent Economy, AI Agents, AI Infrastructure, Compute Wars, DeFi, OpenAI, Anthropic, Google AI, Meta AI

**Someone just trained a function-calling model smaller than a smartphone app that runs at 1200 tokens per second.**

### The Summary

- [Cactus Compute released Needle](https://github.com/cactus-compute/needle?ref=wire.fourthweb.ai), a 26 million parameter model distilled from [Gemini](https://wire.fourthweb.ai/tag/google-ai/) 3.1 that runs function calls on consumer devices at 6000 tok/s prefill and 1200 tok/s decode
- The entire model uses a "Simple Attention Network" architecture with just 8 decoder layers, 12 encoder layers, and no feed-forward networks in the encoder
- Trained on 200B tokens in 27 hours on 16 TPU v6e chips, then post-trained on 2B function call examples in 45 minutes, with full weights and dataset open-sourced

### The Signal

The race to the bottom just got interesting. While [Anthropic](https://wire.fourthweb.ai/tag/anthropic/) and [OpenAI](https://wire.fourthweb.ai/tag/openai/) fight over who can build the biggest context window, Cactus Compute went the other direction: a model so small it could run on your watch, but smart enough to actually call functions correctly.

Twenty-six million parameters. For context, GPT-3 has 175 billion. Meta's [Llama](https://wire.fourthweb.ai/tag/meta-ai/) 3.1 405B model requires a server rack. Needle runs on a phone. And it beats Google's FunctionGemma-270m, which is 10x larger, on single-shot function calling tasks.

> "Small models can be finicky, but they run where big models can't."

The architecture is what makes this work. Most language models use transformer blocks stacked with attention layers and massive feed-forward networks. Needle strips that down to a "Simple Attention Network" that ties weights between the embedding and output layers, uses grouped query attention with rotary position embeddings, and completely removes feed-forward networks from the encoder. The result is a model that knows how to parse a text query, understand available tools, and make a single clean function call without hallucinating parameters or getting lost in multi-turn conversation.

This matters because function calling is the actual hard part of agent automation. Every AI assistant needs to translate "book me a flight to Chicago next Tuesday" into structured API calls with the right parameters. Getting that wrong once breaks user trust. Getting it right at 1200 tokens per second on local hardware changes what's possible on mobile devices.

The training approach is pure efficiency: 200 billion tokens of pretraining in 27 hours, then 2 billion tokens of function call examples in 45 minutes. Total training cost was likely under $10,000\. Compare that to Llama 3's reported $100 million training run. [Cactus open-sourced the weights and the dataset generation code](https://github.com/cactus-compute/needle?ref=wire.fourthweb.ai), which means anyone can finetune this for their specific function set without cloud [GPU](https://wire.fourthweb.ai/tag/compute-wars/) bills.

### The Implication

The agent economy doesn't need foundation models for everything. It needs specialized models that do one thing well and run where the user is. Needle proves you can build production-ready function calling for phones, watches, and edge devices without waiting for network latency or paying per-token API fees.

If you're building consumer AI products, this is your signal to stop assuming you need cloud inference. The models that win local deployment will be the ones that ship in apps, run offline, and still work when your user is on a plane. Download the weights, test it on your function set, and finetune if it's not perfect. That's the whole point of 26 million parameters: you can actually retrain it on your laptop.

### Sources

[GitHub Trending Python](https://github.com/cactus-compute/needle?ref=wire.fourthweb.ai)