The best benchmark scores in the world don't mean much when your agent can't send an email.
The Summary
- DeepSeek's V4 Flash topped leaderboards but only completed 53.8% of real-world agent tasks when Composio tested it across eight different agent frameworks on 30 multi-step workflows.
- The same model produced wildly different results depending on orchestration layer, showing that harness quality matters more than raw model capability for production agents.
- DeepSeek is hiking prices on V4 Flash and Pro, moving beyond the "cheap Chinese model" narrative as enterprises figure out where models actually fit.
The Signal
Composio put DeepSeek V4 Flash through a gauntlet that actually matters: 240 runs across eight agent harnesses, executing 30 complex, multi-step tasks on live production tools. Gmail, GitHub, Slack, Google Sheets. The stuff agents need to do in the real world, not on a static benchmark.
The results tell you everything about the gap between model capability and deployment reality. Only 129 of 240 runs passed. Just six of the 30 workflows succeeded across every harness tested. The same model, same task, different orchestration layer meant the difference between success and failure.
"Orchestration, not raw model capability, may decide whether the model succeeds in enterprise settings."
This is the infrastructure layer everyone's been ignoring while obsessing over parameter counts and benchmark wars. The harness is the translation layer between what a model can theoretically do and what it actually does when you tell it to update a spreadsheet based on Slack messages and file a GitHub issue if something breaks.
The variation in results comes down to tool configuration, caching behavior, retry logic, and how the provider stack handles context. One harness might cache tool outputs intelligently and retry failed API calls. Another might not. Same model, different success rate.
DeepSeek's price hikes arrive right as this orchestration gap becomes visible. The models that dominated OpenRouter's usage charts since late July are getting more expensive. The cheap-but-capable pitch that made them irresistible to developers building coding assistants is shifting. But the move makes sense if you're DeepSeek: you've proven the models work, demand is "insane" according to early adoption numbers, and enterprises are starting to figure out which workflows justify which models.
Key facts emerging from early production use:
- V4 Flash (284B parameters) is built for volume and speed, V4 Pro (1.6T parameters) for complex workflows
- Both models have flexible reasoning modes (low, high, max) and thinking modes that apply chain-of-thought reasoning
- Flash currently dominates OpenRouter's weekly token usage, the most-used model on the platform
The enterprise question isn't "which model is cheapest" anymore. It's "which orchestration stack can actually deploy this model reliably at scale." That's a harder problem than training a bigger model, and it's where the money will concentrate as agents move from demos to production.
The Implication
If you're building agents, test on real tasks with real tools, not benchmarks. The orchestration layer is where agents live or die in production, and right now most teams are using whatever harness shipped with their model provider without thinking about retry logic, caching strategy, or tool configuration.
Watch for orchestration platforms to become the new chokepoint. The model wars are giving way to the harness wars. DeepSeek's price hikes won't matter if the infrastructure to reliably deploy their models doesn't exist. And if it does, higher prices just mean someone else captured the margin.