Skip to main content

Neon and Castform's 4B Model Beats GPT-5.6 in Search, Costs 1%

In the world of AI, bigger has often seemed better. But Neon and Castform have just flipped that idea on its head. They've trained an open-source model with just 4 billion parameters that not only matches but beats the mighty GPT-5.6 on document search tasks—and it does so at a fraction of the cost. We're talking about one percent of the cost per inference. That's a game-changer.

So, what's the secret? It's all about rethinking how search works. Traditionally, document retrieval has relied on embedding-based search. You convert documents into numerical vectors and then find similar ones. It's like looking for a book in a library by matching keywords on the spine. But with the rise of AI agents, the search paradigm is shifting. Instead of a one-shot query, agents break down complex questions into smaller ones, decide what to search for, review the results, and then search again. It's a more thoughtful, iterative process that can handle trickier problems.

But here's the catch: that kind of agent-based search usually demands a high-performance model for each step, which means high latency and high costs. According to Neon's typical request specs, a single search with GPT-5.6Sol takes over 10 seconds and costs about $0.03. That might not sound like much, but when you're doing millions of searches, it adds up fast.

Neon and Castform each brought something to the table. Neon provided the document storage and search capabilities, while Castform focused on training the model to decide "what to search for." They used reinforcement learning, where the model tries to complete a task, gets a score based on the results, and then uses that feedback to improve. The evaluation wasn't just about whether the final answer was correct. It also checked if the model found the right documents and cited the appropriate sections. That's a more holistic way to measure success.

The results were impressive. Castform's model scored an average of 1.447 in Neon's validation, beating GPT-5.6Sol's 1.369 and even the control group GPT-5.4's 1.377. That's a new record for this validation.

What does this mean for the future? It suggests that you don't always need a massive, expensive model to get top-tier performance. With clever training and a clear division of labor, smaller models can punch above their weight. For businesses and developers, this could mean faster, cheaper, and more efficient search capabilities. And for the rest of us, it's a reminder that sometimes, the underdog can win.

Image

Key Points

  • Neon and Castform trained a 4B-parameter model that outperforms GPT-5.6 in document search accuracy.
  • The cost per inference is just 1% of GPT-5.6's, making it a cost-effective alternative.
  • The model uses reinforcement learning and a shift from embedding-based to agent-based search.
  • Evaluation considered not just final answers but also document retrieval and citation accuracy.
  • This breakthrough challenges the assumption that bigger models are always better.