AI Costs Are Rising, What Are We Doing Differently?

A significant trend has emerged over the last six months: the increasing cost of AI. We anticipated this shift two years ago, recognizing that the era of inexpensive, powerful AI wouldn't last indefinitely. This foresight guided us to develop agentic infrastructure that leverages low-cost, highly efficient models. We've found that a cluster of these smaller, more affordable agents can be as, or even more, effective than a single, powerful large language model (LLM).

This strategy has provided us with a distinct advantage in the AI landscape. While many companies focused on creating impressive proof-of-concept systems with powerful AI, we've been meticulously building internal systems driven by large networks on on-rail, low cost LLMs and SLMs. These systems utilize sophisticated orchestration layers that coordinate numerous smaller models to achieve meaningful outcomes. This approach demands more development effort and isn't as immediately plug-and-play as using models like Opus 4.8. However, our innovative agentic infrastructure allows us to construct agent networks capable of diverse tasks, from building websites and organizing CRM data to guiding users on data organization within our project management tools.

This foundational work is crucial for navigating the rising costs of AI. Even with slight price increases for our low-cost models, the overall expense remains significantly lower than if we had adopted powerful, less constrained models.

A Peak Behind the Curtain

How is this possible, and how did OMNIUX build these systems? While we can't reveal all our "secret sauce," the key takeaway for anyone looking to leverage a low-cost agentic system is the importance of compartmentalizing your agents.

For instance, imagine a system designed to read a file, clean and parse its data, and then ingest it into a CRM. While it's entirely possible for a single agent to handle all these tasks, it would also be incredibly expensive.

Consider our Opus 4.8 example: this model is more than capable of performing such a task in a single pass, with minimal back-and-forth. If you increase the "thinking budget," the model can reason about the data it's parsing and the instructions provided in a system prompt. However, this could result in a $5 cost for a single file ingestion, with two minutes spent reasoning and one minute acting.

This approach is acceptable for one-off jobs. But what if you need to upload an entire CSV file? Every token added to the equation can exponentially increase the operational cost of the model. If the thinking budget is increased to handle the load, you could be looking at a two to three-fold increase in both time and cost.

With a low-cost agent infrastructure, you would break the problem down into distinct steps: one agent for reading and ingesting the file, another for cleaning the data, and a third for formatting the output and performing operations with that data. By distributing the problem across multiple agents, you significantly reduce the workload for each. This allows for smaller context windows and even parallelization of tasks.

The Context Problem

Back in 2024, a significant challenge emerged concerning the relationship between model capability and their context window. For those unfamiliar, a model's context window refers to the amount of information it can retain as part of its short-term memory.

Every token in the user's prompt, the system prompt, and the model's response consumes space within this window. While most current models can retain up to 1 million tokens of context (roughly 750,000 words), and some utilize caching to compress this information, exceeding this limit leads to a problem known as "context rot." This is where hallucinations become particularly problematic.

We are familiar with hallucinations occurring due to issues in a model's training data, but they also arise when a context window is maximized. Larger models are adept at mitigating context rot through constant error correction, reasoning, and problem-solving. However, this process can lead to a recursive minefield, quickly resulting in substantial cost increases.

Lower-cost models also suffer from context rot and will hallucinate more frequently once 50-60% of their context window is filled. Realistically, though, you typically do not need to use more than 15-20% of a model's context window at any given time, especially in agentic systems.

The Age of Headless AI

With our data ingestion example, we discussed breaking a problem down across multiple agents. By doing this, we can dramatically decrease the amount of context required for a model to perform a task. As part of a wider orchestration loop, a single "supervisor" agent sits at the very top. This agent is in charge of proofreading the work of smaller models. If the work is incorrect or does not correctly follow a guided plan, the supervisor can instruct the model to correct itself.

By adopting this approach, we achieve the following:

• Reduce the number of tokens consumed by a single agent.
• Dramatically reduce the likelihood of context rot.
• Spend significantly less money on a task because the context window is smaller, and thinking budgets can be greatly reduced.
• Get a response back faster than if a single model were used.

This ushers in a new era of "Headless AI," where agents work behind the scenes to complete a task, rather than a single user. As mentioned before, this requires more initial setup. You will need an agent harness, focused system prompts, and most likely a bit of patience.

AI has kick-started a new digital revolution, and when the era of cheap, highly capable models is over, companies will be reaching for this low-cost, agent-first approach to complex problem-solving.

What Should I Build With?

If you're a developer or someone aiming to reduce agentic infrastructure costs, consider our approach: begin with the smallest available models and utilize frameworks that offer maximum control. We've found Google's Flash and Flash Lite models to be excellent workhorses. Anthropic's Haiku models are also highly capable, particularly for tasks requiring a touch of creative flair.

For building this infrastructure, LangGraph is the optimal choice. This framework provides unprecedented control over your agent graph. While it requires a bit more effort than LangChain, the performance and cost benefits make it the leading framework on the market.