Scott Turner The teams building production AI agents that actually work have shifted their investment from chasing frontier models to engineering the system around the model. Six architectural patterns make the difference: semantic tool selection to prevent tool overload, context engineering to combat attention decay, query decomposition for reliable complex tasks, orchestration layers that route intelligently, standardized integration protocols like MCP, and LLM-as-judge quality evaluation loops.
There is a persistent belief in the AI world that progress comes from bigger models. More parameters, more training data, more compute. And for a while, that was true. But the organizations actually shipping production AI systems have quietly moved on from that equation.
The pattern I keep seeing with clients and in my own work is the same: the teams getting reliable results are not chasing the frontier model leaderboard. They are investing in the architecture around the model. Orchestration logic. Context management. Quality evaluation loops. The model itself is becoming a commodity. The system you build around it is the differentiator.
This is a fundamental shift from “which model should we use?” to “how do we design a system where the model is just one component?” And the answers are often counterintuitive. Giving your AI more tools can make it worse. Larger context windows can degrade performance. The most important layer in your stack might be the one your users never see.
Here are six truths about building AI agents that actually work, drawn from the patterns that separate demo-grade prototypes from systems you can trust in production.
1. More Tools Can Make Your AI Dumber
It seems obvious that giving an AI agent more tools should make it more capable. In practice, the opposite is often true. There is a threshold where adding tools causes a catastrophic drop in the agent’s ability to pick the right one.
The numbers are stark. With around 50 tools available, most models maintain 84 to 95% accuracy in selecting the correct tool. Scale that catalog past 700 tools and accuracy collapses to somewhere between 0 and 20%. The agent is not getting smarter with more options. It is drowning in them.
Two things drive this. First, loading hundreds of tool definitions into the context window for every request burns through tokens and compute. Second, models suffer from a “lost in the middle” effect where tools positioned in the center of a long list are systematically overlooked. The model pays attention to what is at the top and bottom, and everything in between becomes noise.
The fix is a pattern called semantic tool selection. Before the main model ever sees the request, a fast, lightweight classifier analyzes the user’s query and filters the tool catalog down to a small, relevant subset. The agent only sees the tools it is likely to need. Systems using this approach report roughly 3x improvement in tool selection accuracy and a 99% reduction in token consumption.
The lesson for anyone building agent systems: do not hand your model a Swiss Army knife with 700 blades. Give it the three it needs for this specific job. The intelligence is in the routing, not the catalog.
2. Bigger Context Windows Are Not a Silver Bullet
The race toward larger context windows creates an appealing illusion: if the model can hold more information, it will produce better answers. But LLMs operate on a finite attention budget, similar to human working memory. The more you pack into the context, the thinner that budget gets stretched. Details from earlier in the conversation start to fade. The model “forgets” things it was told 10,000 tokens ago, even though the text is technically still there.
This is what I call context rot, and it is one of the most common failure modes in production agents. Teams dump entire conversation histories, full documents, and verbose tool outputs into the context window, then wonder why the model gives inconsistent answers.
The discipline that fixes this is context engineering: curating the smallest possible set of high signal tokens to guide the model effectively. Two strategies consistently work.
The first is compaction. Before the context limit is reached, the agent summarizes the critical details of the conversation so far. That compressed summary becomes the foundation of a fresh context, preserving what matters while shedding the noise.
The second is structured note taking. Instead of keeping the entire transcript in memory, the agent writes key information to an external store: a to do list, a strategic plan, progress notes, confirmed user details. This information gets pulled back into the context selectively, as needed. The agent gets reliable long term memory without polluting its immediate attention.
The teams that treat context as a precious, finite resource consistently outperform the ones that treat it as an unlimited bucket.
3. Decompose Before You Solve
Real user queries are messy. A single sentence can contain multiple hidden questions and intents packed together: “What are the baggage fees and flight status for flight 1234?” Treating that as one task is a recipe for an incomplete or hallucinated answer.
The pattern that reliably solves this is query decomposition. Before the agent tries to answer anything, a planning step breaks the complex query into smaller, independent sub queries that can be executed one at a time.
That baggage and flight status question becomes two distinct tasks: retrieve the baggage policy from the knowledge base, and check real time flight status through an API. Each sub query gets routed to the right tool, executed independently, and then a final synthesis step combines the results into a single coherent answer.
This sounds like it would be slower. In practice, it is faster and dramatically more reliable. Each sub task is simple enough that the model rarely fails. The tool selection is obvious because the scope is narrow. And the final synthesis step is working with verified data instead of trying to juggle multiple objectives simultaneously.
I have started applying this principle beyond just query handling. Any time a product team asks an agent to do something complex in a single pass, I ask them to break it apart first. Complex prompts that try to accomplish five things at once are brittle. A sequence of simple, verifiable steps is robust. The decomposition itself is often the most valuable engineering work in the entire agent pipeline.
The principle is old, but it applies perfectly here: divide, then conquer.
4. The Orchestrator Matters More Than the Model
If you ask most teams what the most important part of their AI system is, they point to the model. Whatever frontier model they are running this quarter. But in every production system I have worked with or reviewed, the value has shifted away from the model and into the orchestration layer.
Think of it this way. A brilliant employee who does not know which tasks to work on, who to hand things off to, or when to escalate is not particularly useful. The same is true of a powerful LLM without a system telling it what to do and when. The orchestrator is the manager. The model is the employee.
In practice, this means a lightweight, inexpensive router model often sits in front of the expensive frontier model. The router classifies incoming requests and decides: does this need the full reasoning power of a large model, or can a smaller, faster model handle it? This prevents your most expensive resource from doing grunt work, which saves real money at scale and reduces latency for straightforward requests.
The framework landscape reflects this shift. LangGraph is built for structured workflows where state management matters. CrewAI handles team style collaboration between multiple agents. The common thread is that the intelligence of the system lives in how tasks get routed, sequenced, and handed off, not in any single model’s capabilities.
When clients ask me which model they should use, my real answer is: it matters less than you think. Get the orchestration right first. The model is a replaceable component. The system design is not.
5. Standardize the Interface, Not the Intelligence
Integration has always been the quiet tax on AI systems. Every new tool, API, or data source requires a custom connection. Every new client application needs its own integration for every backend service. This is the N times M problem, and it scales exactly as badly as it sounds.
The Model Context Protocol (MCP) is the most promising answer I have seen to this problem. The simplest way to understand it: MCP is the USB-C of AI. Before USB-C, every device had its own charger, its own cable, its own connector. MCP does the same thing for AI integrations. It provides a universal interface that lets any AI assistant connect to any external data source or tool through a single standard.
What makes this powerful in practice is what I call the single endpoint pattern. Instead of exposing hundreds of complex REST API endpoints to your model (which clutters the context window and invites errors), you give the model one tool. The model describes what it wants in plain language, and the backend translates that into the right API calls, chains them together, and returns the result.
This keeps the model’s context clean while letting the system scale to handle dozens of backend services, including legacy APIs that were never designed for AI consumption. The model stays focused on reasoning. The integration layer handles the plumbing.
For teams evaluating their AI infrastructure, this is the question worth asking: are you building point to point integrations that will need to be rebuilt every time you add a new model or tool, or are you building on a standard that scales?
6. Use AI to Judge AI
One of the hardest problems in production AI is quality control. When your system handles thousands of interactions per day, you cannot have a human review every response. But you also cannot ship without knowing whether the outputs are accurate, on tone, and grounded in real data.
The approach that has become an industry standard is using an LLM as a judge. A separate, powerful model evaluates the outputs of your production system against defined criteria. It scores responses for faithfulness (did the agent stick to the source documents or hallucinate?), correctness (does the answer match what a verified response would say?), tone (is it professional and on brand?), and relevance (did it actually answer the question that was asked?).
This is not a perfect system. An AI judge can have its own blind spots. But it provides something that was previously impossible: continuous, scalable quality monitoring. Instead of sampling 50 conversations a week and hoping they are representative, you can evaluate every interaction and surface the ones that need human attention.
The real impact is in the feedback loop this creates. When you can measure quality at scale, you can iterate at scale. You can identify which types of queries your agent handles poorly, which tool integrations produce unreliable results, and where your prompts need refinement. The system stops being a black box you deployed and hope works. It becomes something you can actively improve with data.
For teams that are serious about production AI, this is not optional. If you are not measuring the quality of your agent’s outputs systematically, you are flying blind.
The Bottom Line
The through line across all six of these lessons is the same: the model is not the product. The system is the product. The teams building AI that actually works in production are spending their energy on tool selection logic, context management, query decomposition, orchestration, integration standards, and quality evaluation. The model sits in the middle of all of that, doing what it does well, but it is not carrying the weight alone.
This is good news for most organizations. It means you do not need to train your own foundation model or wait for the next parameter breakthrough to build something useful. You need good architecture, clear thinking about where deterministic logic beats probabilistic reasoning, and the discipline to measure what your system actually produces.
The question worth asking is not “which model should we use?” It is “what system are we building around it?”
