Building AI Agents in Production: Hard-Won Lessons on Hybrid Architectures, Structured Outputs, and Agentic RAG

Over several months, the author's work on AI agents moved from side projects and curiosity to paid, production work with real users. Being paid to build is what marks the shift from hobbyist to professional. Despite reading widely, the author encountered many challenges that are not yet commonly discussed, and this article aims to share those hard-won lessons. The projects range from toy experiments, such as a LinkedIn job finder and a QA automation tool, to internal work-related integrations and a product analytics assistant now entering closed beta.

An AI agent orchestrates multiple LLM calls to make decisions, use tools, and reach goals. It is code wrapped around LLM calls that lets the model determine its own path toward an objective rather than producing a single response. In practice, single LLM calls for specific tasks are often sufficient, and it is more useful to think of these as single-step, tool-less agents than to draw a rigid line between agents and calls. Agents excel at tasks without a conventional algorithm: generating creative content, making nuanced subjective judgments, extracting meaning from unstructured data, and adaptive control flow where guidelines replace rigid if-else logic. LLMs can also mimic traditional machine learning systems such as recommenders, handling arbitrary text or images without training on custom data.

Before assigning a task to an agent, ask whether it can be coded without losing functionality. Traditional code is typically more performant, accurate, predictable, testable, and cost-effective, since it incurs no token charges. Use code for mechanical work such as scraping, loops, and file operations, and reserve agents for reasoning and adaptation. The author learned this with a LinkedIn job finder: an agent with browser access struggled with a virtual scrolling list, and replacing it with a simple Playwright script improved speed, accuracy, and cost. The tradeoff is fragility, since scrapers can break when markup changes. Similarly, loops over documents should be normal code that spawns an agent per item, potentially in parallel. The most powerful systems are hybrids in which code handles determinism and integration while LLMs handle understanding and creativity.

Every model fails beyond a certain complexity threshold, after which instruction-following degrades and hallucinations rise sharply. Once a task crosses that threshold, the options are to reduce requirements, upgrade to a more capable and costly model, or split the work across specialized agents. In the hypothetical marketing email system, a single agent handling data collection, product selection, writing, and review would be unreliable. Splitting it into a Data Collector, Product Selector, Writer, and Reviewer/Editor yields focused prompts, simpler error recovery, and independently testable and reusable steps. Multiple agents make sense when a task decomposes naturally, when subtasks require different reasoning types, when prompts become unwieldy, or when different parts of the process need separate state.

Structured outputs, defined by JSON schemas, transform agent development by guaranteeing that the model returns data in an exact format. This removes the need to beg for correct formatting or retry malformed responses. A less obvious benefit is that schemas can shape reasoning itself: requiring a justification field, a relevance score, or a sellingPoints array forces the model to think through its choices before answering. With structured outputs, calling an LLM becomes akin to calling any other remote API, and the results can be parsed and passed along by code or other agents.

Python dominates the AI and ML ecosystem, offers rich data processing, has Python-first agent frameworks and tutorials, and is familiar to data scientists. TypeScript offers compile-time static typing, first-class runtime validation through Zod, native JSON handling, robust async/await patterns, and unified frontend and backend development. The author personally leans toward TypeScript because Zod schemas can infer static types automatically, giving autocompletion and type checking over validated model output. Both languages are excellent, and the choice should depend on team expertise and specific requirements.

Production agents cannot rely on the iterative clarification that makes casual chat work, so their prompts must anticipate diverse situations without human intervention. Prompts grow long, often dwarfing the user query, and must define domain terminology, tool usage, strategies, output formats, and error handling. A writer agent prompt, for example, may specify brand voice rules, email structure, process steps, and critical guidelines such as word limits and pricing restrictions. The author recommends being explicit and detailed, iterating through testing across diverse queries, structuring prompts logically with separate sections, and including examples that cover edge cases. This work takes real time and iteration and directly determines system performance.

LLMs are stateless: each call is new, and the appearance of memory comes from resending the entire conversation history. The context window sets the maximum tokens a model can see at once, and modern windows range from 128K to 1M tokens. However, larger contexts increase latency and cost, and API token-per-minute limits often constrain usage well below the window size. For instance, a 30,000 TPM limit means a 128K window cannot be fully used even with one request per minute. Effective token management includes pruning irrelevant history, including only critical tool outputs, and parallelizing tool calls to reduce round trips. Executing multiple tool calls concurrently and returning their results together improves latency and makes better use of TPM quotas.

Retrieval-Augmented Generation is central to agents that need external knowledge, but basic implementations fall short. Traditional RAG is code-driven, with static queries suited to predictable needs. Agentic RAG places the agent in control, enabling dynamic, multi-step retrieval for exploratory tasks. The author initially held two misconceptions: that embeddings are simple keyword matching, when they actually capture semantic relationships, and that everything should be stuffed into the context window, when extra material degrades performance by burying signal in noise. Traditional RAG works well when retrieval needs are predictable, such as pulling example email templates into a writer's prompt, while agentic RAG is better when the agent must decide what to look for and when to look again.

Effective agent systems are hybrids in which traditional code handles deterministic work and LLM agents handle reasoning, creativity, and adaptation. Specialized agents keep tasks below complexity thresholds, structured outputs make LLM calls reliable and API-like, prompt engineering deserves genuine engineering rigor, and the context window must be managed deliberately through pruning and parallelization. Agentic RAG extends agents toward dynamic knowledge retrieval. The author is still early in this journey, and these lessons reflect ongoing learning from real projects rather than settled expertise.