Optimizing Token Usage in Agentic Loops

Running AI agents in multi-step background loops can generate massive value, but it can also consume a significant amount of tokens. Unlike a standard chatbot where a human sends a 20-word prompt and gets a 100-word response, an autonomous agentic loop involves multiple unseen turns.

An agent might fetch a webpage (10,000 tokens), summarize it (500 tokens), search the database (2,000 tokens), and finally write a report (1,000 tokens). Every time the agent makes a tool call, the entire conversation history and tool schema must be re-sent to the model. If not optimized, this can lead to astronomical API bills from providers like OpenAI or Anthropic.

At remii, we employ several advanced engineering patterns to optimize token usage without sacrificing the agent's reasoning capabilities.

### 1. Dynamic Context Pruning

The biggest driver of token bloat is the conversation history array. As a background task progresses through dozens of steps, the history grows linearly, but the cost grows quadratically because every new turn re-processes all previous turns.

Instead of sending the full history, remii implements dynamic context pruning: - **Rolling Windows**: We only send the last $N$ turns of the conversation directly to the LLM. - **Summarization Checkpoints**: When the history exceeds a certain threshold (e.g., 8,000 tokens), a smaller, cheaper model (like Claude 3 Haiku) generates a dense summary of the early conversation. This summary replaces the raw turns in the context window. - **Tool Result Truncation**: If a tool call (like a web scraper) returns 50,000 characters of HTML, we do not feed the raw HTML back into the main reasoning loop. Instead, a specialized extraction model parses the HTML, extracts just the relevant text, and feeds a 500-token summary back to the main agent.

### 2. Structured Schema Design

Every tool provided to an agent requires a JSON Schema definition in the system prompt. If you give an agent 50 complex tools, the schema definitions alone might consume 3,000 tokens per request.

To optimize this: - **Lazy Loading Tools**: We only inject the schemas for tools relevant to the agent's current state. If the agent is in a "Research" state, it does not need the schema for the "Send Slack Message" tool. - **Concise Descriptions**: We write highly optimized, dense descriptions for parameters. Instead of a paragraph explaining what a `user_id` is, we rely on semantic naming conventions (e.g., `uuidv4_tenant_id`) which the LLM natively understands without extra tokens.

### 3. Tiered Model Routing

Not every task requires the reasoning capabilities of GPT-4o or Claude 3.5 Sonnet.

remii uses intelligent model routing based on the complexity of the specific step: - **High-Cognition Tasks**: Planning, writing final reports, and complex code generation are routed to frontier models. - **Low-Cognition Tasks**: Data extraction, sentiment classification, JSON formatting, and summarization are routed to faster, cheaper models (like Llama 3 or Claude Haiku). This hybrid approach can reduce overall execution costs by up to 80% while maintaining identical output quality.

### 4. RAG over Full-Context

Instead of pasting an entire 50-page PDF into the context window for the agent to analyze, we chunk the document and store it in our pgvector database. When the agent needs information from the document, it uses a `search_document` tool to retrieve only the 3 or 4 relevant paragraphs. This Retrieval-Augmented Generation (RAG) approach replaces a 40,000-token input with a 1,000-token input, drastically cutting costs and reducing "Lost in the Middle" hallucination errors.

### Conclusion

Building autonomous AI systems requires a paradigm shift in how we think about compute costs. By treating tokens as a highly constrained resource and aggressively optimizing context windows, tool schemas, and model routing, remii ensures that background automation remains economically viable at enterprise scale.