Why is your AI bill high?

Introducing an AI coding agent often starts the same way for development teams.

The first test looks impressive: the agent responds quickly, explains code, suggests tests, can even run tests on its own, and at first glance seems to find its way around the codebase surprisingly well. Then the bill arrives at the end of the month, together with the obvious question:

Why is using AI so expensive?

The simple answer: the cost depends largely on how much text, code, documentation and previous conversation history the AI has to process for every request before it can understand and solve the task. All of that gets turned into tokens, again and again. With usage-based AI pricing, the amount of input tokens processed and output tokens generated affects the final cost.

And of course, a token is often smaller than a word. So a large amount of text usually means an even larger number of tokens.

As a quick reminder: large language models – and the AI coding agents built on top of them – do not “read” input as words. A token may be a short word, part of a word, punctuation, or a piece of code. The exact number of tokens depends on the language you use to communicate with the agent, the programming language, the model and the tokenizer.

Two things are certain:

• the exact number of tokens is not always easy to predict in advance;
• there is a natural tension between users and LLM providers: we pay for tokens, while providers earn more when more tokens are processed.

There is also something we regularly forget: it is not only the question we directly ask that counts. Tokens are typically used by:

• the system prompt;
• previous conversation history – I keep forgetting this one myself too! :) (/compact)
• the user's current question;
• source code sent to the model – more on this later;
• related documentation – more on this too;
• content retrieved by a RAG system and added to the model's prompt;
• results returned by tool calls, such as MCP tools;
• and the model's answer itself.

So behind what looks like a very short question, there may actually be thousands of tokens worth of background information.

Let's look at a simple example of how one question can turn into several thousand tokens.

Imagine a developer asks an AI agent: “Which systems could be affected if we change the customer credit limit check?” To answer, the application might send the following:

And that is just one question. If 100 developers use an AI tool every day, each making an average of 10 requests, and each request uses the 6,000 tokens from the example above, that adds up to 6 million tokens per day. In a larger organisation – or with agent-based automation – that number can grow very quickly.

Important: the exact cost varies by model and provider. Input and output tokens are not always priced the same way, and other components such as embeddings, search infrastructure and agent tool calls can also add to the total AI cost. Still, token usage remains one of the major cost drivers.

So rising token costs do not necessarily mean that developers are asking too many questions. A common reason is that the AI has to process too much information that is only partly relevant, just to find the information it actually needs.

And we have not even talked about one of the biggest token consumers yet: source code. Most agents “look through” the source code – but they do not do this quietly on their own. They usually work together with the LLM. A typical process may start with a grep search. If you know grep, you know how useful it is – probably one of the best Swiss Army knives ever made for finding text inside text. The agent sends the search results to the LLM, and the LLM processes them as tokens and sends back an answer. If the result is not useful enough, the process may repeat. Remember: the agent often relies on the LLM to decide whether the information it found is relevant. So it may search again, read another part of a directory, run another grep query, and send even more content to the LLM.

Another major token consumer is documentation. Very often, we do not have the time – or simply do not want – to manually find and copy the exact chapter or section of a document that contains the answer. So the agent sends the documentation to the LLM instead, and the LLM does that work for us. That consumes input tokens.

One way to address these problems is to place a memory and search layer in front of the AI that understands both the code and the documentation, and can search more like an experienced senior developer – something like Sofeei. The benefit is not only lower token usage. It can also mean faster responses and better answers.

Coming next: what does this mean in practice?

This article showed why the token usage of an AI coding agent can grow so quickly, and why the number of questions developers ask is not necessarily the real problem.

In the next parts, we will look at how better context can reduce unnecessary token usage while also improving the quality of AI responses.