AI coding with large codebases - Larger Context Window: more miserable results. Or: why understanding context matters

In the previous article, we looked at why an AI coding agent’s token usage can increase so quickly, and why the number of questions is not necessarily the real problem. In the next parts, we explore how better context can reduce unnecessary token usage while also improving the quality of AI responses.

Why understanding context matters

As we know, the context window is the amount of information a model can consider at once during a request.

That is useful to know, but it is also easy to misunderstand. What does “consider” actually mean?

Modern models support increasingly large context windows, but that does not mean it makes sense to send entire repositories, hundreds of pages of documentation, or complete application packages with every question.

Why?

More context is not the same as better context. The model does not simply need more information. It needs the right amount of the right information for the task. The real value is knowing what actually matters in that specific situation.

Why?

Because models cannot pay equal attention to every piece of information they receive. This is a well-known limitation of LLMs. And the more of the context window we fill, the harder it can become for the model to find the information that really matters. One useful way to think about these models is as extremely smart, sometimes even wise-looking people with no real memory. Their abilities may be impressive, but identifying which part of a large amount of text – and sometimes images or audio – is the most relevant for a specific task is not necessarily what LLMs are best at.

The halo effect is a well-known idea: when someone has one strong positive quality, we tend to assume they are strong in other areas too. The same can happen with LLMs. If a model gives a logical answer to a question, we may assume it must also be good at searching through all the information we gave it. But that is far from guaranteed.

What matters when assessing the impact of a change?

In a business-critical enterprise system, for example, understanding the impact of a change may require knowing:

• which program or piece of code implements the business rule;
• which physical and logical files it uses;
• which batch processes are affected;
• which programs call each other;
• what customer-specific logic has been added over the years;
• which documentation or previous explanations are related to the behaviour;
• which use cases are affected;
• which other systems contain relevant information or need to be connected.

Sending the entire relevant codebase and documentation is not a good substitute for real, meaningful search.

In other words, a large context window is simply a capability. The result we actually want comes from giving the model the output of relevant understanding and search. That is a knowledge retrieval problem.

And that is where sofeei.com is useful.

How Can the Same Answer Have a Completely Different Cost?

Imagine two applications trying to answer the same developer question.

Application A sends the agent a broad, lightly filtered context with every request:

• the full contents of a 150-page technical document;
• several related source files containing tens of thousands of lines;
• a large amount of previous chat history.

This gives the model much more information. But token usage is high, and the relevant details can easily get lost in the noise.

Application B, on the other hand, uses targeted retrieval. First, it finds:

• the programs related to the question;
• the referenced fields and files;
• the relevant call chain or dependencies;
• a few relevant sections of business documentation containing the related rules.

Then it sends only the most relevant content to the agent.

The developer’s question is the same. The expected answer quality may be similar or even better. But token usage can be very different.

So the goal should be to give the agent enough relevant information for the task, ideally backed by clear sources.

That is why a good RAG-based enterprise code search approach is not simply about “uploading documents”. The goal is to give the model or agent the code, dependencies and business knowledge that matter for the specific task – not the entire digital archive.

Coming next: what does this mean in practice?