Search for a command to run...
No comments yet. Be the first to comment.
Every October, the developer community buzzes with Hacktoberfest energy. PRs fly, t-shirts are earned, and GitHub turns green. But here's what nobody talks about: what happens in November? For most contributors, the answer is simple: nothing. The rep...
Learnings from a Research Paper
Lately, I have been working on a feature to capture images from media streams and scale them down to reduce their size. This helps save storage and reduce costs before uploading the images to a storage service. For this, I used Canvas to render the i...
AI helps you do more, but only if you know the basics well enough to tell it what you want.
The debate over AI coding agents is missing the most important factor. It’s not about prompt engineering, it’s about understanding the context window.
Developers are divided. One side claims coding agents suck. The other insists you’re just using them wrong; it’s a skill issue. While there is truth to both perspectives, the most common skill issue isn’t about prompt engineering , it’s about a fundamental misunderstanding of the tool’s primary constraint.
If there is a skill issue that I see most often with devs, it is not thinking enough about the context window.
The context window is the complete set of input and output tokens an LLM processes in a single session. Think of it as the model’s working memory, everything it can see and consider when generating a response.
This includes:
Input Tokens: Your system prompt, instructions, and user messages
Output Tokens: The assistant’s generated responses
As your conversation grows, so does the token count. Eventually, you’ll hit a limit set by the model’s provider. This can happen if a conversation becomes too long, or even from a single, very large input like uploading extensive documentation. Exceeding this limit results in an error, and generation will stop cold.
Model providers set different limits based on architecture and cost. These can range from a few thousand to several million tokens , but as we’ll see, bigger isn’t always better.
Gemini has really large context windows, but as we’ll see… bigger is not always better.
Here’s the counterintuitive truth: the more information you give a model, the worse it performs at retrieving specific details. This is true for all models, from the smallest to the largest.
Why doesn’t infinite context exist? Two reasons:
LLM processing is expensive. Larger contexts consume significantly more memory per process, driving up both computational costs and latency.
This is the critical one. An LLM’s attention is not distributed evenly across the context. Information at the very beginning and very end of a conversation has the most impact on the output. Information in the middle? It’s often de-prioritized or ignored entirely.
This is called the Lost in the Middle problem.
This isn’t a bug, it’s an emergent property of transformer architecture. Interestingly, it mirrors human cognitive biases:
Primacy Bias: Better recall for items at the beginning
Recency Bias: Better recall for items at the end
Just like humans do, models do better with less, more focused information.
When Meta announced a model with a 10-million-token context window, it seemed like a breakthrough. However, real-world testing quickly revealed it suffered from severe lost-in-the-middle problems. You could feed it vast amounts of information, but the model would fail to retrieve or act on it effectively.
When you’re assessing an LLM, you shouldn’t just look at how big the context window is. You should look at how well it retrieves information from its context window.
Since shorter context windows suffer less from the lost-in-the-middle problem, the key to better performance is proactive management. Regularly clearing your coding agent’s chat history refreshes its memory and ensures your instructions are given high priority.
This is the single most effective way to improve results.
You cannot manage what you cannot measure. A good coding agent provides tools to inspect the current state of your context window. For example, in Cursor:
clear the ConversationWhen you start a new, unrelated task, or when context usage gets high (e.g., less than 50k tokens free), the best practice is to clear the conversation history entirely. This frees up the entire context window, giving you a blank slate and ensuring maximum performance for the new task.
compact When You Need to Preserve the Conversation's Intentcompact is an alternative that clears the detailed history but generates an LLM-powered summary. This preserves the vibes or core goals of the conversation in a much smaller package.
This preserves some of the intention… like a mini rules file just for this conversation.
While useful, be aware that generating the summary itself takes time and consumes tokens.
Be extremely cautious about tools and configurations that add large amounts of hidden context. This bloats your window from the start, pushing your actual conversation into the dreaded middle.
Common Culprits:
LSP/MCP Servers: These can inject enormous toolsets into your system prompt
Large Rule Files: Overly complex or numerous custom rules in tools like Cursor or Claude Code
When hidden tools consume most of your context, your actual messages end up in the lost in the middle zone , exactly where the model pays the least attention.
The context window is the model’s entire memory (input + output). As conversations grow, this fills up quickly and performance suffers.
All models have a hardcoded limit and suffer from lost-in-the-middle attention decay. Even million-token windows aren’t immune to this problem.
A leaner, more focused context consistently yields better performance. Clear early, clear often.
Develop a healthy paranoia about what’s in your context. Actively manage it with tools like clear and compact. This skill is what separates frustrating interactions from productive partnerships with AI.
Mastering the context window is the key to great results.